Activity test methods and equipment
Through the processor combining color and infrared images in face verification, the problem of difficult to distinguish between real and deceptive faces in the prior art is solved, and higher verification accuracy and security are achieved.
Patent Information
- Application Number
- CN201811276147.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-01-22
- Filing Date
- 2018-10-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2038-10-30
AI Technical Summary
The prior art is difficult to effectively distinguish between real and deceptive faces in face verification, especially when facing deceptive methods such as images, videos or masks, resulting in false acceptance or rejection.
The activity test method implemented by a processor is used to obtain color images and infrared images, and perform first and second activity tests using neural network models. The activity of the object is determined based on the test results, and the test data is generated by the fusion of color images and infrared images. , perform a third activity test to improve accuracy.
It improves the accuracy of face verification, effectively filters out or prevents deceptive face attempts, enhances the ability to recognize real faces, and reduces the risk of wrong acceptance.
Smart Images

Figure CN110069970B_ABST
Abstract
Description
[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0007814 filed on January 22, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field
[0002] The following description relates to apparatus and methods having liveness detection and verification of objects in images. Background Art
[0003] In a user authentication system, a computing device may determine whether to allow a user to access the computing device based on authentication information provided by the user. For example, the authentication information may include a password entered by the user and the user's biometric information. For example, the biometric information may include information associated with the user's fingerprint, iris, and face.
[0004] Recently, there are face anti-spoofing methods, where, for example, face spoofing can be an attack using an image, video, or mask, intended to deceive or defraud an underlying device or system into believing that an authorized user or individual is attempting to access or use the device. Face anti-spoofing technology can be used to determine whether a user's face input to such a device or system is an impostor face or a real face. Face anti-spoofing technology may include: extracting features (such as local binary patterns (LBP), histograms of oriented gradients (HOG), and Gaussian difference (DoG)) from an input image, and determining whether the input face is impostor or real based on the extracted features. Summary of the invention
[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0006] In one general aspect, a processor-implemented activity testing method is provided, comprising: obtaining a color image including an object and an infrared (IR) image including the object; performing a first activity test using the color image; performing a second activity test using the IR image; and determining the activity of the object based on a result of the first activity test and a result of the second activity test.
[0007] The method may further include: verifying whether the object corresponds to an authorized user; switching the device from a locked state to an unlocked state based on a result of the verification, wherein the step of obtaining a color image and an IR image includes obtaining a color image and an IR image using one or more image sensors of the device.
[0008] The verifying step and the switching step may be selectively performed according to the result of determining the activity of the object.
[0009] The step of performing the first liveness test may be implemented using a neural network based first liveness test model using a color image as input.
[0010] The step of performing the second liveness test may include using a second liveness test model based on a neural network using the IR image as input.
[0011] The step of determining the liveness of the subject may include determining that the subject is live in response to a result of the first liveness test indicating that the subject is live and a result of the second liveness test indicating that the subject is live, determining that the subject is live.
[0012] The step of determining the activity of the object may include determining whether the object is alive based on a first score obtained from a result of the first activity test and a second score obtained from a result of the second activity test.
[0013] The determining of the activity of the object may include determining that the object is alive in response to a combined value obtained based on the first score and the second score being greater than a threshold value.
[0014] The image sensor may include: one or more color light receivers configured to measure any one of the red (R) color value, green (G) color value, and blue (B) color value of a color image, or any combination of any two or more thereof; and one or more IR light receivers configured to measure IR values of an IR image.
[0015] The color image and the IR image may include one or both of a face region of the subject and an iris region of the subject.
[0016] The method may further include: detecting a first region of interest (ROI) of the object in the color image, wherein the step of performing a first activity test includes performing the first activity test based on the detected first ROI; and detecting a second ROI of the object in the IR image, wherein the step of performing a second activity test includes performing a second activity test based on the detected second ROI.
[0017] The first ROI and the second ROI may correspond to different regions of the object.
[0018] Performing the first liveness test may include determining whether the color image has captured the IR representation of the object; performing the second liveness test may include determining whether the IR image has captured the color representation of the object.
[0019] The IR representation of the object may be a material IR image of the object, and the color representation of the object may be a material color image of the object.
[0020] The method may further include performing a third activity test using the color image and the IR image, wherein the determination of the activity of the object is based on a result of the third activity test.
[0021] The method may further include generating a fused image by fusing the IR image and the color image, wherein a third liveness test is performed using the fused image.
[0022] In another general aspect, a processor-implemented activity testing method is provided, comprising: generating test data for an activity test to be performed on an object based on a color image including the object and an infrared (IR) image including the object; and determining the activity of the object based on a result of implementing an activity testing model using the test data as input.
[0023] The step of generating the test data may include generating the test data by combining a color image and an IR image.
[0024] The determining of the activity of the object may include determining whether the object is alive based on a result of comparing a score obtained from the activity test model with a preset threshold value.
[0025] In another general aspect, a processor-implemented activity testing method is provided, comprising: performing a first activity test using a color image including a test object; performing a second activity test using an infrared (IR) image including the object; performing a third activity test using the color image and the IR image; and determining the activity of the object based on results of the first activity test, results of the second activity test, and results of the third activity test.
[0026] The step of performing the third activity test may include: combining the color image and the IR image into a single image; and implementing the activity test model using the single image.
[0027] The step of determining the activity of the object may include determining whether the object is alive based on a first score obtained from a result of the first activity test, a second score obtained from a result of the second activity test, and a third score obtained from a result of the third activity test.
[0028] A non-transitory computer-readable storage medium may store instructions, wherein when the instructions are executed by one or more processors, the one or more processors are caused to perform the method.
[0029] In another general aspect, an activity testing device is provided, comprising: one or more image sensors configured to obtain a color image including an object and an infrared (IR) image including the object; one or more processors configured to: perform a first activity test using the color image; perform a second activity test using the IR image; and determine the activity of the object based on a result of the first activity test and a result of the second activity test.
[0030] The one or more processors may also be configured to perform a first activity test using a first neural network-based activity test model using a color image as input and to perform a second activity test using a second neural network-based activity test model using an IR image as input.
[0031] The one or more processors may be further configured to: perform a third activity test based on a combination of the color image and the IR image; and determine activity of the object based on a result of the first activity test, a result of the second activity test, and a result of the third activity test.
[0032] The one or more processors may also be configured to: detect a first region of interest (ROI) of an object in a color image; perform a first activity test based on the detected first ROI; detect a second ROI of the object in an IR image; and perform a second activity test based on the detected second ROI, wherein the first ROI and the second ROI correspond to different regions of the object.
[0033] The one or more image sensors may also be configured to measure color values of the color image and IR values of the IR image.
[0034] The one or more image sensors may also include: one or more color light receivers configured to measure any one of the red (R) color value, green (G) color value, and blue (B) color value of a color image, or any combination of any two or more thereof; and one or more IR light receivers configured to measure IR values.
[0035] The apparatus may further include a memory storing instructions, wherein when the instructions are executed by the one or more processors, the one or more processors are configured to perform the first and second liveness tests and the determination of liveness.
[0036] In another general aspect, an activity testing device is provided, comprising: one or more image sensors configured to obtain a color image including an object and an infrared (IR) image including the object; one or more processors configured to: implement an activity testing model using data generated based on the color image including the object and the IR image including the object; and determine the activity of the object based on the result of implementing the activity testing model.
[0037] In another general aspect, a processor-implemented activity test method is provided, comprising: performing a first activity test using one or both of an infrared (IR) image including a detected object and a color image including the detected object; generating a fused image by fusing the IR image and the color image; performing a second activity test using the fused image; and determining the activity of the object based on a result of the first activity test and a result of the second activity test.
[0038] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a diagram illustrating an example execution of a liveness test.
[0040] Figures 2A to 4 is a flow chart illustrating an example activity testing method.
[0041] Figure 5A and Figure 5B is a diagram showing an example of using a color image and an infrared (IR) image in an example activity test.
[0042] Figure 6 is a diagram illustrating an example of a pixel array of an integrated image sensor in which a color sensor and an IR sensor are combined.
[0043] FIG. 7A to FIG. 9 is a diagram illustrating an example activity testing method.
[0044] Fig.10 is a diagram illustrating an example of an active test device.
[0045] Fig.11 is a diagram illustrating an example of a computing device.
[0046] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same or similar elements, features and structures. For clarity, illustration and convenience, the drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated. DETAILED DESCRIPTION
[0047] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will become clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will become clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and brevity, the description of known features may be omitted.
[0048] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many possible ways of implementing the methods, devices and / or systems described herein that will be clear after understanding the disclosure of the present application.
[0049] The terms used herein are only for the purpose of describing specific examples, rather than for limiting the present disclosure. Unless the context clearly indicates otherwise, as used herein, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As used herein, the terms "include", "comprise" and "have" indicate that there are features, quantities, operations, elements, components and / or combinations thereof that are set forth, but do not exclude the presence or addition of one or more other features, quantities, operations, elements, components and / or combinations thereof.
[0050] In addition, terms such as first, second, A, B, (a), (b), etc. may be used herein to describe components. Each of these terms is not used to limit the essence, order or sequence of the corresponding component but is only used to distinguish the corresponding component from other components.
[0051] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on", "connected to," or "coupled to" another element, it may be directly "on," "connected to," or "coupled to" the other element, or one or more other elements may be present between them. Conversely, when an element is described as being "directly on," "directly connected to," or "directly coupled to" another element, no other elements may be present between them. Similarly, for example, the expressions "between" and "directly between," as well as "adjacent to" and "immediately adjacent to" may also be interpreted as described above.
[0052] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present disclosure belongs, after and consistent with the understanding of the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as an idealized or overly formal meaning.
[0053] Figure 1 is a diagram illustrating an example of a liveness test.
[0054] The activity test may be performed to determine whether the test object as the target of the activity test is alive. As a non-limiting example, the activity test may be performed to determine whether the face in the image captured by the camera is a real face or a fake face, or to determine whether the face in the image belongs to a living person in the image or an inanimate object in the image. In one example, the activity test may be used to determine the activity of the verification object for further user verification performed for user login, payment service, access control, etc. in various examples. Here, the verification object of the user verification represents the target under user verification. As a non-limiting example, in the user verification system, such a liveness test may be performed to distinguish between inanimate objects (as a non-limiting example, photos, images, paper and copies as a mechanism or form of impersonation) and animate objects (as a non-limiting example, animate people) using the captured image, such as by distinguishing whether the object in the captured image is a living object or more likely to be a living object or the object is a deceptive object or more likely to be a deceptive object, or by positively determining whether the object is or is not a living object or is or is not a deceptive object. Liveness testing may also be performed to prevent false acceptance by filtering out or blocking attempts at authentication (as non-limiting examples, spoofing attacks using substitutes (as non-limiting examples, images, videos, masks, contact lenses, replicas, etc.)). The use of the term "may" herein with respect to examples or embodiments (e.g., with respect to what an example or embodiment may include or may implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples and embodiments are not limited thereto.
[0055] An invalid user may use spoofing techniques in an attempt to obtain a false acceptance by the user verification system. As a non-limiting example, an invalid user may present a color image, video, replica, etc., of what a valid user's face appears to a color camera of the user verification system to obtain a false acceptance in face verification. For another example verification where iris detection may be optionally performed using infrared (IR) images, an invalid user may present an IR image, video, replica, etc., of what a valid user's iris area appears to an IR camera of the optional user verification system to obtain a false acceptance in iris verification. Thus, liveness testing may also be performed to prevent such false acceptances by filtering out or blocking attempts to verify based on such spoofing techniques.
[0056] The liveness test may be performed using images (as non-limiting examples, still images and videos) acquired by an image acquirer (as non-limiting examples, a camera). As non-limiting examples, the liveness test to be described below may be performed using both color images and IR images. The face area in the color image may be analyzed by a typical face verification in one method, while the iris area in the IR image may be analyzed by a typical iris verification in a significantly different method. Such typical methods are different and only one is performed separately for a specific application, i.e., when face verification is performed, color image analysis of the detected face area may be performed, while when iris verification is performed, IR image analysis of the detected iris area may be performed. However, compared to performing liveness tests based on only one of the color image and the IR image as in the above general method, the example herein performs an example liveness test based on consideration of both the color image and the IR image of the object according to the underlying face or iris application. Such an example herein may individually improve the accuracy of liveness tests compared to any of the aforementioned methods. As a non-limiting example, such technical improvements may be achieved because features in the color image and features in the IR image may complement each other when performing the example liveness test or the example may rely on complementary aspects of such features in the color image and the IR image.
[0057] Reference Figure 1 , an activity testing device configured to perform activity testing on a test object may be included in or represented by computing device 120. As non-limiting examples, computing device 120 may be a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, and a vehicle starting device. In one example, the activity testing device may correspond to Fig.10 Active testing device 1000 and / or Fig.11 The computing device 1100.
[0058] exist Figure 1 In the example shown in , user 110 may attempt user authentication to access computing device 120 through face authentication or iris authentication performed by an active test device or computing device. As a non-limiting example, in the case where user 110 attempts face authentication to cancel the locked state of computing device 120, computing device 120 obtains a facial image of user 110 using color camera 130, analyzes the obtained facial image, and determines whether to cancel the locked state of computing device 120 based on the analyzed authentication result. For another example, in the case where user 110 attempts iris authentication to cancel the locked state of computing device 120, computing device 120 activates IR light source 135, analyzes the iris image of user 110 obtained by IR camera 140, and determines whether to cancel the locked state of computing device 120 based on the analyzed authentication result. In this example, when user authentication is successful, computing device 120 cancels the locked state and allows user 110 to access computing device 120. Conversely, when user authentication is unsuccessful, computing device 120 continues to work in the locked state.
[0059] In such a user authentication example embodiment, a liveness test may be performed to determine whether the authentication object of the user authentication is a living object or an inanimate object by an activity testing device before the result of the user authentication is determined. Here, in response to determining that the authentication object is a living object as a result of the liveness test, the computing device 120 may then perform a determination of whether the user authentication is successful. Conversely, in response to determining that the authentication object is an inanimate object, the computing device 120 may continue to operate in a locked state without performing such an operation of determining whether the user authentication is successful.
[0060] In such a liveness test process, the liveness test device may, for example, perform liveness tests on the test object using both the color image captured by the color camera 130 and the IR image captured by the IR camera 140, regardless of the type of user verification. Using the color image and the IR image together may enable comprehensive consideration of the color, reflection, and texture features of the test object for the liveness test. Here, although the IR image may indicate IR distribution features instead of color distribution features, so that the IR image cannot be used to determine the color features of the test object and the texture features indicated by the reflection of visible light, for example, the color image may be used to determine the color features of the test object and the texture features indicated by the reflection of visible light to perform liveness tests. In addition, the IR image may also be used to determine the reflection features of the test object and the texture features indicated by the reflection of IR light to perform liveness tests. The IR image obtained using the IR camera 140 may indicate facial skin features that are different from the facial skin features indicated in the color image. In addition, the color image may also be used to determine whether an IR representation of an object has been captured, and the IR image may also be used to determine whether a color representation of an object has been captured, wherein the IR representation of an object is a material IR image of the object, and the color representation of an object is a material color image of the object.
[0061] As a non-limiting example, when an invalid user presents a color image including a face (impersonated face) to computing device 120 to induce a false acceptance in face verification, the IR image obtained by capturing the color image by IR camera 140 may be different from the IR image obtained by directly capturing the test subject by IR camera 140. Therefore, although color camera 130 may capture a color image including a face for liveness verification, since capturing an IR image including a color image of a face is different from capturing an IR image of an animate test subject, it is also beneficial for IR camera 140 to capture a color image including a face for liveness verification. For another example, when an invalid user presents an IR image including an iris to computing device 120 to induce a false acceptance in iris verification, the color image obtained by capturing the IR image by color camera 130 may be different from the color image obtained by directly capturing the animate test subject by color camera 130. Therefore, although the IR camera 140 can capture an IR image including an iris for liveness testing, since capturing a color image including an IR image of an iris is different from capturing a color image of a living test object, it is also beneficial for the color camera 130 to capture an IR image including an iris for liveness testing. Therefore, the liveness testing device of the present disclosure that determines the liveness of a test object using both a color image and an IR image obtained by capturing the same test object is more effective than a typical liveness testing device that determines the liveness of a test object based on either a color image or an IR image but not both in effectively filtering out or preventing attempts at such false acceptance based on the aforementioned differences. Compared to a typical liveness testing device that determines the liveness of a test object based on, for example, only an IR image, the liveness testing device of the present disclosure that determines the liveness of a test object using a color image obtained by the color camera 130 can effectively help determine whether an input test object is an actual person or an IR image presented to induce false acceptance. In addition, using the IR image obtained by the IR camera 140 can help effectively determine whether an input test object is an actual person or a color image presented to induce false acceptance. As described, the activity testing device of the present disclosure that uses both color images and IR images to determine the activity of a test object can improve the accuracy of activity testing compared to the aforementioned typical activity testing devices that determine the activity of a test object based on, for example, color images or IR images but not both.
[0062] Therefore, in one example, the activity test device may use an activity test model for activity testing. In one example, the activity test model may be based on a neural network or other machine learning example configured to output a value calculated by an internal parameter based on input data. As a non-limiting example, the activity test model may be an activity (or activity verification) model based on a neural network and configured so that when its trained parameters are read from a memory by one or more processors, the one or more processors may implement an activity test model of an input image to calculate, determine or extract an activity score specific to the training target, for example, an activity score of a color image, an activity score of an infrared image, and / or an activity score of a synthetic image. Using such a neural network-based activity test model to obtain the result of the activity test by jointly considering the color image and the IR image can improve the performance and accuracy of the activity test. The internal parameters of the activity test model may be determined by training the activity test model. As a non-limiting example, in training, there may be multiple sets of training data and the expected values of the activity test model corresponding to the multiple sets of training data, respectively. When a particular set of training data is input to the activity test model, the initial internal parameters of the activity test model may be adjusted so that the activity test model may output the expected values corresponding to the input training data. Such a training process may be performed for each set of training data, so that the initial internal parameters or temporary internal parameters may be adjusted throughout the process so that the activity test model may gradually output a more desired value. The training may be performed by back propagation or gradient-based training, but the examples are not limited thereto. In the training process, the activity test model may be trained to distinguish differences in color features, reflection features, texture features, etc. between a color image obtained by a color camera and an IR image obtained by an IR camera, for example, until a predetermined accuracy and / or a predetermined inaccuracy is met, and the finally trained activity test model is stored in a memory.
[0063] In one example, the activity test model may be a model configured to perform a function of extracting feature values from image data (as a non-limiting example, pixel values), rather than being based in whole or in part on a neural network. As a non-limiting example, the activity test model may extract feature values reflecting color features, reflection features, texture features, etc. of the test object using an image feature extraction method known in the art to which the examples described herein belong. In one example, the extraction may be performed, for example, by a neural network extractor trained to extract such feature values, and then the activity may be determined based on the extracted feature values, respectively.
[0064] Hereinafter, how an activity test apparatus performs an activity test method on a test object using a color image and an IR image will be described in detail with reference to the accompanying drawings.
[0065] Figure 2Ais a flow chart showing an example of an activity test method. Figure 2A The described activity test method can be performed by an activity test device (such as, Figure 1 , Fig. 9 or Fig.10 Active test equipment (note that the example is not limited to this) is performed.
[0066] Reference Figure 2A In operation 210, the active test device receives a color image and an IR image. The color image and the IR image may be obtained by, for example, a plurality of image sensors or cameras of the active test device configured to independently sense or measure color values and IR values, or a single image sensor configured to measure both color values and IR values. The color image and the IR image may be captured simultaneously or at preset time intervals.
[0067] In one example, the color image and the IR image may be obtained by capturing an image of the same test object (as a non-limiting example, an object for which user verification is attempted) and include a region of interest (ROI) of the test object. According to one example, the same type of ROI (as a non-limiting example, a face area or an iris area) may be detected in the color image and the IR image, respectively, to perform a liveness test on the test object, or different types of ROIs or ROIs with different features may be detected in the color image and the IR image, respectively. As a non-limiting example, for a liveness test, the entire face area may be extracted from the color image, and a partial face area may be detected in the IR image. Alternatively, for a liveness test, a face area may be detected in the color image, and an iris area may be detected in the IR image. In the following, the color image used for the liveness test may be an image corresponding to the ROI detected in the color image, and the IR image used for the liveness test may be an image corresponding to the ROI detected in the IR image.
[0068] Although not shown, according to one example, the activity test device may perform image preprocessing on at least one of the received color image and the received IR image. The image preprocessing may include at least one process for processing the color image and / or the IR image to be more suitable for activity testing. As non-limiting examples, the image preprocessing may include adjusting the size of the image, rotating the image, removing noise included in the image, increasing the contrast of the image, deblurring the image, removing a background area from the image, warping to correct distortion in the image, cropping the image, and binarizing the image.
[0069] In operation 220, the activity testing device obtains a first activity test result by performing an activity test using a color image. In one example, the activity testing device may obtain a first activity test result for the color image using a first activity testing model. The first activity test result may include information about whether a feature related to the activity of the test object is indicated in the color image. The first activity testing model may provide information (as a non-limiting example, a probability value and a feature value) to be used to determine the activity of the test object based on the information input to the first activity testing model. In one example, the activity testing device may also determine whether the color image has captured an IR representation of the object, wherein the IR representation of the object is a material IR image of the object.
[0070] In one example, the first activity test model may be a neural network-based activity test model using a color image as input. As a non-limiting example, the first activity test model may be an activity (or activity verification) model based on a neural network and configured so that when its trained parameters are read from a memory by one or more processors, the one or more processors may implement a first activity test model of an input image to calculate, determine or extract an activity score specific to a training target, for example, an activity score of a color image. As a non-limiting example, the first activity test model may be based on a deep convolutional neural network (DCNN). The DCNN may include a convolutional layer, a pooling layer, and a fully connected layer. Through the operations performed by each layer, the DCNN may provide information to be used to determine activity based on image information input to the first activity test model. Here, the image information may be a pixel value (as a non-limiting example, a color value and / or a brightness value) of a pixel included in the image. DNCC is provided herein only as an example, so the first activity test model may be based on a neural network of another structure other than the DCNN described above.
[0071] Here, the parameters configuring the first activity test model can be determined by training performed based on the training data. In such a training process, there may be multiple sets of training data and expected values corresponding to the multiple sets of training data respectively. The first activity test model can receive the training data as input, and output the result value corresponding to the training data through calculation or operation based on the parameters. Here, the loss can be calculated based on the difference between the output result value and the expected value, and the first activity test model can be trained to reduce the loss by adjusting the parameters configuring the first activity test model. Such a training process can be repeatedly performed for each set of training data in the multiple sets of training data, and therefore, the parameters of the first activity test model can be adjusted more as expected throughout the training process.
[0072] The first activity test model may be based on a neural network as described above, but is not limited thereto. As a non-limiting example, the first activity test model may be a model configured to output a feature value reflecting the color feature, reflection feature, or texture feature of the test object from a color image input to the first activity test model by a known feature extraction method rather than based on a neural network.
[0073] The first activity test result may be a result of determining the activity of the test object (as a non-limiting example, determining whether the test object is a living object) based on the first output value output from the first activity test model. As a non-limiting example, the first activity test result may be one of a result value indicating that the test object is a living object and a result value indicating that the test object is an inanimate object. For another example, the first activity test result may be a first score based on the first output value output from the first activity test model. As a non-limiting example, the first score may be equal to the first output value or a value obtained by a predefined calculation including the first output value as an element.
[0074] In operation 230, the activity testing device obtains a second activity test result by performing an activity test using the IR image. The activity testing device may obtain the second activity test result using a second activity test model based on a neural network using the IR image as an input. The second activity test result may include information about whether a feature related to the activity of the test object is indicated in the IR image. In one example, the activity testing device may also determine whether the IR image has captured a color representation of the object, wherein the color representation of the object is a material color image of the object.
[0075] Similar to the first activity test model, the second activity test model may be a model configured to provide information (as a non-limiting example, probability values and feature values) to be used to determine the activity of the test object based on information input to the second activity model. The second activity test model may also be based on a DCNN or a neural network of another structure different from DCNN. As a non-limiting example, the second activity test model may be an activity (or activity verification) model based on a neural network and configured such that when its trained parameters are read from a memory by one or more processors, the one or more processors may implement a second activity test model of the input image to calculate, determine or extract an activity score specific to the trained target, for example, an activity score of an IR image. Alternatively, the second activity test model may be a model configured to output feature values reflecting the color features, reflection features or texture features of the test object from an IR image input to the second activity test model by a known feature extraction method rather than based on a neural network.
[0076] In one example, the second activity test result may be a result of determining the activity of the test object (as a non-limiting example, determining whether the test object is a living object) based on the second output value output from the second activity test model. As a non-limiting example, the second activity test result may be one of a result value indicating that the test object is a living object and a result value indicating that the test object is an inanimate object.
[0077] For another example, the second activity test result may be a second score based on a second output value output from the second activity test model. As a non-limiting example, the second score may be equal to the second output value or a value obtained by a predefined calculation including the second output value as an element.
[0078] In operation 240 , the liveness test apparatus determines whether the test object is a living object based on the first liveness test result and the second liveness test result.
[0079] In one example, in the case where the first liveness test result and the second liveness test result are result values indicating whether the test object is a living object, the liveness test device may finally determine that the test object is a living object in response to determining that the test object is a living object based on both the first liveness test result and the second liveness test result. Conversely, the liveness test device may finally determine that the test object is an inanimate object in response to determining that the test object is an inanimate object based on at least one of the first liveness test result and the second liveness test result.
[0080] In another example, in the case where the first activity test result and the second activity test result are the first score and the second score, respectively, the activity test device may determine that the test object is a living object in response to a combined value obtained based on the first score and the second score being greater than a threshold value. Conversely, the activity test device may determine that the test object is an inanimate object in response to the combined value not being greater than the threshold value. Here, as a non-limiting example, the combined value may be determined by the sum, average, and weighted sum of the first score and the second score. However, the examples are not limited to the examples described above, and thus the combined value may be determined by various methods.
[0081] The liveness testing device may perform a control operation in response to the result of the liveness test. In one example, in response to a final determination that the test object is a living object, the liveness testing device may generate a control signal to request the execution of user verification. Conversely, in response to a final determination that the test object is not a living object, the liveness testing device may generate a control signal to block the user's access without requesting the execution of user verification.
[0082] exist Figure 2AIn the example shown in , a color image and an IR image can be used for activity testing, and the ROI of the color image to be used for the activity testing and the ROI of the IR image to be used for the activity testing may correspond to the same test area or different test areas of the test object. As a non-limiting example, in the case where the test areas of the ROI to be used for the activity testing in the color image and the IR image are the same, the activity testing device may detect a face area or an iris area in both the color image and the IR image, and perform the activity test based on the detected face area or the detected iris area. However, the test areas of the ROI in the color image and the IR image may be different from each other, which will be referred to below. Figure 2B Describe the situation in detail.
[0083] Figure 2B is a flow chart illustrating another example of an activity testing method.
[0084] Reference Figure 2B In operation 252, the liveness test device receives a color image including a face. In operation 262, the liveness test device detects a first ROI including a face region in the color image. Here, as a non-limiting example, the liveness test device may use a haar-based cascaded adaboost classifier to detect the first ROI in the color image. For another example, the liveness test device may detect a facial landmark in the color image and determine the first ROI based on the position of the detected facial landmark.
[0085] In operation 272, the activity test device obtains a first activity test result by performing an activity test based on the first ROI. Here, except that the image data of the first ROI is input into the first activity test model, the first activity test result is obtained in the same manner as in the above reference. Figure 2A The operation 220 of obtaining the first activity test result is the same as described above. Therefore, for the sake of brevity and clarity, the detailed description and repeated description of operation 272 will be omitted here.
[0086] In operation 254, the liveness test device receives an IR image including an iris. In operation 264, the liveness test device detects a second ROI including an iris region in the IR image. The liveness test device may detect facial landmarks in the IR image and detect an eye region based on the detected facial landmarks. The liveness test device may detect an iris region in the detected eye region and set a second ROI including the iris region.
[0087] In operation 274, the activity test device obtains a second activity test result by performing an activity test based on the second ROI. Here, except that the image data of the second ROI is input into the second activity test model, the second activity test result is obtained in the same manner as in the above reference. Figure 2AThe operation 230 of obtaining the second activity test result is the same as described above. Therefore, for the sake of brevity and clarity, a detailed and repeated description of operation 274 will be omitted here.
[0088] In operation 280, the activity test device determines whether the test object is a living object based on the first activity test result and the second activity test result. For a detailed description of operation 280, reference may be made to the above reference. Figure 2A Operation 240 is described.
[0089] exist Figure 2B In the example shown in , the first ROI and the second ROI correspond to different test areas of the test object. As shown, the first ROI is the face area and the second ROI is the iris area. However, the examples are not limited to the examples described above, and the types of ROIs to be detected in the color image and the IR image are not limited to the examples shown. As a non-limiting example, the iris area detected in the color image and the face area detected in the IR image can be used for liveness detection. For another example, the entire face area detected in the color image and the partial face area detected in the IR image can also be used for liveness testing.
[0090] Figure 3 is a flow chart illustrating an example of an activity testing method.
[0091] Reference Figure 3 In operation 310, the activity test device receives the color image and the IR image. As a non-limiting example, for a detailed description of operation 310, reference may be made to the above reference. Figure 2A Operation 210 is described.
[0092] In operation 320, the activity test device generates test data for the activity test to be performed on the test object based on the color image and the IR image. In one example, the activity test device may generate the test data by combining the color image and the IR image. As a non-limiting example, the activity test device may generate the test data by concatenating the color image and the IR image. As another example, the activity test device may generate the test data by fusing the color image and the IR image.
[0093] In operation 330, the activity test device determines whether the test object is a living object by performing an activity test using the test data generated in operation 320. The activity test device may determine whether the test object is a living object based on the output result of the activity test model using the test data as input. In one example, the activity test model may correspond to a DCNN configured to use the test data as input and output a result value corresponding to the input test data. The DCNN may output a result value corresponding to the test data based on an internal parameter, and a score to be used to determine the result of the activity test may be calculated based on the output result value. As a non-limiting example, the score may be equal to the result value output from the activity test model, or a value obtained by a predefined calculation using the result value as an element. The activity test device may determine whether the test object is a living object based on the result of comparing the score with a preset threshold. For example, in response to the score being greater than the threshold, the activity test device may determine that the test object is a living object. On the contrary, in response to the score being not greater than the threshold, the activity test device may determine that the test object is an inanimate object. The activity test model may be based on a DCNN as described above, but the example is not limited thereto. Therefore, the activity test model may be based on a neural network different from the DCNN. Alternatively, the activity test model may be a model configured to extract feature values reflecting color features, reflection features, or texture features of the test object using an image feature extraction method known in the art to which the examples of the present disclosure described herein pertain, rather than being based on a neural network.
[0094] Figure 4 is a flow chart showing another example of an activity test method. As a non-limiting example, reference will be made to Figure 4 The activity test method described is to refer to Figure 2A Examples and references described Figure 3 Examples of example combinations described.
[0095] Reference Figure 4 In operation 410, the activity test device receives the color image and the IR image. For a detailed description of operation 410, please refer to the above reference. Figure 2A Operation 210 is described.
[0096] In operation 420, the activity test device performs an activity test using a color image to obtain a first activity test result. Here, the first activity test result is obtained as described above with reference to Figure 2A The operation 220 of obtaining the first activity test result is the same as described above, and therefore, for the sake of brevity and clarity, a more detailed and repeated description will be omitted here.
[0097] In operation 430, the activity test device performs an activity test using the IR image to obtain a second activity test result. Here, the second activity test result is obtained as described above with reference to Figure 2A The operation 230 of obtaining the second activity test result is the same as described above, and therefore, for the sake of brevity and clarity, a more detailed and repeated description will be omitted here.
[0098] In operation 440, the activity test device performs an activity test using the test data generated based on the color image and the IR image to obtain a third activity test result. For a detailed description of operation 440, reference may be made to the above reference. Figure 3 The operation 320 of generating test data is described. As a non-limiting example, the activity test device may generate test data by combining the color image and the IR image into a single image, and obtain a third activity test result using an activity test model that uses the generated test data as an input. For another example, each of the color image and the IR image may be input to the activity test model, and test data may be generated based on the color image and the IR image in an intermediate layer of the activity test model, and then a score to be used to determine the activity of the test object may be output by the activity test model.
[0099] In operation 450, the activity testing device determines whether the test object is a living object based on the first activity test result, the second activity test result, and the third activity test result. In one example, the activity testing device may determine whether the test object is a living object based on a first score obtained from the first activity test result, a second score obtained from the second activity test result, and a third score obtained from the third activity test result. The activity testing device may determine that the test object is a living object in response to a combined value obtained based on the first score, the second score, and the third score being greater than a threshold value, and determine that the test object is an inanimate object in response to the combined value being less than or equal to a threshold value. Here, as a non-limiting example, the combined value may be determined by the sum, average, or weighted sum of the first score, the second score, and the third score. However, the examples are not limited to the examples described above, and the combined value may be determined by various methods.
[0100] In another example, the activity testing device may determine whether the test object is a living object based on each of the first activity test result, the second activity test result, and the third activity test result. In this example, when the test object is determined to be a living object based on at least two of the first activity test result, the second activity test result, and the third activity test result, the activity testing device may finally determine that the test object is a living object in response to determining that the test object is a living object based on at least two of the three activity test results. According to one example, in response to determining that the test object is a living object based on at least one or all three of the three activity test results, the activity testing device may finally determine that the test object is a living object.
[0101] Figure 5A and Figure 5B is a diagram showing an example of how to obtain a color image and an IR image to be used for an activity test.
[0102] Reference Figure 5A , a color image and an IR image are obtained by the image sensor 515 and the image sensor 535, respectively. As a non-limiting example, in the case where the user 510 desires to capture an image of a face or iris for user verification, a color image for user verification may be obtained by the color image sensor 515, and an IR image for user verification may be obtained by the IR image sensor 535. The IR image may be obtained when IR light emitted from the IR light source 530 is reflected by the user 510 and then the reflected IR light is sensed by the IR image sensor 535. In one example, as shown, the color image sensor 515 includes an IR cutoff filter 520 for blocking IR light from entering the color image sensor 515 and a color pixel array 525 for sensing red, green, and blue (RGB) color values. The IR image sensor 535 includes an IR pass filter 540 for transmitting IR light incident on the IR image sensor 535 and an IR pixel array 545 for sensing IR values. In Figure 5A In the example shown in , a portion of the entire color pixel array is shown as color pixel array 525 , and a portion of the entire IR pixel array is shown as IR pixel array 545 .
[0103] When it is necessary to perform a liveness test on user 510 , liveness test device 500 may receive a color image and an IR image from color image sensor 515 and IR image sensor 535 , respectively, and perform a liveness test using the received color image and the received IR image.
[0104] Reference Figure 5B, a color image and an IR image are obtained by a single integrated image sensor 550. When a color image and an IR image of a test object are needed to perform an activity test on the test object, the activity test device 500 activates the IR light source 530 and receives image information sensed by the integrated image sensor 550. In one example, as shown, the integrated image sensor 550 includes an optical filter and a pixel array 570, wherein the optical filter includes an IR cutoff region 555 that blocks IR light incident on the integrated image sensor 550 and an IR pass region 560 that transmits IR light, and the pixel array 570 is configured to sense color values and IR values. Figure 5B In the example shown in , only a portion of the entire pixel array is shown as pixel array 570 , and integrated image sensor 550 may include a pixel array in which the pixel structure shown in pixel array 570 is repeatedly arranged.
[0105] The integrated image sensor 550 measures both the color value and the IR value through the pixel array 570. The pixel array 570 includes a first light receiver for measuring a red (R) color value, a second light receiver for measuring a green (G) color value, a third light receiver for measuring a blue (B) color value, and a fourth light receiver for measuring an IR value. The activity test device 500 may separate the color value and the IR value measured by the integrated image sensor 550 from each other, and may generate a color image and an IR image based on the separation results. Then, the activity test device 500 may perform an activity test based on the generated color image and the generated IR image.
[0106] Figure 6 is a diagram illustrating an example of a pixel array of an integrated image sensor in which a color sensor and an IR sensor are combined.
[0107] Figure 6 FIG. 1 shows a pixel array of an integrated image sensor configured to obtain both a color image and an IR image. The pixel array may correspond to Figure 5B Pixel array 570 is shown in FIG. Figure 6, the pixel array includes a plurality of pixel units 610, and each of the plurality of pixel units 610 includes a color pixel 622, a color pixel 624, and a color pixel 626 for sensing RGB color values and an IR pixel 630 for sensing IR values. The pixel array may be provided in a structure in which each of the plurality of pixel units 610 is repeatedly arranged. Each of the color pixel 622, the color pixel 624, and the color pixel 626 includes a light receiver for sensing a color value corresponding to each of the color pixel 622, the color pixel 624, and the color pixel 626, and the IR pixel 630 includes a light receiver for sensing an IR value. The arrangement of the color pixel 622, the color pixel 624, the color pixel 626, and the IR pixel 630 of each pixel unit is not limited to the example shown, and various modifications or changes may be made to the example arrangement. Using such an integrated image sensor can reduce the difference between the position or capture angle at which a color image and an IR image are obtained.
[0108] FIG. 7A to FIG. 7D is a diagram illustrating an example of an activity test method.
[0109] Fig. 7A An example showing how liveness testing is performed during iris authentication. Fig. 7A , for the liveness test, the color image 710 including the iris area is input to the first liveness test model 720, and the IR image 715 including the iris area is input to the second liveness test model 725. Here, before the color image 710 and the IR image 715 are respectively input to the first liveness test model 720 and the second liveness test model 725, various image preprocessing methods may be performed, such as, as a non-limiting example, image cropping, ROI extraction, and image resolution conversion. The first liveness test model 720 outputs a first liveness test result corresponding to the color image 710. As a non-limiting example, the first liveness test model 720 may output a first score corresponding to the color image 710 through a calculation or operation determined by an internal parameter of the neural network based on the image data (as a non-limiting example, color value and brightness value) of the input color image 710. The second liveness test model 725 outputs a second liveness test result corresponding to the IR image 715. As a non-limiting example, the second activity test model 725 may output a second score corresponding to the IR image 715 based on image data (as a non-limiting example, IR value) of the input IR image 720 through a calculation or operation determined by internal parameters of the neural network.
[0110] In operation 730, the activity test device determines the result of the activity test based on the first score output from the first activity test model 720 and the second score output from the second activity test model 725. As a non-limiting example, in response to each of the first score and the second score being greater than a preset threshold, or in response to a combined value obtained based on the first score and the second score being greater than a preset threshold, the activity test device may determine that the test object of the activity test is a living object. The example of determining the result of the activity test based on the first score and the second score is not limited to the example described above, and thus the result of the activity test may be determined by various methods.
[0111] Figure 7B An example of how liveness testing is performed during face verification is shown. Figure 7B For the activity test, the color image 740 including the face area is input to the first activity test model 750, and the IR image 745 including the face area is input to the second activity test model 755. Fig. 7A Similar to the example shown in , in operation 760 , the activity test apparatus determines a result of the activity test based on the first score output from the first activity test model 750 and the second score output from the second activity test model 755 .
[0112] Figure 7C An example showing how to perform liveness testing using a single integration test model. Figure 7C , Fig. 7A The first activity test model 720 and the second activity test model 725 shown in FIG7 are replaced with a single integrated activity test model 770. The color image 710 and the IR image 715 are respectively input to the activity test model 770, and the activity test model 770 outputs a score through a calculation or operation operation performed based on the internal parameters of the activity test model 770. Here, the features of the color image 710 and the features of the IR image 715 may be combined in an intermediate layer of the activity test model 770, and a score may be calculated based on the result of the combination. In operation 780, the activity test device determines the result of the activity test based on the score. As a non-limiting example, in response to the score satisfying a preset criterion, the activity test device may determine that the test object is a living object.
[0113] Fig.7D An example showing how to perform activity testing using different types of ROIs in color images and IR images for activity testing. Fig.7D , the face area in the color image 712 and the iris area or eye area in the IR image 714 are used.
[0114] For the activity test, a first ROI 722 including a face area is detected in the color image 712, and the detected first ROI 722 is input to a first activity test model 762. The first activity test model 762 outputs a first activity test result corresponding to the first ROI 722 by a calculation or operation operation performed based on internal parameters of the first activity test model 762. In addition, for the activity test, a second ROI 724 including an iris area or an eye area is detected in the IR image 714, and the detected second ROI 724 is input to a second activity test model 764. The second activity test model 764 outputs a second activity test result corresponding to the second ROI 724 by a calculation or operation operation performed based on internal parameters of the second activity test model 764. In operation 790, the activity test device determines a final result of the activity test based on the first activity test result and the second activity test result.
[0115] Figure 8 is a diagram illustrating another example of an activity test method.
[0116] Reference Figure 8 , based on the color image 710 and the IR image 715, test data to be used for performing an activity test on the test object is generated. As a non-limiting example, the test data can be generated by combining, fusing, synthesizing or linking the color image 710 and the IR image 715 into a single set of data (such as a combined image, a fused image, a synthesized image or a linked image). As a non-limiting example, the relevant information of the color image 710 and the relevant information of the IR image 715 can be combined to create a single fused image with a higher spatial resolution than the spatial resolution of any one of the color image 710 and the IR image 715, while retaining the spectral information of the color image 710 and the IR image 715. The generated test data is input to the activity test model 810, and the activity test model 810 outputs a result value corresponding to the test data through a calculation or operation performed based on the internal parameters of the activity test model 810. In operation 815, the activity test device determines the result of the activity test performed on the test object based on the result value. As a non-limiting example, in response to the result value or a score corresponding to the result value satisfying a preset criterion, the liveness testing device may determine that the test object is a living object.
[0117] Fig. 9 is a diagram illustrating another example of an activity test method.
[0118] Reference Fig. 9 ,and Fig. 7ASimilar to the example shown in FIG. 1 , the color image 710 and the IR image 715 are input to the first activity test model 910 and the second activity test model 920, respectively, and the first activity test model 910 and the second activity test model 920 output the first activity test result and the second activity test result, respectively. Figure 8 Similar to the example shown in , test data to be used to perform an activity test on the test object is generated based on the color image 710 and the IR image 715, and the generated test data is input to the third activity test model 930, and then the third activity test model 930 outputs a third activity test result.
[0119] In operation 940, the activity test device determines the final result of the activity test based on the first activity test result, the second activity test result, and the third activity test result. In one example, when the final score determined based on the first score obtained from the first activity test result, the second score obtained from the second activity test result, and the third score obtained from the third activity test result is greater than a threshold value, the activity test device may determine that the test object is a living object. In this example, the final score can be determined by a predefined calculation or operation. However, the calculation operation is not limited to a specific example. As a non-limiting example, the final score can be determined by the sum of the scores or the weighted sum of the scores. For another example, the final result of the activity test for the test object can be determined based on whether at least two of the first score, the second score, and the third score meet a predefined standard.
[0120] In another example, the liveness test device may use only some of the three liveness test results to determine the final result of the liveness test. As a non-limiting example, in the case where the type of user authentication is face authentication, the liveness test device may use only the first liveness test result and the third liveness test result to determine the final result of the liveness test. For another example, in the case where the type of user authentication is iris authentication, the liveness test device may use only the second liveness test result and the third liveness test result to determine the final result of the liveness test.
[0121] Fig.10 is a diagram illustrating an example of an active test device.
[0122] The activity test device 1000 can perform the activity test using the color image and the IR image obtained by capturing the same test object of the activity test, and provide the result of the activity test. The activity test device 1000 can perform one or more or all of the operations or stages described herein with respect to the activity test, such as the verification operation and the execution of further operations of the activity test device based on the result of the activity test and / or the result of the verification operation.
[0123] Reference Fig.10, the activity test device 1000 includes a processor 1010 and a memory 1020. The memory 1020 may be connected to the processor 1010 and store instructions executable by the processor 1010, data to be operated or calculated by the processor 1010, and / or data processed by the processor 1010. The memory 1020 may include a non-transitory computer-readable medium (as a non-limiting example, a high-speed random access memory (RAM)) and / or a non-volatile computer-readable storage medium (e.g., at least one disk storage device, a flash memory device, and other non-volatile solid-state memory devices).
[0124] The processor 1010 may execute the above reference Figures 1 to 9 As a non-limiting example, the processor 1010 may perform an activity test on the test object based on the color image and the IR image obtained by the image sensor, and output a result of the activity test.
[0125] In one example, as referenced Figure 2A and Figure 2B As described, the processor 1010 may perform an activity test using a color image to obtain a first activity test result, and perform an activity test using an IR image to obtain a second activity test result. The processor 1010 may then determine whether the test result is a living object based on the obtained first activity test result and the obtained second activity test result. In this example, a first activity test model based on a neural network using a color image as an input may be used to obtain a first activity test result, and a second activity test model based on a neural network using an IR image as an input may be used to obtain a second activity test result.
[0126] In another example, as shown in Figure 3 As described, the processor 1010 may generate test data for performing a liveness test on the test object based on the color image and the IR image, and determine whether the test object is a living object based on an output result of a liveness test model using the generated test data as input.
[0127] In another example, as shown in Figure 4 As described, in addition to the first and second activity test results, the processor 1010 may also obtain a third activity test result by performing an activity test based on the test data in which the color image and the IR image are combined. The processor 1010 may then determine whether the test object is a living object based on the first, second, and third activity test results.
[0128] In response to determining that the test object is a living object, liveness testing device 1000 may output notification data indicating that the liveness test is satisfied. Conversely, in response to determining that the test object is an inanimate object, liveness testing device 1000 may output notification data indicating that the liveness test is not satisfied.
[0129] Fig.11 is a diagram illustrating an example of a computing device.
[0130] The computing device 1100 may perform a liveness test on a test object included in an input image and perform user authentication. The computing device 1100 may correspond to Figure 1 The computing device 120 shown in FIG. 1 includes Fig.10 The active testing apparatus 1000 shown in, or is representative of such active testing apparatus described herein.
[0131] Reference Fig.11 , the computing device 1100 includes a processor 1110, a memory 1120, an image sensor 1130, a light source 1140, a storage device 1150, an input device 1160, an output device 1170, and a network interface 1180. The processor 1110, the memory 1120, the image sensor 1130, the light source 1140, the storage device 1150, the input device 1160, the output device 1170, and the network interface 1180 can communicate with each other via a communication bus.
[0132] The processor 1110 may perform functions and execute instructions to perform a liveness test using a color image and an IR image, and to perform verification and control or non-control of the computing device 1100 based on the results of the liveness test and / or the results of the liveness test and verification operation. As a non-limiting example, the processor 1110 may process instructions stored in the memory 1120 or the storage device 1150. The processor 1110 may execute the above-referenced Figures 1 to 10 One or more or all of the operations or phases described.
[0133] The memory 1120 may store information for performing a liveness test. The memory 1120 may include a computer-readable storage medium or a computer-readable storage device. The memory 1120 may store instructions to be executed by the processor 1110 and information required to perform a liveness test.
[0134] The image sensor 1130 may obtain a color image and an IR image of a test object including an activity test. In one example, the image sensor 1130 may include a Figure 5A In another example, the image sensor 1130 may be an image sensor for obtaining a color image and an image sensor for obtaining an IR image. Figure 5BAn integrated image sensor for obtaining both a color image and an IR image is shown in . In this example, the image sensor 1130 can measure both the color values of the color image and the IR values of the IR image, and includes a first light receiver for measuring the R color value, a second light receiver for measuring the G color value, a third light receiver for measuring the B color value, and a fourth light receiver for measuring the IR value.
[0135] The light source 1140 may emit IR light under the control of the processor 1110. The IR light emitted from the light source 1140 may be reflected by the test object, and the reflected IR light may be sensed by the image sensor 1130.
[0136] Storage device 1150 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1150 may store a larger amount of information than memory 1120 and store the information for a longer period of time. As non-limiting examples, storage device 1150 may include: a magnetic hard disk, an optical disk, a flash memory, an electrically erasable programmable read-only memory (EPROM), a floppy disk, and other types of non-volatile memory known in the art to which the examples described herein belong.
[0137] Input device 1160 may receive input from a user, as non-limiting examples, tactile input, video input, audio input, or touch input. As non-limiting examples, input device 1160 may detect input from a keyboard, a mouse, a touch screen, a microphone, and a user, and include other devices configured to transmit the detected input to computing device 1100.
[0138] The output device 1170 can provide the output of the computing device 1100 to the user through visual, auditory or tactile channels. As non-limiting examples, the output device 1170 may include a display, a touch screen, a speaker, a vibration generator, and other devices configured to provide output to the user. The network interface 870 can communicate with external devices through a wired or wireless network.
[0139] This is achieved through hardware components Figures 1 to 11The computing device 120, color camera 130, IR light source 135, IR camera 140, activity test device 500, color image sensor 515, IR cut filter 520, color pixel array 525, IR light source 530, IR image sensor 535, IR pass filter 540, IR pixel array 545, integrated image sensor 550, IR cut region 555, IR pass region 560, pixel array 570, pixel unit 610, color pixel 622, color pixel 624 and color pixel 626, IR pixel 630, first activity test model 720, second activity test model 725, first activity test 1000, processor 1010, memory 1020, computing device 1100, processor 1110, memory 1120, image sensor 1130, light source 1140, storage device 1150, input device 1160, output device 1170, network interface 1180 and other devices, units, modules, devices and other components. Examples of hardware components that can be used to perform the operations described in appropriate locations in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components for performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components may also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both.For example, a single hardware component, or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components. The hardware component may have any one or more different processing configurations, wherein examples of one or more different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.
[0140] Figures 1 to 11 The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as described above to execute instructions or software for performing the operations performed by the method described in the present application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.
[0141] Instructions or software for controlling a computing device (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or a special-purpose computer to perform the operations performed by the hardware components and the methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein disclose algorithms for performing the operations performed by the hardware components and the methods described above.
[0142] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device that is configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors and computers can execute instructions. In one example, the instructions or software and any associated data, data files, and data results are distributed across a networked computer system so that the instructions and software and any associated data, data files, and data results are stored, accessed, and executed in a distributed manner by one or more processors or computers.
[0143] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered in a descriptive sense only, and not for the purpose of limitation. The description of the features or aspects in each example is considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, structures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, appropriate results can be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.
Claims
1. An activity test method, include: obtaining a color image including the object and an infrared (IR) image including the object using a single integrated image sensor; performing a first liveness test using the color image; performing a second activity test using the IR image; A fused image is generated by fusing the IR image and the color image; performing a third liveness test using the fused image; determining an activity of the subject based on a result of the first activity test, a result of the second activity test, and a result of the third activity test, The color image includes the face area of the object, the IR image includes the iris area of the object, in: The steps of performing a first liveness test include: determining whether the color image has captured an IR representation of the object; The step of performing a second liveness test comprises: determining whether the IR image has captured a color representation of the object, wherein the single integrated image sensor comprises a pixel array configured to obtain both color values and IR values, The step of obtaining a color image including the object and an infrared (IR) image including the object using a single integrated image sensor comprises: Separate the color value from the IR value, and A color image and an IR image are generated based on the separated results.
2. The activity testing method according to claim 1, further comprising: include: Verify that the object corresponds to an authorized user; The device is switched from a locked state to an unlocked state based on the result of the verification.
3. The activity testing method according to claim 2, in, The verifying step and the switching step are selectively performed according to the result of determining the activity of the object.
4. The activity testing method according to claim 1, in, The step of performing the first liveness test is implemented using a neural network based first liveness test model using a color image as input.
5. The activity testing method according to claim 1, in, The step of performing the second liveness test is implemented using a neural network based second liveness test model using the IR image as input.
6. The activity testing method according to claim 1, in, The steps for determining the activity of an object include: In response to at least one of a result of the first liveness test, a result of the second liveness test, and a result of the third liveness test indicating that the object is alive, it is determined that the object is alive.
7. The activity testing method according to claim 1, in, The steps for determining the activity of an object include: Whether the object is alive is determined based on a first score obtained from the results of the first liveness test, a second score obtained from the results of the second liveness test, and a third score obtained from the results of the third liveness test.
8. The activity testing method according to claim 7, in, The step of determining the activity of the object based on the first score, the second score, and the third score includes determining that the object is alive in response to a combined value obtained based on the first score, the second score, and the third score being greater than a threshold value.
9. The activity testing method according to claim 2, in, The single integrated image sensor comprises: One or more color light receivers configured to measure any one of a red R color value, a green G color value, and a blue B color value of a color image, or any combination of any two or more thereof; One or more IR light receivers are configured to measure IR values of the IR image.
10. The activity testing method according to claim 1, further comprising: include: detecting a first region of interest ROI of an object in the color image, wherein the step of performing a first activity test comprises performing a first activity test based on the detected first ROI; A second ROI of the object in the IR image is detected, wherein the step of performing a second activity test includes performing a second activity test based on the detected second ROI.
11. The activity testing method according to claim 10, in, The first ROI and the second ROI correspond to different regions of the object.
12. The activity testing method according to claim 1, in, The IR representation of the object is a material IR image of the object, and the color representation of the object is a material color image of the object.
13. An activity testing method, include: performing a first liveness test using a color image including the object; performing a second activity test using an infrared (IR) image including the object; Combine the color image and the IR image into a single image; performing a third liveness test using the single image; determining an activity of the subject based on a result of the first activity test, a result of the second activity test, and a result of the third activity test, The color image includes the face area of the object, the IR image includes the iris area of the object, in: The steps of performing a first liveness test include: determining whether the color image has captured an IR representation of the object; The step of performing a second liveness test comprises: determining whether the IR image has captured a color representation of the object, Wherein, the activity testing method further comprises: Separate the color and IR values obtained from the pixel array, and A color image and an IR image are generated based on the separated results.
14. The activity testing method according to claim 13, in, The step of determining the liveness of the object includes determining whether the object is alive based on a first score obtained from the results of the first liveness test, a second score obtained from the results of the second liveness test, and a third score obtained from the results of the third liveness test.
15. A non-transitory computer-readable storage medium storing instructions, in, When the instructions are executed by one or more processors, the one or more processors are caused to perform the activity testing method of claim 1 .
16. An activity testing device, include: a single integrated image sensor configured to obtain a color image including the object and an infrared (IR) image including the object; One or more processors configured to: performing a first liveness test using the color image; performing a second activity test using the IR image; performing a third activity test based on a combination of the color image and the IR image; determining an activity of the subject based on a result of the first activity test, a result of the second activity test, and a result of the third activity test, The color image includes the face area of the object, and the IR image includes the iris area of the object, wherein: The process of performing a first liveness test includes: determining whether the color image has captured an IR representation of the object; The process of performing the second liveness test includes determining whether the IR image has captured a color representation of the object, wherein the single integrated image sensor comprises a pixel array configured to obtain both color values and IR values, Wherein, the integrated image sensor is configured as: Separate the color value from the IR value, and A color image and an IR image are generated based on the separated results.
17. The activity test device according to claim 16, in, The one or more processors are further configured to: performing a first liveness test using a neural network-based first liveness test model using the color image as input; The second liveness test is performed using a neural network based second liveness test model using the IR image as input.
18. The activity test device according to claim 16, in, The one or more processors are further configured to: Detecting a first region of interest ROI of an object in a color image; performing a first activity test based on the detected first ROI; detecting a second ROI of an object in the IR image; performing a second activity test based on the detected second ROI, The first ROI and the second ROI correspond to different regions of the object.
19. The activity test device according to claim 16, in, The single integrated image sensor is configured to measure color values of the color image and IR values of the IR image.
20. The device according to claim 19, in, The single integrated image sensor comprises: One or more color light receivers configured to measure any one of a red R color value, a green G color value, and a blue B color value of a color image, or any combination of any two or more thereof; One or more IR light receivers are configured to measure IR values of the IR image.
21. The activity test device according to claim 16, further comprising: include: A memory stores instructions, wherein the instructions, when executed by the one or more processors, configure the one or more processors to perform a first liveness test and a second liveness test and a determination of liveness.
22. An activity testing method, include: performing a first activity test using one or both of an infrared (IR) image including the detected object and a color image including the detected object; A fused image is generated by fusing the IR image and the color image; performing a second liveness test using the fused image; determining an activity of the subject based on a result of the first activity test and a result of the second activity test, The color image includes the face area of the object, the IR image includes the iris area of the object, The step of performing the first activity test includes: determining whether the color image has captured an IR representation of the object; and / or determining whether the IR image has captured a color representation of the object, Wherein, the activity testing method further comprises: Separate the color and IR values obtained from the pixel array, and A color image and an IR image are generated based on the separated results.
