Liveness detection using a device including a light source

Abstract

A computer-implemented method for identifying a user is provided using a computing device comprising a light source that emits visible light when activated, the method comprising the steps of: capturing two images of a scene potentially containing a living body part having biometric characteristics—a first image captured when the light source is not activated and a second image captured when the light source is activated—transmitting the first image and the second image to a neural network, and processing the first image and the second image by the neural network—the processing step comprises comparing the first image and the second image and determining accordingly whether the first image and the second image are images of a living body part—the method further comprises the step of performing an identification algorithm to find biometric characteristics for identifying a user when it is determined that the first image and the second image are images of a living body part, and not performing the identification algorithm when it is determined that the first image and the second image are not images of a living body part.

Description

Technology Field

[0001] The present invention relates to a computer-implemented method for user identification according to claim 1. Background Technology

[0002] Smartphones and other mobile computing devices are increasingly becoming a part of daily life. In this regard, for example, logging into a bank account or making payments at a restaurant using a smartphone is common these days.

[0003] To enable users to log in to such accounts more easily, application EP3540835 provides a means for identifying a user by identifying the user, for example, using a camera of a mobile computing device, and by processing each acquired image to extract biometric characteristics such as fingerprints.

[0004] While biometric characteristics offer significant security advantages because they are very difficult to forge, it is necessary to ensure that the camera is not deceived by a malicious attacker placing an image of each biometric characteristic (e.g., a fingertip) in front of it and taking a picture in order to gain access to sensitive information, such as bank accounts. The problem to be solved

[0005] With regard to the disclosed prior art, the object of the present invention is to provide a method for distinguishing between a living object and a spoof of said object. means of solving the problem

[0006] This problem is solved by a computer implementation method according to independent claim 1. Preferred embodiments are provided in the dependent claims.

[0007] A computer-implemented method for identifying a user is provided using a computing device comprising a light source that emits visible light when activated, the method comprising the steps of: capturing two images of a scene potentially containing a living body part having biometric characteristics—a first image captured when the light source is not activated and a second image captured when the light source is activated—transmitting the first image and the second image to a neural network, and processing the first image and the second image by the neural network—the processing step comprises comparing the first image and the second image and determining accordingly whether the first image and the second image are images of a living body part—the method further comprises the step of performing an identification algorithm to find biometric characteristics for identifying a user when it is determined that the first image and the second image are images of a living body part, and not performing the identification algorithm when it is determined that the first image and the second image are not images of a living body part.

[0008] In the sense of the present invention, a light source may be considered, for example, as a torch light of a smartphone. However, the present invention is not limited thereto, and a light source that is part of a corresponding computing device is considered suitable as long as it can be selectively activated and emit visible light.

[0009] In this context, it is clear that the second image may not only be taken with the light source activated, but the scene (especially the object) in which the second image was taken may also be illuminated by the light source.

[0010] The neural network may preferably be any neural network trained to perform each task according to the present invention. In this regard, embodiments in which the neural network is a convolutional neural network specifically adapted to compare each image may be preferred. This means that the neural network has been correspondingly trained and that the network structure and architecture (e.g., number of layers) have been adapted to this task.

[0011] The identification algorithm may be any algorithm adapted to process image information capable of separating or at least identifying biometric features (regardless of whether this image information was taken from a first image or a second image). For example, a cropping algorithm that separates a region of interest from an image (e.g., a fingertip portion or region from another part of the image) may be used here.

[0012] The present invention has discovered that an image taken from a living body part, such as a finger, when a light source is not activated can be distinguished from a corresponding image taken from the same body part when an additional light source, such as a smartphone torch, is activated. This is because different parts of visible light (i.e., different wavelengths) are reflected differently from the living body part. This means, for example, that red light is emitted differently compared to blue light. By comparing these images, this difference can be determined, and if no corresponding difference is observed, it is highly likely that the two images are spoofed, such as when an image of an actual living body part is placed in front of the camera during image capture. Therefore, using this method, it is possible to reliably distinguish between spoofs of objects and living objects without requiring additional components that are generally unavailable to smartphone users.

[0013] In another embodiment, a computer-implemented method for identifying a user is provided using a device comprising a light source that emits visible light when activated, the method comprising the steps of: capturing two images of a scene potentially containing a living body part having biometric characteristics—a first image is captured with the light source not activated and a second image is captured with the light source activated—acquiring a difference image from the first image and the second image; transmitting the difference image to a neural network; processing the difference image by the neural network—the processing step includes determining brightness difference information from the difference image and determining from the brightness difference information whether the first image and the second image are images of a living body part—and, if the first image and the second image are determined to be images of a living body part, performing an identification algorithm to find biometric characteristics for identifying the user, and if the first image and the second image are determined not to be images of a living body part, not performing the identification algorithm.

[0014] The difference image may preferably be considered as a pixel-wise difference image of a first image and a second image, which preferably means that brightness values ​​and / or color values ​​of each pixel are taken from the first image and the second image and subtracted from each other, thereby obtaining new pixel values ​​for color and brightness for each pixel of the first image and the second image. This "new" image is considered as a difference image in the context of the present invention.

[0015] As previously explained above, there is a difference in the light reflection characteristics when the torch is turned on or off for a living body part compared to a spoof of a living body part, such as an image placed in front of a camera. From each processing using a neural network, the difference image can determine whether the first and second images represent evidence taken from a living body part or evidence taken from a spoof.

[0016] This method has the advantage of requiring only a single image containing each piece of information to be processed, without the need to process both images in a neural network to perform the comparison.

[0017] In one embodiment, the step of comparing a first image and a second image includes the step of comparing the images with respect to the pixel brightness distribution of the images. Since the brightness values ​​obtained for the first image and the second image differ significantly for living body parts, this can increase the reliability of distinguishing between spoofs and living objects.

[0018] Additionally, the step of comparing images with respect to the pixel brightness distribution of the images includes the step of obtaining red, blue, and green intermediate brightness values ​​of the pixels of the first image and the pixels of the second image, and the step of determining whether the intermediate brightness values ​​of the second image exceed the intermediate brightness values ​​of the pixels of the first image by a predetermined threshold value such that the images are images of living body parts.

[0019] This threshold can be a threshold that the neural network has been trained to identify and distinguish between spoofs and real (living) objects, for example, because the threshold can be set to an amount where it is almost certain that the object is a real object when the threshold is exceeded (or not exceeded).

[0020] In a more specific implementation, pixel brightness distributions for red, blue, and green pixels for a first image and a second image are obtained, and the pixel brightness distribution of each color of the obtained first image is compared with the pixel brightness distribution of the corresponding color of the obtained second image.

[0021] By separating the brightness values ​​for each color and comparing them individually, the reliability of distinguishing between real objects and spoofs can be further increased.

[0022] Additionally, the step of processing images by a neural network may include, before comparison, using a neural network to identify a part of a first image and / or a second image that may include a body part, and a step of extracting pixel information from the first image and the second image corresponding to this part, and the comparison of the first and second images may include comparing only the pixel information extracted from the first image and the second image corresponding to the part.

[0023] This may likewise apply to the method according to the second embodiment described above, wherein the method of this embodiment is performed before calculating the difference image, that is, using a neural network, a portion of the first image and / or the second image that may contain a body part is processed, and pixel information of this portion is extracted from the first image and the second image to calculate the difference image. This reduces the amount of pixels to be processed, thereby reducing the required computing resources without negatively affecting the reliability of the method. In another embodiment, a lighting check step is performed before capturing the first image and the second image, the lighting check step includes a step of checking the current lighting conditions, and the method further includes a step of not performing other steps of the method if the brightness corresponding to the current lighting conditions exceeds a given threshold.

[0024] Through this, it can be ensured that liveliness detection provides reliable results even in terms of environmental conditions. Therefore, false positive or false negative identifications can be prevented.

[0025] More specifically, if the brightness corresponding to the current lighting conditions exceeds a given threshold, information can be presented to the user that identification cannot be performed under the current lighting conditions.

[0026] This information may, for example, indicate to the user that other means must be used to distinguish between actual objects and spoofs of such objects, or simply indicate that neither liveness detection nor user identification using biometric features obtained from the first and / or second images can be performed. While the first alternative still allows identification to be performed using other methods, the second alternative ensures that false identification by a malicious user using a spoof of a living body part is impossible.

[0027] In addition, each step of this method is performed on a mobile computing device, and the neural network may be fully installed on the mobile device.

[0028] This means that not only are images captured on a mobile computing device, but additional processing by a neural network and each subsequent step are also performed on the mobile device. Accordingly, for example, a permanent connection to the mobile internet is not required, which can be particularly advantageous in regions of countries where such mobile internet is not always available.

[0029] In an additional embodiment, the neural network comprises three sub-networks, wherein the first sub-network processes a first image to extract first information from the first image, the second sub-network processes a second image to extract second information from the second image, and the third sub-network compares the first information and the second information to determine whether the first image and the second image are images of living body parts. Through this separation into first, second, and third sub-networks, a neural network (in the form of a sub-network) can be specifically used for each task of processing images and obtaining a determination as to whether the captured images represent actual objects or their spoofs.

[0030] Additionally, in one embodiment, the step of determining whether the first image and the second image are images of living body parts includes determining whether the first image and the second image are images of images on paper and / or whether the first image and the second image are images of images displayed on a screen and / or whether the first image and the second image are images of 3D-spoofs of living body parts and / or whether the first image and the second image are images of living body parts.

[0031] For example, presenting images on paper to a camera exhibits very specific characteristics depending on whether they are illuminated or unilluminated, making it possible for a neural network to make such a decision. Consequently, it can determine not only whether the image represents a real object or a spoof of a real object, but also what kind of spoof it actually is.

[0032] In one embodiment of the second embodiment, the brightness difference is obtained from pixel-wise subtraction of the brightness values ​​of the first image and the second image when the difference image is obtained.

[0033] This pixel-by-pixel subtraction may be performed through color-by-color and pixel-by-pixel subtraction, which means that for each pixel and each color value (green, blue, and red), a brightness value is obtained and pixel-by-pixel subtraction is performed. Accordingly, specific and accurate information regarding the actual characteristics of the first image and the second image is obtained, and this accurate information is used in the difference image.

[0034] In one embodiment, the light source is a torch of a computing device.

[0035] Since such torch-shaped light sources are provided in modern mobile computing devices such as laptops, tablet PCs, and smartphones, there is no need to use additional devices to perform the methods according to the present invention, and accordingly, the user can apply each method more easily. Brief explanation of the drawing

[0036] Figure 1 shows a schematic diagram of image capture and corresponding brightness contributions. FIG. 2 shows a flowchart of a method according to one embodiment of the present invention. FIG. 3 shows a schematic diagram of a method in which a neural network processes an input according to one embodiment. Figure 4 shows an example of neural network training. FIG. 5 shows an exemplary mobile device for carrying out the method of the present invention. Specific details for implementing the invention

[0037] FIG. 1 illustrates a schematic diagram of capturing two images of an object. FIG. 1 provides a mobile computing device (101), such as a smartphone, comprising an optical sensor (not otherwise shown) for capturing images of scenes, such as, for example, a hand (102) shown herein. It is evident that such scenes may include some surrounding environments. However, for the convenience of explanation, they are not shown here.

[0038] When methods are used to identify a user by capturing images of their body parts (e.g., face, eyes, hands, or palms), for subsequent identification steps, it is necessary to determine whether the captured image is actually of a living body part or an image of a spoof of that body part. Such a spoof could, for example, be an image obtained from a real person by a malicious user and placed in front of the camera. If this image is of sufficient quality, it could potentially spoof the identification mechanisms currently used for biometric features; thereby, access to sensitive information, such as bank accounts, could be granted to an unauthorized user by using this spoof of a real object.

[0039] According to the present invention, this can be prevented by taking more than one image of a potential real object (unless it is known whether it is an image of the real object or merely a spoof image of the object) and processing these images by neural networks as described later.

[0040] To provide a more general context, the left side of FIG. 1 shows a mobile computing device, such as a smartphone (101), capturing a potential real object (102) (in this case, an image of a hand). Under general conditions, this produces a brightness distribution as illustrated in diagrams (103 and 104) in any case. This brightness distribution is not necessarily a brightness distribution obtainable under any potential lighting conditions and is merely an exemplary diagram representing the lighting conditions of an object in which an image is captured when the torch of the mobile computing device (or any other suitable light source) is not activated. From this alone, it cannot be determined whether the image is from a living object or from a spoof of a living object. Next, the right side of FIG. 1 shows the same scene in which an image is captured, this time with the light source of the mobile computing device (101) activated and the scene illuminated by light (111).

[0041] The diagrams (105 and 106) show the light distribution of different color channels (e.g., red, blue, and green) and the respective brightness values ​​for this situation.

[0042] It can be clearly seen that diagram (105) mirrors diagrams (103 and 104) and diagram (106) is clearly distinguishable from the original diagram (104).

[0043] The present invention has discovered that when the image is an image of a real object, that is, when the hand (102) is actually a real human hand, a significant difference is obtained in the brightness distribution for each color value of the image. In contrast, a spoof of such a real object, specifically a spoof in the form of an image of a real object placed in front of the camera in the scenes of FIG. 1, is represented by diagrams (103 and 105) where there is only a slight, though not a complete, difference between the image taken with the light source of the computing device not activated and the image taken with the light device activated.

[0044] This can be used to distinguish between situations where only the spoof of a real object is captured and situations where the real object is captured. However, since the diagrams (103 to 106) depend heavily on environmental conditions (backlight, background, distance of the object to the camera, etc.), they cannot be easily processed by a computing algorithm to make a corresponding decision as to whether the images represent images of real objects or spoofs of objects.

[0045] The present invention has discovered that a neural network specifically trained to process corresponding first and second images captured in a single scene is better suited to making each decision with high accuracy, that is, there are almost no false positive and false negative results.

[0046] FIG. 2 shows a schematic flowchart of a method according to the present invention for processing corresponding first and second images to determine whether the captured images are images of actual living body parts.

[0047] In FIG. 2, in the first steps (201 and 202), a first image and a second image are captured. For the present invention, it does not matter whether the first image is captured with the light source activated or whether the second image is acquired with the light source activated. What is important in the context of the present invention is that one of the images is captured with the light source activated and the other is captured with the light source deactivated. For example, it is assumed that the first image captured in step 201 is captured with the light source deactivated, and the second image captured in step 202 is captured with the light source activated. In this regard, it is evident that the first image and the second image are captured in sequence. To avoid negative effects on image evaluation due to changes in conditions, the time between step 101 and step 102 should preferably be very short. Preferably, the time between capturing the images is less than 1 second, preferably less than 0.5 seconds.

[0048] As already explained above, the light source can be a torchlight in modern mobile computing devices, such as smartphones and tablet PCs, but it could potentially be provided in laptops.

[0049] Next, the two images captured in steps 101 and 102 may be preprocessed in step 203. This preprocessing may include some normalization, or other processing steps such as reducing the number of original pixels captured in the first and second images to a specific number, such as 256x256 pixels, or separating parts of the images for further processing and discarding other parts of the images. However, step 203 is not mandatory, and the process described in FIG. 2 may be performed without applying any preprocessing to the first and second images. Additionally, it is possible to preprocess only one of the images, for example, the image captured with the light source activated or the other image.

[0050] In the following optional step (204), a lighting check may be performed. This lighting check may be performed using specific software that checks lighting conditions from captured images (or from a third image taken independently of the first and second images). Alternatively or additionally, specific hardware of a mobile device, such as a smartphone, may be used. If the mobile device includes a brightness sensor (e.g., as part of a camera), information obtained from this brightness sensor may be used to perform the lighting check described below. Information from the brightness sensor may be obtained without any captured images. Alternatively or additionally, the lighting check may be performed using the brightness sensor while the images are being taken or after the images have been taken. Additionally, the results of the lighting check may be improved by using a combination of obtaining information from the brightness sensor and performing the lighting check using software (e.g., based on the captured images).

[0051] This lighting check is performed after the images have been taken, but it may also be performed before the images are taken. The lighting check may include checking ambient brightness or environmental lighting conditions, for example, whether it is day or night, or checking whether the person actually holding or operating the mobile computing device to perform the method described in FIG. 2 is inside a building with artificial lighting, etc. As already explained above, the brightness distributions actually obtained (see FIG. 1) depend significantly on environmental conditions. If the ambient lighting conditions are as follows, that is, if two images are taken, one with the light source activated and the other with the light source deactivated, it is highly likely that a reasonable result will not be obtained to distinguish the brightness values, and (for example, in broad daylight) performing the method according to FIG. 2 may also not be reasonable because reliable results cannot be obtained. In such cases, it may be determined that the method cannot be reasonably applied after the lighting check (204) has been performed. In such cases, information may be provided to the user in step 205 indicating that the present method may be interrupted and / or that the lighting conditions are not suitable for reliably performing the present identification method. This may be determined, for example, based on the brightness values ​​of the current lighting conditions. If this brightness value potentially exceeds a preset threshold, said information may be transmitted to the user in step 205 and / or the present method may be interrupted.

[0052] If the lighting conditions and the corresponding brightness values ​​are below their respective thresholds, the present method may proceed to step 206, or if a lighting check (204) is performed before taking the first image and the second image, the present method may proceed to steps 201 and 202.

[0053] In the next step, the images are processed by a neural network to determine whether the first and second images are images of living body parts.

[0054] In one embodiment of the present invention, this comprises processing a first image and a second image by comparing the images with respect to their pixel brightness distributions through a neural network and obtaining from this a result indicating whether the images were taken from a real object. This is determined by the neural network because the neural network is specifically trained to make this decision. The pixel brightness contributions of the first image and the second image can be processed by the neural network, and during this processing, the neural network obtains a specific result by processing each of the images (or both simultaneously) through one or more layers. This result will be nearly identical for the first image and the second image taken when the images are taken from spoofs of a real object (see FIG. 1), and will be different as described in relation to FIG. 1 when the images are images of a real object.

[0055] However, since these differences are not always the same, a specially trained neural network (described in relation to Fig. 3) is required to provide specific and well-defined results depending on whether the images are images of actual objects.

[0056] Specifically, it may be considered that each image, namely the first image and the second image, is separated into their respective color values, red, blue, and green, and / or separated into brightness distributions of each red, blue, and green color. Through this, the median brightness value of each pixel for each color can be obtained for each image. From these median brightness values, it can be determined whether the image taken with the light source activated exceeds the median brightness values ​​of the pixels of the image taken with the light source deactivated. If it is determined that the median brightness value of the second image (the image taken with the light source activated) exceeds the median brightness value of the first image by a specific threshold indicating that the second image and the first image are images taken from a living body part, the neural network can determine that the first image and the second image are images of a real object rather than a spoof. However, if the intermediate brightness values ​​are not different by at least the aforementioned threshold, this determination may be negative, and the first image and the second image may be determined not to be images of living body parts.

[0057] More specifically, pixel brightness distributions for each of the red, blue, and green pixels can be obtained for the first image and the second image, and the obtained pixel brightness distributions for each of the colors of the first image can be compared with the pixel brightness distributions of the corresponding colors of the second image. This implies that, in practice, there are three comparison steps according to this embodiment of the present invention, wherein the red brightness distribution of the first image is compared with the red brightness distribution of the second image, the green brightness distribution of the first image is compared with the green brightness distribution of the second image, and the blue brightness distribution of the first image is compared with the blue brightness distribution of the second image. These become three fundamentally independent decisions that can lead to a more reliable assessment of whether the first image and the second image represent a living body part.

[0058] For this processing, it may also be considered to apply a step in which a neural network or other correspondingly adapted neural network is used, for example during preprocessing (203), to separate the parts of the first image and the second image that are most likely to include the body part of interest, for example, the fingertip from which the fingerprint is extracted for further identification as a biometric feature. In this case, the so-called "region of interest" ROI may be separated, for example, during the preprocessing step and then further processed by each neural network.

[0059] Although the above example has been described with reference to only a single neural network, implementations using more than one neural network to perform each step can also be considered.

[0060] Specifically, as described above, before actually making a decision by comparing the values ​​of the first image and the second image, information such as brightness distribution is obtained from each image. Brightness information can be obtained without using a neural network, but other information may be more difficult to obtain from the images and may require the use of a neural network. To this end, a first "sub-network" may be used to extract each piece of information from the first image, while a second "sub-network" may be used to obtain each piece of information from the second image. Of course, the sub-networks are also specially trained neural networks. More specifically, the first sub-network may be trained to obtain information from an image taken when the light source of the mobile computing device is not activated, whereas the second sub-network may be specially adapted and trained to obtain corresponding information from an image taken when the light source of the computing device is activated.

[0061] Generally, since neural networks are trained using exemplary images of scenes that need to be identified, it is highly likely that the parameters of each neural network (first sub-network and second sub-network) differ at least slightly, so the neural networks extract corresponding information more accurately from the trained images.

[0062] Next, a third sub-network (likewise a specially trained neural network) is provided so that information obtained from the first image and information obtained from the second image can be compared to make respective decisions, thereby determining whether the first image and the second image are images of actual living body parts.

[0063] These decisions may involve more sophisticated determinations, wherein it may be determined whether the first and second images are images of living body parts, or whether the first and second images are images on paper captured by a mobile computing device and / or whether the first and second images are images of images presented on a screen and / or whether the first and second images are images of 3-D spoofs of living body parts (such as silicone imitations). Since each spoof and actual living body part exhibits specific light reflection characteristics to which a neural network can be trained, these decisions are possible using a correspondingly trained neural network. In this case, the result of each decision is more complex in that the type of spoof is also identified, rather than being "binary" in the sense that the images are images of spoofs or images of living body parts.

[0064] In an alternative embodiment to the cases described above, in step 206, the first image and the second image are not actually processed by the neural network, and a difference image is obtained from the first image and the second image before the processing by the neural network is actually performed. This difference image may be obtained, for example, by pixel-wise subtraction of the color values ​​and / or brightness values ​​of the first image and the second image. This may be done, for example, by subtracting the red value of the first image from the red value of the second image and performing corresponding actions on the other color values ​​for each pixel of the images. Additionally, or alternatively, corresponding brightness values ​​may be subtracted for each pixel of the first image from the corresponding brightness values ​​of each pixel of the second image. This results in a "difference image" having the same size as the first image and the second image but showing only the differences in brightness and / or color values. Since these differences contain the most important information when trying to identify whether the first and second images represent images of actual objects, it is reasonable to make this decision before processing by the neural network is performed.

[0065] By doing so, in one embodiment of the present invention, brightness difference information is obtained from the difference image, and subsequently, by processing the brightness difference information by a neural network, it may be provided to determine from the brightness difference information whether the first image and the second image are images of living body parts. This determination is basically obtained by the same processing described above for an embodiment in which two images are compared with each other, but the actual comparison does not need to be performed by a neural network. This can have advantages regarding the processing power required of a computing device and the actual capabilities of a neural network, and accordingly, implementation becomes easier even in computing devices with limited available computing resources, particularly computing devices that cannot access external computing resources such as external servers via a mobile internet connection.

[0066] Accordingly, for this embodiment as well as for the above-described embodiment in which two images are compared with each other, all steps of each method may be provided to be performed on a mobile device, and this applies to the steps described below as well. By doing so, the method of the present invention may be advantageously applied to situations where the user of a mobile computing device cannot access external processing capabilities. However, not all steps of the method are performed on a mobile device; for example, only images may be actually captured on the mobile device and then transmitted to an external server to perform additional processing, and then only the result of this processing may be provided back to the mobile device, where, for example, the extraction of user identification or biometric characteristics may be performed.

[0067] Regardless of how step 206 is actually implemented or where the steps of the method are performed, the method proceeds to step 207, which actually determines whether the first image and the second image are images of living body parts. If the result is negative, the method may simply terminate at step 209, or the information may be transmitted to some person, e.g., a person identified by these biometric characteristics generally displayed on the first image and the second image, to provide that person with information that someone wants to use spoofs of their biometric characteristics to gain unauthorized access to sensitive information.

[0068] If it is determined that the first image and the second image represent images of actual body parts, the present method may proceed to extract biometric features from the images (if this has not already been done during preprocessing) (step 210). This may be done, for example, by separating a part of the image containing biometric features, so-called region of interest (ROI), from other parts of the image and further processing only this part of the image, for example, by extracting minutia from the fingertips or extracting specific information from the user's face or the iris of the user's eyes or the lines of the user's palms.

[0069] After extracting biometric features in step 210, the present method may proceed to an optional step 211 in which the biometric features extracted from the images are actually used. The use of such biometric features may include, for example, identifying a user who has been proven that the images used to provide the biometric features represent a real living body part of a user with a bank account or other sensitive and restricted access information.

[0070] FIG. 3 illustrates the internal processing of a received input in a layer (300) of a neural network that can be used to determine whether an object on two images is a living object or a spoof of a living object. According to one embodiment of the present invention, the network (also referred to below) is a convolutional neural network (CNN). However, other networks may be used, and the input processing for obtaining the output depends on the specific structure of the neural network used.

[0071] The aforementioned layer (300) may be a layer that receives the original input after some preprocessing, for example, according to the processing order of the original input through the neural network. The received input may be two images, a single image, or a difference image as described above.

[0072] In any case, the layer (300) receives an input (330) corresponding to the original images or difference images (or some preprocessed counterpart) in at least some way. This input is preferably provided in the form of at least one matrix having N×M dimensions, where N and M are integers greater than 0. The matrix may represent pixels of the image for, for example, at least one color value (e.g., red) and / or corresponding brightness values. Thus, the components of this matrix may have values ​​corresponding to the respective color (in the exemplary case, red) and / or brightness value of this particular pixel. As will be evident from the following, the input may not be identical to the original images or difference images, but may be a matrix P obtained from a matrix representing the original images or difference images through some processing or some preprocessing (e.g., the resolution reduction described above) through the preceding layers of the neural network.

[0073] However, for the convenience of discussion, the input (330) is assumed to correspond to an N×M matrix representing at least one of the originally acquired images, and each component of this N×M matrix corresponds to a color value (e.g., red) of a pixel in each image. It is simple to apply this instruction to other transformed matrices derived from the original N×M matrix and obtained by processing this matrix in layers of a neural network, or to cases where (individual) brightness values ​​are provided for each pixel of the matrix.

[0074] Now, according to the process illustrated in FIG. 3, the input (330) is received by the depth-wise convolution layer (311) for processing. Below, a relatively simple example of how the input matrix (330) can be processed by the depth-wise convolution layer will be provided. Here, a kernel K is used to calculate the inner product using the matrix. The kernel is executed on the matrix as a so-called "stride." In the following example, horizontal and vertical stride width values ​​of 1 are used, but any other value greater than 1 may be used as long as the stride widths are integers greater than 0. The size of the kernel K is 5×T, where S and T are integers and are smaller than N and M.

[0075] Additionally, it is assumed that only the original input matrix 1 (i.e., input matrix (330)) of size N×M is used to calculate the inner product using the kernel. However, it is also considered that an extension matrix Z can be used to calculate the inner product using the kernel. This extension matrix Z is obtained by "attaching" lines and rows to the original matrix I, above the first line and below the last line, and to the left of the first row and to the right of the last row.

[0076] This is called "padding." Padding is generally P in the line direction. w N lines are added and P in the low direction h It consists of adding rows. Number P w can be equal to S-1 and the number P h can be equal to T-1, and accordingly, every inner product calculated between Z and the kernel contains at least one component of the original matrix I. Therefore, the size of the resulting matrix Z is (N + 2P W )×(M + 2P h .... Considering this, the matrix Z has the following components:

[0077]

[0078] In this context, the size of the new matrix obtained by calculating all inner products and arranging them appropriately according to lines and rows is generally This becomes, and here W w and W h defines the stride widths in the line direction and the row direction, respectively. Only these paddings and these stride widths are allowed for a given kernel K of size S×T, which consequently become integers for the size of the new matrix. Additionally, the stride width W w and W h is preferably smaller than S and T, respectively, because otherwise, when calculating the new matrix, the kernel will be shifted on matrix I in such a way that some lines or rows of the original matrix are omitted.

[0079] For the convenience of discussion, it is assumed below that the original matrix I is not padded and that the stride widths for the horizontal and vertical strides are 1. Additionally, it is assumed that the kernel is a matrix of size S×S (i.e., a special case where S=T is assumed). Applying the descriptions provided below to arbitrary padding and stride widths, as well as arbitrary kernel sizes, is simple according to the teachings provided below.

[0080] In the depth-wise convolution layer (311), the received input matrix (330) is of size S×S (where S <N,M)를 가진 커널 K를 사용한 내적을 형성하는데 사용된다. 이 내적은 축소 매트릭스가 크기 S×S를 가지면서 원본 N×M 매트릭스의 일관된 성분들을 포함하는 원본 N×M 매트릭스의 각 축소 매트릭스에 대해 계산된다. 예를 들어, S=3을 고려하면, N×M 원본 매트릭스의 제 1 축소 매트릭스 R은 성분 i=1,2,3; j=1,2,3을 포함하고 이에 따라 축소 매트릭스 (N×M)s가 9개의 성분으로 구성되며, 커널 K를 사용한 내적이 계산되어 단일 숫자가 생성된다. 원본 N×M 매트릭스의 라인 방향에 있어서의 다음 축소 매트릭스는 i가 1만큼 증가된 매트릭스이며, 이에 따라 이 방향에 있어서의 다음 매트릭스는 원본 N×M 매트릭스의 성분들로 구성된다(여기서 i=2,3,4; j=1,2,3). 이 매트릭스는 커널과 함께 다음 내적을 계산하는데 사용될 수 있다. S=3인 S×S 매트릭스의 주어진 예는 하나의 예시일 뿐이며 다른 커널들이 사용될 수도 있다.

[0081] To calculate the next reduced matrix R of size (N×M)s in the row / column direction, the index j of the elements in the original N×M matrix is ​​increased by 1. This is performed up to the last reduced matrix in the line direction, where i = N - S + 1, N - S + 2, N - S + 3, for the case where S=3. For the rows, this is performed in a corresponding manner, where j = M - S + 1, M - S + 2, M - S + 3. By calculating these inner products, the new matrix P is calculated to have a size of (N - S + 1)×(M - S + 1). Element P ij It corresponds to each inner product calculated using the kernel K and the corresponding reduced matrix of the original N×M matrix. In reality, a matrix of this size is forwarded to the point-by-point convolution layer of the layer (300).

[0082] In the case according to the present invention, kernel K is composed of components obtained through a learning process in which a neural network is trained to attribute-identify intended objects, thereby determining whether a living body part is present in each image. The kernel K used in the layer (300) of the neural network does not necessarily have to be the same in size and components as the kernels used in other layers of the neural network. Additionally, the components of the kernel do not need to be identical to each other, but must consist of numbers greater than or equal to zero. These components may be considered to represent "weights" obtained through neural network learning.

[0083] The result of processing the matrix (330) in the depth-wise convolution layer is a matrix (331) having a size of (N - S + 1) × (M - S + 1) when the kernel is moved on the original N × M matrix with strides having a distance of Δi = 1 in the line direction and a distance of Δj = 1 in the row direction, as described above. However, when these strides have larger distances such as Δi = 2 or Δi = 3 (and potentially, when a corresponding operation is performed for the rows), the dimensions of the result (331) are changed to correspond to those described above.

[0084] In further processing, this result (331) is forwarded to the first batch normalizer (312) in the processing order illustrated by the arrow in FIG. 3 after the depth-wise convolution layer (311). These components are provided to unintentionally attenuate potentially related information by processing images through the network, particularly the kernel(s) used in the layer(s).

[0085] The batch normalizer attempts to normalize the received result matrix (331). This is achieved by calculating the sum for each component of the (N - S + 1) × (M - S + 1) matrix and then dividing this by the number of components of the (N - S + 1) × (M - S + 1) matrix. The average value V for (N - S + 1) × (M - S + 1) (hereinafter the corresponding component P) ij A matrix denoted by P) is given as follows:

[0086]

[0087] Here, n and m represent the number of lines and columns / rows of an N×M matrix or the number of lines and columns of matrix P. Component P ij are the components of matrix P, where the given component P ijis an element of the matrix at line i and column j.

[0088] The next batch normalizer is each component P of the original matrix. ij P' by subtracting the mean value V from ij = P ij - Calculate the reduced matrix P' by making it V. By doing so, the values ​​of the reduced matrix P' are normalized, thereby filtering out anomalies (very large or very low values) in one direction or the other.

[0089] The result (332) generated by the first batch normalizer (312) is still a matrix with a size of (N - S + 1) × (M - S + 1) because no further dimension reduction of the matrix has been performed yet (in the example provided in FIG. 4).

[0090] Next, the result (332) is provided to the first rectifying linear unit (313) following the first batch normalizer (312).

[0091] The rectified linear unit is a new matrix component Each component of the matrix (332) is further modified by calculating, where

[0092]

[0093] As a result, values ​​less than 0 are set to 0 after passing through the batch normalizer, so they no longer affect the additional processing of the depth-wise convolution layer described below. This means that, for example, color values ​​lower than the average value calculated in the batch normalizer are no longer considered, and only values ​​corresponding to at least the average value V affect the result of the next step in the calculation.

[0094] Accordingly, the result (333) output by the first rectified linear unit (313) is still a matrix of shape / size (N - S + 1) × (M - S + l), and this matrix is ​​forwarded to a pointwise convolutional layer (321).

[0095] This point-by-point convolution layer (321) produces a result (334). This result (334) is produced by the point-by-point convolution layer (321) by taking each component of the (N - S + 1) × (M - S + 1) matrix (333) and multiplying this component by a weight α. α is preferably a number greater than 0 in any case, and this number is the same for each component of the (N - S + 1) × (M - S + 1) matrix. Thus, the result (334) obtained from the point-by-point convolution layer (321) is a matrix of the same size (N - S + 1) × (M - S + 1) but with each component multiplied by a weight α.

[0096] Next, the result (334) is provided to the second batch normalizer (322), where it is normalized in the manner described for the first batch normalizer (312), and a normalized matrix P' of the same dimension is calculated as the result (335), and this matrix / result (335) is forwarded to the second rectifying linear unit (323), where the result / matrix To obtain (236), a rectification function is applied and forwarded to the next layer of the neural network, or if no other layer follows in the neural network, the result (336) is provided as output.

[0097] Next, through the processing results of the input images, it becomes possible to determine whether the input images represent images of living body parts (this is discovered during the processing of images or difference images due to the different behavior of living body parts in response to illumination with light from a light source), that is, whether they are spoofs of living body parts. Depending on how sophisticated the neural network used is, further distinction can be made regarding what kind of actual spoof was used.

[0098] In Figure 4, we will describe how a neural network can be properly trained so that it learns the weights of kernel K and the weights described in relation to Figure 3, and a pattern that can determine whether two images show actual living body parts or only spoofs of the same thing.

[0099] The method of FIG. 4 begins with the provision of training data (401). The training data may consist of multiple images of fingertips or multiple fingers depicted in a single image along with other objects. Additionally, the training data may include images of actual fingers / other living body parts and images representing spoofs of these living body parts, for example, images of fingers. Additionally, or alternatively, the training data may include differential images as discussed above for processing by a neural network.

[0100] Images can be multiplied using rotated copies, highlighted copies, darkened copies, enlarged copies, or modified copies from the same image introduced as training data.

[0101] Additionally, results (408) corresponding to the input are provided, where these results are the results that the neural network must obtain, and thus are used in learning to check the results of processing by the neural network by comparing them with actual results and to adjust the parameters of the neural network accordingly.

[0102] In the next step, a specific input image (402) is provided to the neural network in a training environment (in addition to the neural network, an optimizer (407) and a loss function calculator (406) are provided).

[0103] The input image (or image pair) is processed in the first round using a depth-wise convolution layer, a first batch normalizer, and a first rectified linear unit (403) (summarized as DCBR), and then passed to a point-wise convolution layer, a second batch normalizer, and a second rectified linear unit (summarized as PCBR), where these are processed according to the description given in FIG. 3. This means that the steps or sections (403 and 404) illustrated in FIG. 4 are executed in each section (403 and 404) using corresponding weights for the kernel K of the point-wise convolution layer (PC) and the depth-wise convolution layer (DC). The first and second batch normalizers and the rectified linear units (403 and 404) of the components operate in the manner described above in relation to FIG. 3.

[0104] Consequently, as shown in FIG. 3, an output is provided. This output is a vector It may take the form of, where b, c, and d may represent real numbers, 0 ≤ a, b, c, and d ≤ 1, and each of a, b, c, and d represents the possibility that the input represents an actual living body part (e.g., value a) or a spoof of such a living body part (values ​​b, c, and d, where each of b, c, and d represents a different type of spoof possibility). Instead of four of these values, more or fewer values ​​may be provided. Additionally, it may be provided that only one output value is generated, where the images represent an actual living body part (e.g., when greater than 1) and the images represent spoofs of a living body part (e.g., when less than 1).

[0105] Next, this result (405) is provided to a loss function and compared with the results (408) to identify the differences between the result (405) and the correct result (408). Then, this difference obtained by the loss function (406) is provided to an optimizer (407), and ultimately, the optimizer (407) modifies the weights (i.e., α) of each point-by-point convolution layer and each depth-by-depth convolution layer, and the components of the kernel K. This means that the weights (α) of the point-by-point convolution layer and the components of the kernel K of the depth-by-depth convolution layer are adjusted for all layers of the network at once or for each layer.

[0106] With these new values, the cycle is repeated for the very same image(s) and the resulting vector The result is provided to the loss function and compared with the correct result (408), and the result is provided to the optimizer (407) to modify the weights again.

[0107] This procedure, when compared with the result (408), the result vector and specifically, it is performed insofar as the difference between the decision made regarding whether the images represent living body parts or their spoofs exceeds a given threshold corresponding to the essentially intended identification accuracy.

[0108] Then, the next input image(s) (402) are taken from the training data (401), and the corresponding bounding boxes are provided to the loss function. Then, the above process is repeated again for a new image to obtain optimal weights for the point-wise convolution layer and the depth-wise convolution layer. This is repeated until a combination of specific weights exhibits appropriate identification accuracy for all input images. Then, the combination of obtained weights is output as the final weights (410).

[0109] Next, these final weights are introduced into an application that executes the method of the present invention on a mobile device.

[0110] Accordingly, in the concept of the present invention, since the neural network provided to the mobile device is already fully adapted to the identification of images of living body parts and spoofs of these living body parts, it can be used without the need for additional learning, thereby further reducing the computing resources required for the mobile devices.

[0111] Overall, by using the point-by-point convolution layers, depth-by-depth convolution layers, batch normalizers, and rectified linear units described above with reference to the drawings and embodiments, a small-sized application can be provided, allowing it to be used independently on a mobile device without accessing additional data sources such as the Internet. This makes it suitable for applications in environments where access to wireless networks is unavailable. Furthermore, while the processor power required to run this application is minimized, appropriate spoof detection results can be continuously produced to extract biometric characteristics of living body parts from verified (i.e., non-spoofed) images, which can then be used to perform user identification based on the biometric characteristics described above.

[0112] To provide a context in which the method of the present invention can be performed, FIG. 5 illustrates a mobile device in the form of a smartphone according to one embodiment of the present invention.

[0113] The mobile device (500) is implemented as a smartphone as currently known. It preferably includes an optical sensor (520) (e.g., a camera) on the rear of the mobile device on the opposite side of the side of the mobile device (500) provided with a display (530). According to the present invention, a light source (e.g., as part of the optical sensor (520)) that can be optionally enabled and disabled is provided.

[0114] The camera may be a camera having a resolution of 1MP, 2MP or higher, for example, an HD camera. A flashlight may be provided as a light source, but the invention is not limited thereto and any reasonable light source of such mobile device may be used. It may also be adjusted to capture real-time images at reduced resolution, and when the camera is activated, the display (530) may show a representation of what the camera actually “sees.” This may be, for example, a hand (510).

[0115] In one embodiment of the present invention, when the method of the present invention is performed on captured images, the determination result may be displayed on a display (not shown herein).

Claims

Claim 1 A computer-implemented method for identifying a user using a computing device comprising a light source that emits visible light when activated, comprising the steps of: capturing two images of a scene potentially containing a living body part having biometric characteristics—a first image captured when the light source is not activated, and a second image captured when the light source is activated—; transmitting the first image and the second image to a neural network; and processing the first image and the second image by the neural network—the processing step comprising comparing the first image and the second image and determining accordingly whether the first image and the second image are images of a living body part—the method further comprises the step of performing an identification algorithm to find biometric characteristics for identifying a user when it is determined that the first image and the second image are images of a living body part, and not performing the identification algorithm when it is determined that the first image and the second image are not images of a living body part, wherein the neural network comprises three sub-networks, and A method comprising: a first sub-network processing the first image to extract first information from the first image; a second sub-network processing the second image to extract second information from the second image; and a third sub-network comparing the first information and the second information to determine whether the first image and the second image are images of a living body part. Claim 2 A method according to claim 1, wherein the step of comparing the first image and the second image comprises the step of comparing the images with respect to the pixel brightness distribution of the images. Claim 3 A method according to claim 2, wherein the step of comparing the images with respect to the pixel brightness distribution of the images comprises: the step of obtaining red, blue, and green intermediate brightness values ​​of the pixels of the first image and the pixels of the second image; and the step of determining whether the intermediate brightness value of the second image exceeds the intermediate brightness values ​​of the pixels of the first image by a predetermined threshold value such that the images are images of living body parts. Claim 4 A method according to claim 3, wherein a pixel brightness distribution for the red, blue, and green pixels for the first image and the second image is obtained, and the pixel brightness distribution of each color of the obtained first image is compared with the pixel brightness distribution of the corresponding color of the obtained second image. Claim 5 A method according to claim 1, wherein the step of processing the images by the neural network comprises, before comparing the first image and the second image, using the neural network to identify a part of the first image and / or the second image that may include a body part, and extracting pixel information from the first image and the second image corresponding to said part, wherein comparing the first and second images comprises comparing only the pixel information extracted from the first image and the second image corresponding to said part. Claim 6 A method according to claim 1, wherein, prior to capturing the first image and the second image, a lighting check step is performed, the lighting check step includes a step of checking current lighting conditions, and the method further includes a step of not performing other steps of the method if the brightness corresponding to the current lighting conditions exceeds a given threshold. Claim 7 A method according to claim 6, which provides information to a user that identification cannot be performed under the current lighting conditions when the brightness corresponding to the current lighting conditions exceeds a given threshold. Claim 8 A method according to claim 1, wherein each step of the method is performed on a mobile computing device, and the neural network is fully installed on the mobile computing device. Claim 9 A method according to claim 1, wherein the step of determining whether the first image and the second image are images of living body parts comprises determining whether the first image and the second image are images of images on paper and / or whether the first image and the second image are images of images displayed on a screen and / or whether the first image and the second image are images of 3D spoofs of living body parts and / or whether the first image and the second image are images of living body parts. Claim 10 A method according to claim 1, wherein the light source is a torch of the computing device. Claim 11 delete Claim 12 delete Claim 13 delete

Images