Liveness detection method and apparatus using phase difference

Abstract

To provide a method and apparatus for detecting the liveness based on a phase difference.SOLUTION: In one embodiment, the method includes generating a first phase image based on first visual information of a first phase, generating a second phase image based on second visual information of a second phase, generating a minimum map based on a disparity between the first phase image and the second phase image, and detecting the liveness based on the minimum map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The following embodiments relate to a method and apparatus for liveness detection using phase difference. [Background technology]

[0002] Biometric authentication technology authenticates users using fingerprints, iris, voice, face, blood vessels, etc. The biometric characteristics used for authentication are unique to each individual, and have the advantages of being convenient to carry, being less susceptible to theft or imitation, and remaining unchanged throughout a person's lifetime. Facial recognition, one type of biometric authentication technology, determines whether a user is a legitimate user based on the face shown in a still image or video. Facial recognition technology has the advantage of being able to verify the identity of the person being authenticated without contact. Recently, due to its convenience and efficiency, facial recognition technology has been widely used in various application fields such as security systems, mobile authentication, and multimedia data search. Summary of the Invention [Problem to be solved by the invention]

[0003] SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for liveness detection using phase difference. [Means for solving the problem]

[0004] According to one embodiment, a liveness detection method includes the steps of generating a first phase image based on first visual information of a first phase detected by a first pixel group of an image sensor, generating a second phase image based on second visual information of a second phase detected by a second pixel group of the image sensor, generating a minimum map based on a disparity between the first phase image and the second phase image, and detecting liveness based on the minimum map.

[0005] The step of generating the minimum map may include the steps of: setting a first reference area in the first phase image; setting a second reference area corresponding to the first reference area in the second phase image; setting at least one shift area by shifting the second reference area by a reference shift value; generating a difference image based on a difference between an image of the first reference area and an image of the second reference area and a difference between the image of the first reference area and at least one image of the at least one shift area; and generating the minimum map based on the difference image.

[0006] The step of generating the minimum map based on the difference image may include selecting a minimum value from among corresponding difference values ​​at corresponding positions in the difference image, and determining a pixel value of the minimum map based on the minimum value. The pixel value of the minimum map may be the minimum value or an index (value) of a difference image including the minimum value among the difference images.

[0007] The step of detecting the liveness includes inputting input data including at least one patch based on the minimum map into at least one liveness detection model, and detecting the liveness based on an output of the at least one liveness detection model, wherein the at least one liveness detection model includes at least one neural network, and the at least one neural network can be pre-trained to detect liveness of objects in the input data.

[0008] The liveness detection method may further include generating a reference image by concatenating the first phase image, the second phase image, and the minimum map, and detecting the liveness may further include cropping the reference image based on an area of ​​interest (ROI) to generate the at least one patch. The at least one patch may include a plurality of patches including different characteristics of the object, and the at least one liveness detection model may include a plurality of liveness detection models that process input data including the plurality of patches. Detecting the liveness based on the output of the at least one liveness detection model may include detecting the liveness by fusing outputs of the plurality of liveness detection models in response to input of the input data.

[0009] The liveness detection method may further include generating a reference image by concatenating the first phase image, the second phase image, and the minimum map, and detecting the liveness may include detecting the liveness based on the reference image. The liveness detection method may further include pre-processing the first phase image and the second phase image, and the pre-processing may include applying at least one of downsizing, lens shadow correction, gamma correction, histogram matching, and noise reduction to the first phase image and the second phase image.

[0010] The first pixel of the first pixel group and the second pixel of the second pixel group may be adjacent to each other, and the first phase image may correspond to a left image, and the second phase image may correspond to a right image.

[0011] According to one embodiment, a liveness detection device includes a processor and a memory containing instructions executable by the processor, and when the instructions are executed by the processor, the processor generates a first phase image based on first visual information of a first phase detected by a first pixel group of an image sensor, generates a second phase image based on second visual information of a second phase detected by a second pixel group of the image sensor, generates a minimum map based on a disparity between the first phase image and the second phase image, and detects liveness based on the minimum map.

[0012] According to one embodiment, the electronic device includes an image sensor that detects first visual information of a first phase through a first pixel group and detects second visual information of a second phase through a second pixel group, and a processor that generates a first phase image based on the first visual information, generates a second phase image based on the second visual information, generates a minimum map based on a parallax between the first phase image and the second phase image, and detects liveness based on the minimum map.

[0013] According to one embodiment, the device includes one or more processors and at least one memory storing instructions executable by the one or more processors, and in response to the instructions being executed by the one or more processors, the one or more processors input an image including an object, generate disparity data based on a disparity between a first phase image corresponding to the object and a second phase image corresponding to the object, generate a reference image based on the first phase image, the second phase image, and the disparity data, generate input data based on the reference image, input the input data to a detection model including a neural network, and authenticate the object based on output data of the detection model.

[0014] The one or more processors may determine liveness of the object based on the output data to authenticate the object. The one or more processors may generate the reference image by concatenating the first-phase image, the second-phase image, and the disparity data. [Effects of the Invention]

[0015] According to the present invention, a method and apparatus for detecting liveness using phase difference can be provided. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 2 is a diagram illustrating an operation of a liveness detection device according to an embodiment. [Figure 2] FIG. 1 illustrates a QPD image sensor according to an embodiment. [Figure 3] 10A and 10B illustrate the difference between 2D and 3D objects that can be detected through phase imaging according to one embodiment. [Figure 4] FIG. 10 illustrates a liveness detection process using phase difference according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating phase characteristics in each direction of an input image according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating a process for generating a minimum map according to an embodiment. [Figure 7A] 10A and 10B are diagrams illustrating a phase image shifting process according to another embodiment; [Figure 7B] 10A and 10B are diagrams illustrating a phase image shifting process according to another embodiment; [Figure 8] FIG. 1 illustrates a liveness detection method using reference information and a liveness detection model according to an embodiment. [Figure 9] 10A and 10B are diagrams illustrating a process of generating a reference image according to an embodiment; [Figure 10] FIG. 10 illustrates a process for generating output data using a liveness detection model according to an embodiment. [Figure 11]FIG. 10 illustrates a process for generating output data using multiple liveness detection models according to an embodiment. [Figure 12A] 1 is a block diagram illustrating a liveness detection apparatus according to an embodiment. [Figure 12B] FIG. 10 is a block diagram illustrating a liveness detection device according to another embodiment. [Figure 13] FIG. 1 is a block diagram illustrating an electronic device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0018] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed as being solely for the purpose of distinguishing one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0019] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0020] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0021] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, in which the same reference numerals in the various drawings denote the same elements.

[0022] FIG. 1 is a diagram illustrating an operation of a liveness detection device according to an embodiment. Referring to FIG. 1, a liveness detection device 100 generates a detection result 120 based on visual information of an object 110. The detection result 120 includes information related to liveness. For example, the detection result 120 indicates whether the object 110 is a real user, such as a video, or an attack vector, such as a video of a user being filmed. The detection result 120 can be used for image-based biometric authentication, such as face recognition and iris recognition.

[0023] Visual information of the object 110 can be expressed through multiple phases. The image sensor 130 detects visual information of the multiple phases and generates sensor data related to the visual information of each phase. The image sensor 130 corresponds to a multiphase detection sensor. For example, the image sensor 130 may be a two-phase detection (2PD) sensor that detects two phases or a quad-phase detection (QPD) sensor that detects four phases. However, the number of phases detected by the image sensor 130 is not limited thereto, and the image sensor 130 may detect various numbers of phases. FIG. 1 illustrates the image sensor 130 as a 2PD sensor, and the following description will mainly focus on an embodiment in which the image sensor 130 corresponds to a 2PD sensor. However, this is for convenience of explanation, and the present invention may also be applied to cases in which the image sensor 130 corresponds to other types of multiphase detection sensors, such as a QPD sensor.

[0024] The pixels included in the image sensor 130 belong to either a first group 1 or a second group 2. The first pixel of the first group 1 detects first visual information of a first phase to generate first sensor data, and the second pixel of the second group 2 detects second visual information of a second phase to generate second sensor data. The first pixel and the second pixel may be arranged adjacent to each other. Here, the adjacent arrangement of the first pixel and the second pixel may include at least one of the following: there is no pixel between the first pixel and the second pixel in the direction in which the phase characteristics are differentiated; the first pixels are not arranged contiguously; and the second pixels are not arranged contiguously. The meaning of differentiated phase characteristics will be further described below with reference to FIG. 5.

[0025] 2 is a diagram illustrating a QPD image sensor according to an embodiment. Referring to FIG. 2, the image sensor 210 can detect four different phases by dividing them into a grid pattern. More specifically, a first pixel of a first group 1 of the image sensor 210 detects first visual information of a first phase, a second pixel of a second group 2 detects second visual information of a second phase, a third pixel of a third group 3 detects third visual information of a third phase, and a fourth pixel of a fourth group 4 detects fourth visual information of a fourth phase.

[0026] Referring again to FIG. 1 , the liveness detection apparatus 100 generates a first phase image 141 based on first sensor data and a second phase image 142 based on second sensor data. Depending on the characteristics of the image sensor 130, disparity exists between the first phase image 141 and the second phase image 142, and this disparity can be used to detect the liveness of the object 110. For example, FIG. 3 illustrates a difference between a 2D object and a 3D object detected through phase images according to an embodiment. When a 2D object is photographed, disparity is not detected through the first and second phase images. When a 3D object is photographed, disparity can be detected through the first and second phase images. For example, disparity can be detected in a three-dimensional structure such as a user's nose.

[0027] The liveness detection device 100 generates a minimum map 150 and a reference image 160 based on the first phase image 141 and the second phase image 142, and detects the liveness of the object 110 based on the minimum map 150 and the reference image 160. If the object 110 is a real user, there is a disparity corresponding to the difference between the first phase image 141 and the second phase image 142. Depending on the structural characteristics of the image sensor 130, in which the spacing between the first pixel of the first group 1 and the second pixel of the second group 2 is narrow, the disparity may not be relatively large. The liveness detection device 100 analyzes such subtle disparity using the minimum map 150 and the reference image 160, and can efficiently detect the liveness of the object 110 based on the analysis result.

[0028] According to an embodiment, the liveness detection apparatus 100 shifts one of the first phase image 141 and the second phase image 142 at least once while fixing the other, and generates a minimum map 150 based on a difference between the fixed image and the shifted image. For example, the liveness detection apparatus 100 may set a first base region in the first phase image 141, set a second base region corresponding to the first base region in the second phase image 142, and shift the second base region by a reference shift value to generate at least one shifted region. Thereafter, the liveness detection apparatus 100 may generate a difference image based on a difference between an image of the first base region and an image of the second reference region and a difference between the image of the first base region and an image of the at least one shifted region.

[0029] According to one embodiment, the liveness detection apparatus 100 generates a minimum map 150 based on a difference image. For example, the liveness detection apparatus 100 selects the minimum value among corresponding difference values ​​located at corresponding coordinates in the difference image, and determines a pixel value of the minimum map 150 based on the minimum value. In this manner, each pixel value of the minimum map 150 can be determined. The pixel value of the minimum map 150 corresponds to the minimum value or to an index of the difference image including the minimum value among the difference images. The minimum map 150 includes the minimum value or the index.

[0030] According to an embodiment, the liveness detection apparatus 100 generates a reference image 160 by combining the first phase image 141, the second phase image 142, and the minimum map 150, and detects the liveness of the object 110 based on the reference image 160. For example, the liveness detection apparatus 100 concatenates the first phase image 141, the second phase image 142, and the minimum map 150 to generate the reference image 160, and detects the liveness of the object 110 based on the reference image 160.

[0031] According to one embodiment, the liveness detection device 100 detects the liveness of the object 110 using at least one liveness detection model. Each liveness detection model includes at least one neural network. The liveness detection device 100 generates input data for the liveness detection model based on the reference video 160, inputs the input data to the liveness detection model, and detects the liveness of the object 110 based on output data of the liveness detection model. At least a portion of the neural network may be implemented in software, hardware including a neural processor, or a combination of software and hardware.

[0032] For example, neural networks include deep neural networks (DNNs), which include fully connected networks, deep convolutional networks, and recurrent neural networks. A DNN includes multiple layers, each of which includes an input layer, at least one hidden layer, and an output layer.

[0033] Neural networks are trained to perform given operations by mapping input data and output data that have a nonlinear relationship to each other using deep learning. Deep learning is a machine learning method for solving problems given by big data sets. Deep learning can be understood as an optimization problem solving process that searches for a point where energy is minimized while training a neural network using prepared training data. Through supervised or unsupervised learning in deep learning, weights corresponding to the neural network structure or model are obtained, and input data and output data can be mapped to each other through these weights.

[0034] A neural network may be trained based on training data in a training step, and may perform inference operations such as classification, recognition, and detection on input data in an inference step. The neural network of a liveness detection model is pre-trained to detect the liveness of objects in input data. Here, the term "pre-trained" refers to before the neural network is "started." "Starting" a neural network means that the neural network is prepared for inference. For example, "starting" a neural network may include loading the neural network into memory, or inputting input data for inference into the neural network after loading it into memory.

[0035] 4 is a diagram illustrating a liveness detection process using a phase difference according to an embodiment. Referring to FIG. 4, in step S410, the liveness detection apparatus generates a phase image based on visual information of a plurality of phases. For example, the liveness detection apparatus receives sensor data from pixel groups that detect visual information of different phases, and generates a phase image based on the sensor data. Hereinafter, a representative embodiment will be described in which the phase image includes a first phase image and a second phase image.

[0036] In step S420, the liveness detection apparatus performs preprocessing on the phase image. When detecting liveness from a 2D image, preprocessing such as distortion correction is generally performed. However, in the preprocessing according to the embodiment, such preprocessing such as distortion correction may not be performed. This is because storing the shape of an object is preferable for detecting subtle disparity, and preprocessing such as distortion correction may distort the shape of the object. Instead, according to the embodiment, preprocessing may be performed that includes at least one of downsizing, lens shading correction, gamma correction, histogram matching, and denoising, or a combination thereof. Alternatively, preprocessing may not be performed.

[0037] According to an embodiment, the liveness detection apparatus may apply downsizing to a phase image and then perform preprocessing, such as lens shadow correction, on the downsized phase image. This downsizing may reduce the amount of computation. For example, downsizing may be performed in a direction in which phase characteristics are not differentiated. Since information related to disparity is mainly included in a direction in which phase characteristics are differentiated, information loss during the downsizing process is minimized. Furthermore, other preprocessing, such as lens shadow correction and gamma correction, may be performed to remove noise and improve the accuracy of image information. Hereinafter, an embodiment of the downsizing operation will be further described with reference to FIG. 5.

[0038] FIG. 5 illustrates phase characteristics in each direction of an input image according to an embodiment. Referring to FIG. 5, pixels of a first group 1 and pixels of a second group 2 are alternately arranged in the horizontal direction of an image sensor 510. Therefore, it can be seen that the phase characteristics are reflected in the horizontal direction. In other words, it can be seen that the phase characteristics are differentiated according to pixel values ​​in the horizontal direction. Since the image sensor 510 illustrated in FIG. 5 corresponds to a 2PD sensor, the phase characteristics are not differentiated in the vertical direction. Therefore, the liveness detection apparatus can perform downsizing in a direction in which the phase characteristics are not differentiated in order to maintain the phase characteristics. For example, the liveness detection apparatus may downsize each of a first phase image 521 and a second phase image 522 in the vertical direction.

[0039] Here, the liveness detection apparatus may perform downsizing by removing detection data of a specific row according to a predetermined downsizing ratio, or by statistically processing (e.g., averaging) detection data of multiple rows according to a predetermined downsizing ratio. For example, the liveness detection apparatus may downsize the phase image by half by averaging the sensor data of the first row and the sensor data of the second row adjacent to the first row for each column.

[0040] 4, in step S430, the liveness detection apparatus generates a minimum map based on the disparity between the phase images. As described above, the liveness detection apparatus fixes one of the first and second phase images, shifts the other at least once, and generates a minimum map based on the difference between the fixed image and the shifted image. Embodiments related to generating a minimum map will be further described later with reference to FIGS. 6, 7A, and 7B.

[0041] In step S440, the liveness detection device detects liveness based on the minimum map. According to one embodiment, the liveness detection device generates a reference image by combining the phase image and the minimum map, input data corresponding to the reference image into a liveness detection model, and detects liveness based on output data of the liveness detection model. For example, the liveness detection device crops the reference image based on a region of interest (ROI) to generate at least one patch, and input data for the detection model is generated based on the at least one patch. An embodiment related to liveness detection will be further described later with reference to FIG. 8.

[0042] FIG. 6 is a diagram illustrating a process for generating a minimum map according to an embodiment. Referring to FIG. 6, in step S610, the liveness detection apparatus performs phase image shifting. As described above, the liveness detection apparatus shifts the remaining phase image at least once while fixing one of the first and second phase images. FIG. 6 illustrates an example in which the first phase image is fixed and the XNth phase image is shifted. In FIG. 6, the numbers in each pixel of each phase image indicate pixel values.

[0043] In XN, X indicates that the phase characteristics are divided in the horizontal direction, and N indicates the number of phases. For example, when a first phase image and a second phase image generated by a 2PD sensor are used, the second phase image is referred to as an X2-th phase image. Hereinafter, an embodiment in which the XN-th phase image corresponds to the second phase image will be described. The liveness detection apparatus sets a base region in the first phase image and sets at least one shift region in the second phase image. For example, the liveness detection apparatus sets shift regions of x-1, x0, and x+1 in the second phase image. Here, x0 indicates a base region in which no shift is performed.

[0044] The fundamental domain of the first phase image is called the first fundamental domain, and the fundamental domain of the second phase image is called the second fundamental domain, and the first fundamental domain and the second fundamental domain correspond to each other in position. In x-1 and x+1, - and + indicate the shift direction, and 1 indicates the reference shift value. The fundamental domain is set based on the reference shift value. If the reference shift value is r, a shift domain is set by shifting the fundamental domain by r in a specific direction. Therefore, the fundamental domain is set within a range that can ensure a margin for shifting.

[0045] The liveness detection apparatus shifts the second reference region (i.e., the x0 shift region) by a reference shift value (i.e., 1) according to a shift direction, and sets at least one shift region (i.e., the x-1 shift region and the x+1 shift region). The reference shift value may be set to various values, and the number of shift regions corresponding to the reference shift value may be set. For example, the number of shift regions may be determined based on the reference shift value and the number of shift directions.

[0046] As an example, when the reference shift value is 1 and the number of shift directions is two (left side, right side), there are 2×1+1=3 shift areas. The three shift areas include the x-1, x0, and x+1 shift areas. As another example, when the reference shift value is 5 and the number of shift directions is two (left side, right side), there are 2×5+1=11 shift areas. The 11 shift areas include the x-5 to x-1, x0, and x+1 to x+5 shift areas. As a further example, when the reference shift value is 1 and the number of shift directions is four (left side, right side, up side, down side), there are 2×1+2×1+1=5 shift areas. The five shift areas include the x-1, y-1, xy0, x+1, and y+1 shift areas.

[0047] When a multi-phase detection sensor such as a QPD sensor is used, phase characteristics may be classified in directions other than the horizontal direction. According to one embodiment, the liveness detection apparatus may determine a shift region for each phase image by performing phase image shifts on the phase images of the QPD sensor in the horizontal and vertical directions, as shown in FIG. 7A. In the case of the XNth phase image, shift regions (x-1, x0, and x+1) are determined through a horizontal shift, as shown in phase image shift 610 in FIG. 6. In the case of the YNth phase image, shift regions (y-1, y0, and y+1) are determined through a vertical shift.

[0048] In XN and YN, X indicates that the phase characteristics are divided in the horizontal direction, and Y indicates that the phase characteristics are divided in the vertical direction. N indicates the number of phases. Here, although the same number of phases is used in the vertical and horizontal directions, different numbers of phases can be used in the vertical and horizontal directions. For example, N may be determined based on the number of phases that the sensor can distinguish. In the case of a QPD sensor, N may be 2, and in the embodiment shown in FIG. 7A, there are a first phase image, an X2th phase image, and a Y2th phase image.

[0049] According to another embodiment, the liveness detection apparatus may determine a shift region for each phase image by shifting the phase images of the QPD sensor in the horizontal, vertical, and diagonal directions, as shown in FIG. 7B . For the XNth phase image, shift regions (x−1, x0, and x+1) are determined through horizontal shifting, and for the YNth phase image, shift regions (y−1, y0, and y+1) are determined through vertical shifting. For the ZNth phase image, shift regions (z−1, z0, and z+1) are determined through diagonal shifting. In ZN, Z indicates that the phase characteristics are divided in the diagonal direction, and N indicates the number of phases. When N=2, the first phase image, the X2th phase image, the Y2th phase image, and the Z2th phase image may be used in the embodiment shown in FIG. 7B .

[0050] Once the shift regions are determined in this manner, in step S620, the difference between the image of the fundamental region and the image of each shift region is calculated. The liveness detection apparatus generates a difference image based on the difference between a fixed image (e.g., the image of the first fundamental region) and a shifted image (e.g., the image of the shift region), and generates a minimum map based on the difference image. For example, the liveness detection apparatus generates a first difference image based on the difference between the image of the first fundamental region and the image of the x-1 shift region, generates a second difference image based on the difference between the image of the first fundamental region and the image of the x0 shift region, and generates a third difference image based on the difference between the image of the first fundamental region and the image of the x+1 shift region.

[0051] The liveness detection device assigns an index value to each difference image. For example, the detection device may assign index values ​​in the order of x-1, x0, and x+1. FIG. 6 illustrates an example in which the first difference image is assigned an index value of 0, the second difference image is assigned an index value of 1, and the third difference image is assigned an index value of 2. However, various other index assignment orders are also possible.

[0052] A difference image set including such difference images is generated for each phase image. For example, in the embodiment shown in Fig. 7A, a difference image set for the XNth phase image and a difference image set for the YNth phase image are generated. In the embodiment shown in Fig. 7B, difference image sets are generated for the XNth phase image, the YNth phase image, and the ZNth phase image, respectively.

[0053] In step S630, the liveness detection apparatus generates a minimum map. The liveness detection apparatus selects the minimum value among the corresponding difference values ​​at corresponding positions in the difference images of the difference image set, and determines pixel values ​​of the minimum map based on the minimum value. As an example, in FIG. 6, the corresponding difference values ​​at (1,1) are 1, 0, and 6. Of these, 0 is selected as the minimum value. As another example, the corresponding difference values ​​at (2,2) are 25, 33, and 30. Of these, 25 is selected as the minimum value. In this way, the minimum value among the corresponding difference values ​​is selected, and the pixel of the minimum map can be determined based on the minimum value.

[0054] The pixel value of the minimum map corresponds to the minimum value or corresponds to the index of the difference image containing the minimum value in the difference image. A minimum map containing the minimum value is called a minimum value map, and a minimum map containing the minimum index is called a minimum index map. In the above example, 0 is selected as the minimum value at the position (1,1), and the index of the difference image containing 0 is 1. Therefore, the pixel value of (1,1) in the minimum value map is 0, and the pixel value of (1,1) in the minimum index map is 1. Also, 25 is selected as the minimum value at the position (2,2), and the index of the difference image containing 25 is 0. Therefore, the pixel value of (2,2) in the minimum value map is 25, and the pixel value of (2,2) in the minimum index map is 0.

[0055] As described above, a difference image set can be generated for each phase image. When phase images for multiple directions exist, as in the embodiment shown in Figures 7A and 7B, a minimum map for each phase image is generated based on the difference image set for each phase image. For example, in the embodiment shown in Figure 7A, a minimum map can be generated for each of the XNth phase image and the YNth phase image. In the embodiment shown in Figure 7B, a minimum map is generated for each of the XNth phase image, the YNth phase image, and the ZNth phase image.

[0056] 8 is a diagram illustrating a liveness detection method using reference information and a liveness detection model according to an embodiment. Referring to FIG. 8, in step S810, the liveness detection apparatus generates a reference image by concatenating a phase image and a minimum map. Concatenation is an example of a combination. Hereinafter, an embodiment of generating a reference image will be further described with reference to FIG. 9.

[0057] 9 is a diagram illustrating a process of generating a reference image (S910) according to an embodiment. Referring to FIG. 9, when phase characteristics are divided in the horizontal direction, a reference image is generated by concatenating a first phase image, an XNth phase image (e.g., a second phase image), and a minimum map. Here, to match the sizes of each image, an image of the first fundamental domain is used instead of the first phase image, and an image of the second fundamental domain is used instead of the XNth phase image.

[0058] When the phase characteristics are differentiated in both the horizontal and vertical directions, additional phase images and additional minimum maps are further concatenated. For example, in the embodiment shown in Figure 7A, a reference image is generated by concatenating a first phase image, an XNth phase image, a YNth phase image, a first minimum map, and a second minimum map. In the embodiment shown in Figure 7B, a reference image is generated by further concatenating a ZNth phase image and a third minimum map to the first phase image. Here, the first minimum map is generated based on the first phase image and the XNth phase image, the second minimum map is generated based on the first phase image and the YNth phase image, and the third minimum map is generated based on the first phase image and the ZNth phase image.

[0059] In addition, to match the size of each image, an image of the first basic domain is used instead of the first phase image, an image of the second basic domain is used instead of the XN phase image, an image of the third basic domain is used instead of the YN phase image, and an image of the fourth basic domain is used instead of the ZN phase image. The image of the third basic domain shows an area in the YN phase image corresponding to the first basic domain, and the image of the fourth basic domain shows an area in the ZN phase image corresponding to the first basic domain.

[0060] 8, in step S820, the liveness detection apparatus inputs input data corresponding to a reference image into the liveness detection model. For example, the input data may correspond to the reference image or a cropped version of the reference image. In the latter case, the reference image may be cropped into various versions based on the ROI.

[0061] For example, the ROI corresponds to a face box. In this case, the cropped image corresponding to the face box is denoted as 1t, and the cropped image corresponding to m times the face box is denoted as m×t (e.g., 2t for twice the size). The full-size reference image is denoted as reduced. According to one embodiment, the input data consists of 1t, 2t, and reduced. An embodiment relating to a liveness detection model will be further described below with reference to FIGS. 10 and 11.

[0062] In step S830, the liveness detection device detects liveness based on the output data of the liveness detection model. The output data includes a liveness score. The liveness detection device compares the liveness score with a predetermined threshold to detect the liveness of the object. The detection result indicates whether the object corresponds to a real user, such as a video, or an offensive means, such as a video of a user being filmed.

[0063] 10 is a diagram illustrating a process of generating output data using a liveness detection model according to an embodiment. Referring to FIG. 10, the liveness detection model generates input data 1030 based on a reference image 1010 and ROI information 1020. The ROI information may include information about face boxes and is generated by a face detector. The liveness detection model generates patches by cropping the reference image 1010 based on the ROI information 1020. The input data 1030 includes the patches. If the reference image 1010 includes multiple concatenated images, the liveness detection apparatus may generate patches by cropping each image based on the ROI information, and then concatenate the patches to generate the input data 1030.

[0064] The liveness detection model 1040 may include at least one neural network, which may be pre-trained to detect the liveness of objects in input data. The training data includes input data and a label. For example, if the input data corresponds to a real user, the label may have a high liveness score. If the input data corresponds to an offensive means such as a video, the label may have a low liveness score. The neural network may be trained to output a liveness score for the input data based on such training data. The liveness detection model 1040 shown in FIG. 10 is shown in a state after training is complete.

[0065] The liveness detection device inputs input data 1030 to a liveness detection model 1040, which outputs output data 1050 in response to the input data 1030. The output data 1050 includes a liveness score. The liveness detection device can detect the liveness of an object by comparing the liveness score with a predetermined threshold.

[0066] 11 is a diagram illustrating a process of generating output data using multiple liveness detection models according to an embodiment. Referring to FIG. 11, the liveness detection model generates input data 1130 based on a reference image 1110 and ROI information 1120. The ROI information includes information about face boxes. The liveness detection model crops the reference image 1110 based on the ROI information 1120 to generate multiple patches (e.g., 1t, 2t, reduced).

[0067] For example, the liveness detection model generates patch 1t corresponding to a face box and patch 2t obtained by expanding the face box by two times. The patch (reduced) represents the full-size reference image 1110. Instead of patch (reduced), a patch obtained by expanding the face box by three times (referred to as 3t) may be used. Patches 1t and 2t (reduced) contain different characteristics of the object. For example, patch 1t may contain characteristics of the face, patch 2t may contain characteristics around the face, and patch (reduced) may contain characteristics related to the background or context. The input data 1130 includes multiple such patches.

[0068] The liveness detection model 1140 outputs output data 1150 for each patch in response to the input data 1030. For example, the liveness detection model 1140 includes a first liveness detection model, a second liveness detection model, and a third liveness detection model. The first liveness detection model outputs output data 1150 for patch 1t, the second liveness detection model outputs output data 1150 for patch 2t, and the third liveness detection model outputs output data 1150 for patch 1t.

[0069] The output data 1150 includes a liveness score for each patch. The liveness detection device may perform a statistical calculation (e.g., an average calculation) based on the liveness score for each patch and compare the calculation result with a predetermined threshold to detect the liveness of the object. Additionally, the matters described with reference to FIG. 10 may be applied to the process of generating the output data of FIG. 11.

[0070] 12A is a block diagram illustrating a liveness detection apparatus according to an embodiment. Referring to FIG. 12A, the liveness detection apparatus 1200 includes a processor 1210 and a memory 1220. The memory 1220 is connected to the processor 1210 and stores instructions executable by the processor 1210, data operated by the processor 1210, or data processed by the processor 1210. The memory 1220 may include a non-transitory computer-readable storage medium, such as a high-speed random access memory and / or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0071] The processor 1210 executes instructions to perform one or more of the operations described with reference to Figures 1 to 11. For example, the processor 1210 may generate a first-phase image based on first visual information of a first phase detected by a first pixel group of an image sensor, generate a second-phase image based on second visual information of a second phase detected by a second pixel group of the image sensor, generate a minimum map based on a disparity between the first-phase image and the second-phase image, and detect liveness based on the minimum map.

[0072] 12B is a block diagram showing a liveness detection device according to another embodiment. Referring to FIG. 12B, the liveness detection device 1250 includes a multi-phase detection sensor 1251, a multi-phase image pre-processing unit 1252, an ROI detector 1253, a multi-phase patch generator 1254, and a liveness detector 1255. The multi-phase detection sensor 1251, the multi-phase image pre-processing unit 1252, the ROI detector 1253, the multi-phase patch generator 1254, and the liveness detector 1255 may be implemented by at least one hardware module, at least one software module, and / or a combination thereof.

[0073] Although the operations related to liveness detection will be described below from the perspectives of the multi-phase detection sensor 1251, the multi-phase image pre-processing unit 1252, the ROI detector 1253, the multi-phase patch generator 1254, and the liveness detector 1255, the operations described below do not necessarily have to be performed by separate entities such as the multi-phase detection sensor 1251, the multi-phase image pre-processing unit 1252, the ROI detector 1253, the multi-phase patch generator 1254, and the liveness detector 1255. For example, operations described as being performed by one entity may be performed by another entity, or these operations may be performed by a single integrated entity called the liveness detection device 1250.

[0074] The multi-phase detection sensor 1251 can detect visual information of multiple phases and generate sensor data related to the visual information of each phase. For example, the multi-phase detection sensor 1251 may be a 2PD sensor that detects two types of phases, a QPD sensor that detects four types of phases, or a sensor that detects many types of phases. The multi-phase detection sensor 1251 can detect visual information having different phase characteristics using adjacent detection pixels and generate sensor data based on the detected visual information. A phase image corresponding to each phase characteristic is generated based on the corresponding sensor data.

[0075] The multi-phase image pre-processing unit 1252 may perform pre-processing on the phase images. For example, the multi-phase image pre-processing unit 1252 may perform pre-processing including at least one of downsizing, lens shadow correction, gamma correction, histogram matching, and noise reduction, or a combination thereof. According to an embodiment, the multi-phase image pre-processing unit 1252 may not perform pre-processing such as distortion correction, instead of pre-processing such as distortion correction. This is because storing the shape of an object is preferable for detecting fine disparity, and pre-processing such as distortion correction may distort the shape of the object.

[0076] The ROI detector 1253 detects an ROI in the phase image. For example, the ROI corresponds to a face box in each phase image. The ROI detector identifies the ROI based on coordinate information and / or size information. According to one embodiment, the phase image may be resized to fit the input size of the ROI detector 1253 and then input to the ROI detector 1253.

[0077] The multi-phase patch generator 1254 may generate a minimum map based on a phase image (e.g., a phase image to which preprocessing has been applied) and generate a reference image using the minimum map. For example, the multi-phase patch generator 1254 may fix one phase image, shift at least one remaining phase image at least once, and generate at least one minimum map based on the difference between the fixed image and the shifted image. The multi-phase patch generator 1254 may generate a reference image by concatenating the phase image and at least one minimum map.

[0078] The multi-phase patch generator 1254 then crops the reference image based on the ROI to generate at least one patch. The at least one patch is used to generate input data for the liveness detector 1255. For example, the multi-phase patch generator 1254 crops the reference image based on the ROI to generate patch 1t corresponding to a face box and patch 2t by expanding the face box by two times. The multi-phase patch generator 1254 also prepares a patch (reduced) corresponding to the full-size reference image. The multi-phase patch generator 1254 then generates input data based on patches 1t and 2t (reduced). For example, the multi-phase patch generator 1254 can chain each patch and resize it to suit the input size of the liveness detector 1255.

[0079] The liveness detector 1255 can detect the liveness of an object based on the input data. For example, the liveness detector 1255 includes at least one neural network pre-trained to detect the liveness of an object in the input data. The at least one neural network outputs output data including a liveness score in response to receiving the input data. The liveness detector 1255 can detect the liveness of the object by comparing the liveness score with a threshold.

[0080] FIG. 13 is a block diagram illustrating an electronic device according to an embodiment. Referring to FIG. 13, the electronic device 1300 generates an input image including an object and detects liveness of the object in the input image. The electronic device 1300 may also perform biometric authentication (e.g., image-based biometric authentication such as face authentication, iris authentication, etc.) based on the liveness of the object. The electronic device 1300 may structurally and / or functionally include the liveness detection device 100 illustrated in FIG. 1, the liveness detection device 1200 illustrated in FIG. 12A, and / or the liveness detection device 1250 illustrated in FIG. 12B.

[0081] The electronic device 1300 includes a processor 1310, a memory 1320, a camera 1330, a storage device 1340, an input device 1350, an output device 1360, and a network interface 1370. The processor 1310, the memory 1320, the camera 1330, the storage device 1340, the input device 1350, the output device 1360, and the network interface 1370 communicate via a communication bus 1380. For example, the electronic device 1300 may be embodied as at least part of a mobile device such as a mobile phone, a smartphone, a PDA, a netbook, a tablet computer, a laptop computer, etc.; a wearable device such as a smart watch, a smart band, smart glasses, etc.; a computing device such as a desktop, a server, etc.; a home appliance such as a television, a smart television, a refrigerator, etc.; a security device such as a door rack, etc.; or a vehicle such as a smart vehicle, etc.

[0082] The processor 1310 executes functions and instructions to be executed within the electronic device 1300. For example, the processor 1310 processes instructions stored in the memory 1320 or the storage device 1340. The processor 1310 may perform one or more of the operations described with reference to FIGS. 1 to 12B.

[0083] The memory 1320 stores data for liveness detection. The memory 1320 may include a computer-readable storage medium or a computer-readable storage device. The memory 1320 stores instructions for execution by the processor 1310 and stores relevant information during execution of software and / or applications by the electronic device 1300.

[0084] The camera 1330 captures photos and / or videos. For example, the camera 1330 captures facial images including the user's face. According to one embodiment, the camera 1330 provides 3D images including depth information about objects. According to one embodiment, the camera 1330 includes an image sensor that detects multiple phases (e.g., a 2PD sensor, a QPD sensor, etc.).

[0085] The storage device 1340 may include a computer-readable storage medium or a computer-readable storage device. The storage device 1340 may store various models and data used in the liveness detection process, such as a liveness detection model and a face detector. According to one embodiment, the storage device 1340 may store a larger amount of information than the memory 1320 and may store information for a long period of time. For example, the storage device 1340 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0086] Input device(s) 1350 can receive input from a user through traditional input methods such as a keyboard and mouse, and newer input methods such as touch input, voice input, and image input. For example, input device(s) 1350 can include a keyboard, mouse, touchscreen, microphone, or any other device capable of detecting input from a user and communicating the detected input to electronic device 1300.

[0087] Output device(s) 1360 may provide output of electronic device 1300 to a user via visual, auditory, or tactile channels. Output device(s) 1360 may include, for example, a display, a touchscreen, a speaker, a vibration generator, or any other device capable of providing output to a user. Network interface 1370 may communicate with external devices via a wired or wireless network.

[0088] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, a single processing device may be described; however, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0089] Software includes computer programs, codes, instructions, or a combination of one or more thereof, which can configure a processing device to operate as desired or independently or in combination to instruct the processing device. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by or provide instructions or data to a processing device. The software can be distributed across computer systems coupled to a network and stored and executed in a distributed manner. The software and data can be stored on one or more computer-readable recording media.

[0090] The methods according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable storage medium. The storage medium may include program instructions, data files, data structures, and the like, alone or in combination. The storage medium and program instructions may be specially designed and constructed for the purposes of the present invention, or they may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example. A hardware device may be configured to operate as one or more software modules to perform the operations described in the present invention, or vice versa.

[0091] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents, and still achieve suitable results. [Explanation of symbols]

[0092] 100 Liveness detection device 110 objects 120 detection results 130 Image Sensor 141 First Phase Image 142 Second Phase Image 150 smallest maps 160 Reference Video 210 Image Sensor 1010 Reference Video 1020 ROI information 1030 Input Data 1040 Liveness Detection Model 1050 output data 1110 Reference Video 1120 ROI information 1130 Input Data 1140 Liveness Detection Model 1150 output data 1200 Liveness Detector 1210 processor 1220 memory 1250 Liveness Detector 1251 Multi-phase detection sensor 1252 Multi-phase image preprocessing unit 1253 ROI Detector 1254 Multi-Phase Patch Generator 1255 Liveness Detector 1300 Electronic equipment 1310 processor 1320 memory 1330 Camera 1340 Storage device 1350 Input Device 1360 Output Device 1370 Network Interface

Claims

1. In the liveness detection method, generating a first-phase image based on first visual information of a first phase detected by a first pixel group of the image sensor; generating a second-phase image based on second visual information of a second phase detected by a second pixel group of the image sensor; generating a minimum map based on a disparity between the first phase image and the second phase image; detecting liveness based on the minimum map; Including, The step of generating the minimum map comprises: setting a first reference region in the first phase image; setting a second reference area corresponding to the first reference area in the second phase image; shifting the second reference region by a reference shift value to set at least one shift region; generating a plurality of difference images based on a difference between the image of the first reference region and the image of the second reference region, and a difference between the image of the first reference region and at least one image of the at least one shift region; selecting a minimum value from among the corresponding difference values ​​at corresponding positions in the plurality of difference images; generating the minimum map by determining pixel values ​​at the locations based on the minimum values; A liveness detection method comprising:

2. The liveness detection method of claim 1 , wherein the pixel value of the minimum map is the minimum value or an index of a difference image including the minimum value among the plurality of difference images.

3. The step of detecting liveness comprises: generating a reference image by concatenating the first phase image, the second phase image, and the minimum map, the concatenation including concatenating pixel values ​​at corresponding positions in the first phase image, the second phase image, and the minimum map; inputting input data, including at least one patch based on the reference video, into at least one liveness detection model, the patch being an image generated by cropping the reference video based on a region of interest (ROI); detecting the liveness based on an output of the at least one liveness detection model; Including, the at least one liveness detection model includes at least one neural network; The method of claim 1 or 2, wherein the at least one neural network is pre-trained to detect liveness of objects in input data.

4. the at least one patch includes a plurality of patches including different properties of the object; the at least one liveness detection model includes a plurality of liveness detection models that process input data that includes the plurality of patches; 4. The liveness detection method of claim 3, wherein the step of detecting the liveness based on the output of the at least one liveness detection model includes a step of detecting the liveness by fusing outputs of the multiple liveness detection models in response to input of the input data.

5. The liveness detection method further includes generating a reference image by concatenating the first phase image, the second phase image, and the minimum map, wherein the concatenation includes concatenating pixel values ​​at corresponding positions in the first phase image, the second phase image, and the minimum map; The liveness detection method according to claim 1 or 2, wherein the step of detecting the liveness includes the step of detecting the liveness based on the reference video.

6. The liveness detection method further includes pre-processing the first phase image and the second phase image; 6. The liveness detection method according to claim 1, wherein the pre-processing step includes applying at least one of downsizing, lens shadow correction, gamma correction, histogram matching, and noise removal to the first phase image and the second phase image.

7. The liveness detection method according to claim 1 , wherein a first pixel of the first pixel group and a second pixel of the second pixel group are located adjacent to each other.

8. The liveness detection method includes: generating a third-phase image based on third visual information of a third phase detected by a third pixel group of the image sensor; generating a fourth-phase image based on fourth visual information of a fourth phase detected by a fourth pixel group of the image sensor; further comprising 8. The liveness detection method of claim 1, wherein the disparity between the first phase image and the third phase image and the disparity between the first phase image and the fourth phase image are further taken into consideration when the minimum map is generated.

9. A computer readable storage medium storing one or more programs including instructions for executing the method of any one of claims 1 to 8.

10. In a liveness detection device, a processor; a memory containing instructions executable by the processor; Including, When the instruction is executed by the processor, the processor: generating a first-phase image based on first visual information of a first phase detected by a first pixel group of the image sensor; generating a second-phase image based on second visual information of a second phase detected by a second pixel group of the image sensor; generating a minimum map based on a disparity between the first phase image and the second phase image; and Detecting liveness based on the minimum map; The generating of the minimum map includes the processor: A first reference region is set in the first phase image; setting a second reference area corresponding to the first reference area in the second phase image; Shifting the second reference region by a reference shift value to set at least one shift region; generating a plurality of difference images based on a difference between the image of the first reference region and the image of the second reference region and a difference between the image of the first reference region and at least one image of the at least one shift region; selecting a minimum value from among the corresponding difference values ​​at corresponding positions in the plurality of difference images; and A liveness detection device that generates the minimum map by determining pixel values ​​at the locations based on the minimum values, respectively.

11. 11. The liveness detection device of claim 10, wherein the processor generates a reference image by concatenating the first phase image, the second phase image, and the minimum map, and detects the liveness based on the reference image, wherein the concatenation includes concatenating pixel values ​​at corresponding positions in the first phase image, the second phase image, and the minimum map.

12. The liveness detection device according to claim 10 or 11, wherein a first pixel of the first pixel group and a second pixel of the second pixel group are located adjacent to each other.

13. an image sensor that detects first visual information of a first phase through a first pixel group and detects second visual information of a second phase through a second pixel group; a processor that generates a first phase image based on the first visual information, generates a second phase image based on the second visual information, generates a minimum map based on a parallax between the first phase image and the second phase image, and detects liveness based on the minimum map; Including, the processor sets a first reference area in the first phase image, sets a second reference area corresponding to the first reference area in the second phase image, shifts the second reference area by a reference shift value to set at least one shift area, generates a plurality of difference images based on a difference between the image of the first reference area and the image of the second reference area and a difference between the image of the first reference area and at least one image of the at least one shift area, selects a minimum value among corresponding difference values ​​at corresponding positions in the plurality of difference images, and generates the minimum map by respectively determining pixel values ​​in the minimum map based on the minimum value.

14. In a liveness detection apparatus using phase difference, a multi-phase detection sensor that detects first visual information of a first phase through a first pixel group to generate a first-phase image and detects second visual information of a second phase through a second pixel group to generate a second-phase image; a multi-phase patch generator that generates a minimum map based on the disparity between the first phase image and the second phase image; a liveness detector for detecting liveness based on the minimum map; Including, the multi-phase patch generator sets a first reference area in the first phase image, sets a second reference area corresponding to the first reference area in the second phase image, shifts the second reference area by a reference shift value to set at least one shift area, generates a plurality of difference images based on a difference between the image of the first reference area and the image of the second reference area and a difference between the image of the first reference area and at least one image of the at least one shift area, selects a minimum value among corresponding difference values ​​at corresponding positions in the plurality of difference images, and generates the minimum map by determining pixel values ​​in the minimum map based on the minimum value.

15. the liveness detection apparatus further includes a ROI detector configured to detect an ROI in the first phase image and the second phase image; the multi-phase patch generator generates a reference image by concatenating the first phase image, the second phase image, and the minimum map, and generates at least one patch by cropping the reference image based on the ROI, the concatenation including concatenating pixel values ​​at corresponding positions in the first phase image, the second phase image, and the minimum map; The liveness detection apparatus of claim 14 , wherein the liveness detector detects the liveness based on the at least one patch.

16. 16. The liveness detection device of claim 14, further comprising a multi-phase image preprocessor that applies at least one of downsizing, lens shadow correction, gamma correction, histogram matching, and noise removal to the first phase image and the second phase image.

17. one or more processors; at least one memory for storing instructions executable by the one or more processors; In response to the instruction being executed by the one or more processors, the one or more processors perform the method of claim 1 ; Input a video containing an object, generating parallax data based on a parallax between a first phase image corresponding to the object and a second phase image corresponding to the object; generating a reference image by concatenating the first phase image, the second phase image, and the disparity data, the concatenation including concatenating pixel values ​​at corresponding positions in the first phase image, the second phase image, and the minimum map; generating input data based on the reference image, the input data including at least one patch based on the reference image, the patch being an image generated by cropping the reference image based on a region of interest (ROI); inputting the input data into a liveness detection model including a neural network; authenticating the object based on output data of the liveness detection model, including a liveness score for the patch, and a predetermined threshold; Device.

18. The apparatus of claim 17 , wherein the one or more processors determine liveness of the object based on the output data to authenticate the object.

Images