Liveness detection method and device and face verification method and device

Abstract

A liveness detection method and device and a face verification method and device are disclosed. The liveness detection method includes: detecting a face area in an input image; measuring characteristic information of the face area; adjusting the measured characteristic information in response to the characteristic information not satisfying a condition; and performing liveness detection on the face area with the adjusted characteristic information when the measured characteristic information does not satisfy the condition.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0051495, filed on May 2, 2019, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2019-0105867, filed on August 28, 2019, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference for all purposes. Technical Field

[0002] The following description relates to liveness detection methods and apparatus and face verification methods and apparatus. Background Art

[0003] In a user authentication system, a computing device may determine whether to allow a user to access the computing device based on authentication information provided by the user. For example, the authentication information may include a password entered by the user and the user's biometric information. For example, the biometric information may include information associated with the user's fingerprint, iris, and face.

[0004] There are methods of facial spoofing, where facial spoofing can be a type of attack that uses, for example, an image, video, or mask with the intent to deceive or fool the underlying device or system into believing that an authorized user or person is attempting to access or use the device. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a liveness detection method includes: detecting a facial region in an input image; measuring characteristic information of the detected facial region; in response to the measured characteristic information being determined to not satisfy a condition, generating an adjusted facial region by adjusting the characteristic information of the facial region; and performing liveness detection on the adjusted facial region.

[0007] The step of measuring characteristic information of the facial region may include measuring a hue value of the facial region.

[0008] The step of generating the adjusted facial region may include: in response to determining that the measured tone value is not included in the preset tone value range, performing the step of generating the adjusted facial region by adjusting the tone value of the facial region to a tone value included in the preset tone value range.

[0009] The adjusting of the tone value of the facial region may include adjusting the measured tone value of the facial region by applying a tone adjustment model to the facial region.

[0010] The measuring of the characteristic information of the face region may include measuring a face inclination value indicating a degree of inclination of a face included in the face region.

[0011] The step of measuring the facial inclination value may include: detecting feature points corresponding to a left eye, a right eye, and both ends of a mouth in the facial region; and measuring the facial inclination value based on the detected feature points.

[0012] The step of generating the adjusted facial region may include performing the step of generating the adjusted facial region by adjusting the facial tilt value in response to determining that the measured facial tilt value is not included in a preset facial tilt value range.

[0013] The step of adjusting the facial tilt value may include adjusting the facial tilt value by rotating the facial region based on the measured facial tilt value.

[0014] The step of measuring characteristic information of the facial region may include measuring a white balance value of the facial region.

[0015] The step of generating the adjusted facial area may include: in response to determining that the measured white balance value is not included in the preset white balance value range, performing the step of generating the adjusted facial area by adjusting the white balance value of the facial area to a white balance value included in the preset white balance value range.

[0016] The method may further include, in response to determining that the measured characteristic information satisfies the condition, performing liveness detection based on the detected facial region without performing the step of generating the adjusted facial region.

[0017] The step of performing liveness detection may include: performing liveness detection using a liveness detection model based on a neural network.

[0018] The step of measuring characteristic information of the facial region may include: measuring any one or any combination of any two or more of a hue value, a facial tilt value, and a white balance value of the facial region. In response to any one or any combination of any two or more of the measured hue value, the measured facial tilt value, and the measured white balance value being determined to be outside a preset hue value range, a preset facial tilt value range, and a preset white balance value range, respectively, the hue value, the facial tilt value, and the white balance value are adjusted to fall within the preset hue value range, the preset facial tilt value range, and the preset white balance value range, respectively, to generate an adjusted facial region.

[0019] A non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, configure the one or more processors to perform the above method.

[0020] A neural network may be used to perform face verification on the adjusted face regions.

[0021] In another general aspect, a facial verification method using a neural network, the facial verification method comprising: extracting a current facial image of a facial region from an input image; calculating a current image feature value of the current facial image; comparing the current image feature value with a range of corresponding image feature values ​​of training facial images; in response to determining that the current image feature value is outside the range of the corresponding image feature value, generating an adjusted current facial image by adjusting the current image feature value of the current facial image to be included in the range of the corresponding image feature value; and inputting the adjusted current facial image to the neural network.

[0022] A plurality of facial images each having an image feature value may be used to train the neural network.

[0023] The range of the corresponding image feature values ​​may be an absolute range determined based on the minimum image feature value and the maximum image feature value among the image feature values ​​of the training facial images.

[0024] The range of the corresponding image feature values ​​may be a statistical range determined based on the distribution characteristics of the image feature values ​​of the training facial images. The distribution characteristics may include the mean and standard deviation of the image feature values.

[0025] The statistical range may be a range of 1 standard deviation on both sides from the mean value as the center.

[0026] The statistical range may be a range of 2 standard deviations on either side from the mean as the center.

[0027] The statistical range may be a range of 3 standard deviations on either side from the mean as the center.

[0028] The statistical range may be a range of 4 standard deviations on either side from the mean as the center.

[0029] The neural network may be configured to perform liveness detection on facial objects in an input image.

[0030] The neural network may be configured to verify facial objects in an input image.

[0031] The step of calculating the current image feature value may include calculating a hue value of the current facial image. The step of adjusting the current facial image may include, in response to determining that the calculated hue value is outside the range of the corresponding image feature value, adjusting the hue value of the current facial image to be included in the range of the corresponding image feature value, thereby generating an adjusted current facial image.

[0032] The step of calculating the current image feature value may include calculating a facial tilt value indicating a degree of tilt of a face in the current facial image. The step of adjusting the current facial image may include generating an adjusted current facial image by rotating a facial region of the current facial image so that the tilt value of the adjusted current facial image is included in the range of the corresponding image feature value in response to determining that the calculated facial tilt value is outside the range of the corresponding image feature value.

[0033] The step of calculating the current image feature value may include calculating a white balance value of a facial area in the current facial image. The step of adjusting the current facial image may include generating an adjusted current facial image by adjusting the white balance value of the current facial image so that the adjusted white balance value of the current facial image is included in the range of the corresponding image feature value in response to determining that the calculated white balance value is outside the range of the corresponding image feature value.

[0034] In another general aspect, a device includes: one or more processors configured to: detect a facial region in an input image; measure characteristic information of the detected facial region; in response to the measured characteristic information being determined to not satisfy a condition, generate an adjusted facial region by adjusting the characteristic information of the facial region; and perform activity detection on the adjusted facial region.

[0035] The device may also include: a memory storing instructions, which, when executed by the one or more processors, configure the one or more processors to: detect a facial region in an input image; measure characteristic information of the detected facial region; in response to the measured characteristic information being determined to not satisfy a condition, generate an adjusted facial region by adjusting the characteristic information of the facial region; and perform activity detection on the adjusted facial region.

[0036] The one or more processors may also be configured to: measure a tone value of the facial region; and in response to determining that the measured tone value is not included in a preset tone value range, perform a step of generating an adjusted facial region by adjusting the tone value of the facial region to a tone value included in the preset tone value range.

[0037] The one or more processors may also be configured to: measure a facial tilt value indicating a degree of tilt of a face included in a facial region; and in response to determining that the measured facial tilt value is not included in a preset facial tilt value range, perform a step of generating an adjusted facial region by adjusting the facial tilt value.

[0038] The one or more processors may also be configured to: measure a white balance value of the facial area; and in response to determining that the measured white balance value is not included in a preset white balance value range, perform the step of generating an adjusted facial area by adjusting the white balance value of the facial area.

[0039] The one or more processors may be further configured to, in response to determining that the measured characteristic information satisfies the condition, perform liveness detection based on the detected facial region without performing the step of generating the adjusted facial region.

[0040] The one or more processors may be further configured to perform liveness detection using a neural network based liveness detection model.

[0041] The device may be any of the following: a smartphone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, and a vehicle starting device.

[0042] In another general aspect, an apparatus using a neural network includes: one or more processors configured to: extract a current facial image of a facial region from an input image; calculate a current image feature value of the current facial image; compare the current image feature value with a range of corresponding image feature values ​​of training facial images; in response to determining that the current image feature value is outside the range of corresponding image feature values, generate an adjusted current facial image by adjusting the current facial image so that the current image feature value of the current facial image is included in the range of corresponding image feature values; and input the adjusted current facial image to the neural network.

[0043] The device may also include: a memory storing instructions, which, when executed by the one or more processors, configure the one or more processors to: extract a current facial image of a facial region from an input image; calculate a current image feature value of the current facial image; compare the current image feature value with a range of corresponding image feature values ​​of training facial images; in response to determining that the current image feature value is outside the range of corresponding image feature values, generate an adjusted current facial image by adjusting the current facial image so that the current image feature value of the current facial image is included in the range of corresponding image feature values; and input the adjusted current facial image to the neural network.

[0044] A plurality of facial images each having an image feature value may be used to train the neural network.

[0045] The range of the corresponding image feature values ​​may be an absolute range determined based on a minimum image feature value and a maximum image feature value among the image feature values ​​of the facial images used to train the neural network.

[0046] The range of the corresponding image feature values ​​may be a statistical range determined based on the distribution characteristics of the image feature values ​​of the facial images used to train the neural network. The distribution characteristics may include a mean value and a standard deviation of the image feature values.

[0047] The neural network may be used to verify facial objects in an input image.

[0048] The one or more processors may be configured to: calculate a hue value of a current facial image; and in response to determining that the calculated hue value is outside a range of a corresponding image feature value, generate an adjusted current facial image by adjusting the hue value of the current facial image to be included in the range of the corresponding image feature value.

[0049] The one or more processors may be configured to: calculate a facial tilt value indicating a degree of tilt of a face in a current facial image; and in response to determining that the calculated facial tilt value is outside a range of corresponding image feature values, generate an adjusted current facial image by rotating a facial region of the current facial image so that the tilt value of the adjusted current facial image is included in the range of corresponding image feature values.

[0050] The one or more processors may be configured to: calculate a white balance value for a facial area in a current facial image; and in response to determining that the calculated white balance value is outside a range of a corresponding image feature value, generate an adjusted current facial image by adjusting the white balance value of the current facial image to be included in the range of the corresponding image feature value.

[0051] The one or more processors may be configured to: calculate any one or any combination of any two or more of a hue value, a facial tilt value, and a white balance value of a current facial image; and in response to any one or any combination of any two or more of the calculated hue value, the calculated facial tilt value, and the calculated white balance value being respectively determined to be outside a range of a corresponding image feature value, generate an adjusted facial region by adjusting the image feature values ​​outside the range to be within the range of the corresponding image feature value.

[0052] The device may be any of the following: a smartphone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, and a vehicle starting device.

[0053] Other features and aspects will be apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 and Figure 2is a diagram illustrating an example of liveness detection and face verification.

[0055] Figure 3 is a flow chart illustrating an example of a liveness detection method.

[0056] Figure 4 is a flow chart illustrating an example of a face authentication method.

[0057] Figure 5 is a diagram illustrating an example of the operation of a face authentication device.

[0058] Figure 6 is a diagram showing examples of absolute ranges and statistical ranges.

[0059] Figure 7 is a diagram illustrating an example of a preprocessing operation for adjusting color tone.

[0060] Figure 8 is a diagram illustrating an example of a preprocessing operation for adjusting a facial tilt.

[0061] Fig.9A and Fig. 9B is a diagram illustrating an example of a liveness detection method.

[0062] Fig.10 is a diagram illustrating an example of a face authentication device.

[0063] Fig.11 is a diagram illustrating an example of an electronic device.

[0064] Throughout the drawings and detailed description, like reference numerals denote like elements. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0065] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for increased clarity and conciseness, the description of known features may be omitted.

[0066] The features described herein can be implemented in different forms and should not be interpreted as being limited to the examples described herein. Instead, the examples described herein have been provided only to illustrate some of the many possible ways of implementing the methods, devices and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0067] The terms used herein are only for the purpose of describing various examples and will not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "include", "comprising", and "having" indicate the presence of stated features, quantities, operations, components, elements, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0068] In addition, terms such as first, second, A, B, (a), (b), etc. may be used herein to describe components. Each of these terms is not used to limit the nature, order or sequence of the corresponding component but is only used to distinguish the corresponding component from other components.

[0069] Throughout the specification, when an element (such as a layer, a region, or a substrate) is referred to as being “on”, “connected to”, or “coupled to” another element, the element may be “directly on”, “directly connected to”, or “directly coupled to” the other element, or one or more other elements may be present in between. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may not be other elements in between. Similarly, expressions such as “between” and “directly between” and “adjacent to” and “directly adjacent to” may also be interpreted as described above.

[0070] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs and the meanings commonly understood after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meanings in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense.

[0071] Furthermore, in the description of the exemplary embodiments, when it is considered that a detailed description of thus-known structures or functions will lead to obscure interpretation of the exemplary embodiments after understanding the disclosure of the present application, such description will be omitted.

[0072] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals denote like elements throughout.

[0073] Face anti-spoofing technology can be used to determine whether the face of a user input to such a device or system is a fake face or a real face. Such face anti-spoofing technology may include extracting features (such as, for example, local binary pattern (LBP), histogram of directional gradients (HOG), and Gaussian difference (DoG)) from the input image, and determining whether the input face is fake or real based on the extracted features.

[0074] Figure 1 and Figure 2 is a diagram illustrating an example of liveness (also called livingness) detection and face verification.

[0075] Biometrics is one of the user authentication methods that uses personal biological information (including, for example, a person's fingerprint, iris, face, vein, and skin). Among the user authentication methods, the face authentication method can be used to determine whether the user is a valid user and / or not a valid user based on information on the user's face when the user attempts to perform face authentication using his / her face to, for example, unlock a device, log into a device, perform a payment service, and perform an access control operation (as various embodiments).

[0076] Reference Figure 1 , the electronic device 120 determines whether to verify (or authenticate) the user 110 who is trying to connect to the electronic device 120 through facial verification. For example, when the user 110 attempts or is detected or determined to be trying facial verification to cancel the lock mode of the electronic device 120 through the electronic device 120, the electronic device 120 obtains a facial image of the user 110 using an image sensor or an image acquirer (such as a camera 130), analyzes the obtained facial image, and then determines whether to cancel the lock mode of the electronic device 120. In one example, the analysis of the facial image may include: detecting a facial area in the facial image, and extracting features from the detected facial area using a feature extractor. The extracted features may be compared with the registered features of the valid user, and whether the facial verification is successful may be determined based on the result of comparing one or more extracted features with one or more registered features of the valid user. In response to determining that the facial verification is successful, the user 110 may successfully cancel the lock mode (for example, the electronic device 120 may automatically open access to the electronic device 120 according to the result of the comparison). On the contrary, in response to determining that the facial verification is unsuccessful, the electronic device 120 may continue to operate in the lock mode. As another example, when user 110 attempts facial verification through electronic device 120 to use a payment service, electronic device 120 obtains a facial image of user 110, analyzes the obtained facial image, and selectively approves the payment request from user 110 when user 110 is identified as a valid user as a result of analyzing the obtained facial image, otherwise rejects the payment request.

[0077] In one example, the valid user may register the face of the valid user in the electronic device 120 in advance through a face registration process, and the electronic device 120 may store information to be used for identifying the valid user in a storage device or a cloud storage device. In this example, a facial image of the valid user or facial features extracted from the facial image may be stored as an example of the registration information of the valid user.

[0078] In one example, in biometric verification (such as the above-mentioned facial verification), liveness detection may be performed. Liveness detection may be performed before or after determining the result of the biometric verification. Optionally, biometric verification and liveness detection may be performed simultaneously. Liveness detection may be performed to determine whether the test object that is the target of liveness detection is alive. That is, liveness detection may be performed to determine whether the methodology or biometric characteristic used for verification is authentic. For example, liveness detection is performed to determine whether the face in the image captured by the camera 130 is a real face or a fake face. Liveness detection may be performed to distinguish between inanimate objects (e.g., photos, images, paper, and copies that are counterfeit mechanisms or forms) and animate objects (e.g., animate humans) using one or more captured images by distinguishing whether the object in the captured image is an animate object or is more likely to be an animate object, or whether the object is a spoofed object or is more likely to be a spoofed object.

[0079] Figure 2 An example of a fake face 210 and an example of a real face 220 are shown. The electronic device 120 recognizes the real face 220 in a test object image obtained by capturing an image of the real face of a valid user through liveness detection processing. In addition, the electronic device 120 recognizes the fake face 210 in a test object image obtained by capturing an image of the real face of a valid user displayed on the screen of a smartphone or a screen of a personal computer (PC) or printed on paper, or an image of a replica imitating the real face of a user.

[0080] An invalid user may attempt to use spoofing techniques in an attempt to obtain a false acceptance by the user verification system. For example, an invalid user may present a color image, video, replica, etc., in which a valid user's face appears to the camera 130 to obtain a false acceptance in facial verification. Liveness testing may be used to prevent such false acceptance by filtering out or blocking attempts to verify based on such spoofing techniques. When the verification object is determined to be an inanimate object as a result of liveness detection, the verification object may not be allowed to proceed with user verification that compares the verification object with a registered object to verify whether the verification object matches the registered object, or the user verification may ultimately be determined to be unsuccessful regardless of the result of the user verification.

[0081] Return to reference Figure 1 , the electronic device 120 may perform any one of liveness detection and biometric verification, or both liveness detection and biometric verification. As non-limiting examples, the electronic device 120 may be a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, and a vehicle starting device. Here, it is noted that the use of the term "may" with respect to an example or embodiment (e.g., what an example or embodiment may include or may implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples and embodiments are not limited thereto.

[0082] In such a liveness detection process, the electronic device 120 may comprehensively consider various elements of the image captured by the camera 130. For example, the electronic device 120 may determine whether the test object is alive based on at least one of shape information associated with the overall shape of the face in the captured image, texture information associated with the texture of the local face, and context information. The image captured when the object is at a remote location may also include clues about whether the object is a screen or paper of a display device. For example, the image may include the hand of a user holding the display device or paper, which provides clues for determining that the object is fake. Such attempts for false acceptance can also be effectively detected based on context information. In addition, shape information of the overall facial shape can be used to determine the reflection of light and the distortion of shape, and the reflection of light and the distortion of shape can be used to determine the result of liveness detection. As a non-limiting example, such light reflection in a photo image or paper, which is one of the various means for such an attempt, may be different from the light reflection on a real face, and such shape deformation may occur due to wrinkles or folds of the photo image or paper. Liveness detection can be performed based on these aforementioned elements. In addition, subtle differences in texture that can be used to distinguish between real human skin and paper and / or the screen of a display device can be used as texture information. Therefore, the electronic device 120 may comprehensively apply various levels of such activity determination elements to improve the accuracy of activity detection and perform activity detection more robustly.

[0083] For liveness detection or face verification, the neural network may be used. When the neural network is used for liveness detection, the neural network may provide a liveness value based on input data, the liveness value indicating a value, probability value, or feature value indicating whether the test object corresponds to a real face or a fake face. In contrast, when the neural network is used for face verification, the neural network may provide a feature value based on the input data.

[0084] Through training, parameters forming a neural network may be determined. In the training process, there may be a plurality of training data sets and expected values ​​for each training data set. In the training process, the neural network may receive the training data and output a result value corresponding to the received training data by calculation or operation based on the parameters. Loss may be calculated based on the difference between the output result value and the expected value. The parameters forming the neural network may be adjusted to reduce the loss, and the neural network may be trained as described above. The training process may be repeatedly performed for each training data set, and thus the parameters of the neural network may be adjusted to be more desired.

[0085] In an example, in the training process, the training data may be configured to include various data sets so that the neural network learns or trains various cases. However, it may not always be possible to cover all cases through the training data or include all cases in the training data. Therefore, in actual liveness detection or face verification, there may be cases in the training process that are not learned or trained or not fully learned or trained. The cases are also referred to as corner cases herein. Corner cases are not fully learned or trained, so the results of processing such corner cases may be inaccurate.

[0086] In addition, the amount of training data corresponding to such a corner case may be relatively small compared to the training data corresponding to the general case. Therefore, when training is performed using the training data corresponding to the corner case, the performance of the neural network may be reduced. When the training data corresponding to the corner case is used in an actual training process, the data variation of the training data may increase, and thus it may not be easy to train the neural network to output a desired result value.

[0087] However, according to one or more example embodiments to be described below, although liveness detection and / or face verification are performed in such corner cases, the corner cases may be transformed into fully or completely trained cases in the training process of the neural network through a preprocessing operation based on the type of the corner case, and liveness detection and / or face verification may be robustly performed for the corner cases. Therefore, the accuracy of the result of processing the corner case may be improved.

[0088] Figure 3 is a flow chart showing an example of a liveness detection method. The liveness detection method to be described below may be performed by a face authentication device.

[0089] Reference Figure 3In operation 310, the face verification device detects a face area in an input image. The input image refers to an image input to the face verification device, which is a target for which liveness detection is to be performed. The input image may be obtained by an image acquisition device (such as a digital camera, a video camera, etc.). The image acquisition device may be included in the face verification device, or may exist outside the face verification device.

[0090] The face verification device may detect the face region using a neural network as a non-limiting example, the neural network being configured to detect the face region in the input image by using a Haar-based cascade adaboost classifier, a Viola-Jones detector, etc. in various examples. However, the examples are not limited to those described in the foregoing, and the face verification device may detect the face region in the input image using various face region detection methods. As a non-limiting example, the face verification device may detect face landmarks in the input image, and detect a boundary region including the detected face landmarks as the face region.

[0091] In operation 320, the facial verification device measures characteristic information of the facial region. The characteristic information may be a value indicating one or more image features of the facial region and / or one or more facial objects or image-related features in the facial region. The facial verification device may measure characteristic information such as, as a non-limiting example, a hue of the facial region, a facial tilt indicating a degree of tilt of a face included in the facial region, and a white balance, brightness, gamma value, etc. of the facial region. The facial verification device may measure at least one of such various values ​​described in the foregoing.

[0092] Characteristic information such as hue, white balance, brightness, and gamma value of a facial region may be measured using an image quality assessment method for assessing the quality of an image.

[0093] For characteristic information associated with facial inclination, the facial verification device may detect feature points corresponding to the left eye, the right eye, and both ends of the mouth in the facial region, and measure the facial inclination based on the detected feature points. In one example, the facial verification device measures the facial inclination based on an angle formed between a reference line (e.g., a vertical line or a horizontal line) and a determined straight line passing through a determined first center point between the feature points of the left eye and the right eye and a determined second center point between the feature points corresponding to both ends of the mouth.

[0094] In operation 330, the facial verification device determines whether the measured characteristic information satisfies a condition. In one example, the facial verification device may determine whether the characteristic information calculated for the facial region in the input image is included in the feature value range of the facial image used to train the neural network-based liveness detection model. Each of the facial images used for training may have characteristic information (or feature values), and the feature value range may be determined based on the corresponding feature values ​​of the facial image. Here, the characteristic information may be a feature value, and therefore, the term "characteristic information" will be used interchangeably with the term "feature value". As a non-limiting example, the feature value range may be an absolute range defined as a range between a minimum feature value and a maximum feature value among the feature values ​​of the facial image, or a statistical range determined by a distribution characteristic (e.g., the mean and standard deviation of the feature values ​​of the facial image). The statistical range may be wider or narrower than the absolute range based on the standard for defining the absolute range.

[0095] For example, for characteristic information associated with the hue of a facial region, the facial verification device may determine whether at least one of the hues (e.g., red (R) elements, green (G) elements, and blue (B) elements) is included in a preset range or respective preset ranges. For characteristic information associated with facial tilt, the facial verification device may determine whether the facial tilt is included in a preset facial tilt range. For characteristic information associated with white balance, brightness, or gamma value, the facial verification device may determine whether such characteristic information is included in a preset range or respective ranges. The conditions may be set before activity detection, or may be changed during or in the middle of activity detection.

[0096] In operation 340, in response to determining that the measured characteristic information does not satisfy one or more conditions, the facial verification device adjusts the characteristic information of the facial region. In one example, the facial verification device may adjust the characteristic information of the facial region so that the characteristic information measured from the facial region of the input image is included in one or more corresponding feature value ranges of the facial image used to train the activity detection model. For example, the feature value range may be an absolute range or a statistical range. The fact that the characteristic information of the facial region is not included in the feature value range may indicate that the input image as the current activity test object corresponds to a corner case. Here, the facial verification device may preprocess the facial region so that the characteristic information of the facial region is included in the feature value range of the facial image used for training, thereby improving the accuracy of the result of the activity detection. Here, preprocessing may be understood as processing performed or input before, for example, performing an inference operation of the activity detection model on the input information using the result of a preprocessor.

[0097] For example, when the measured hue of the facial region is not included in the preset range, the facial verification device adjusts the hue of the facial region to a hue included in the range. The facial verification device adjusts the hue of the facial region by applying a hue adjustment model to the facial region. The hue adjustment model may perform a function of changing the hue of the facial region to an average hue of a training image as a non-limiting example. Through such hue adjustment, the color distribution (e.g., the distribution of RGB elements) of the facial region may be changed to be similar to the color distribution of the training image.

[0098] For another example, when the measured facial tilt is not included in the preset range, the facial verification device adjusts the facial tilt. The facial verification device adjusts the facial tilt by rotating the facial region of the input image based on the measured facial tilt. The facial verification device rotates the facial region by a certain angle so that the facial tilt is included in the range.

[0099] For another example, when the measured white balance, brightness and / or gamma values ​​are not included in a preset range, the facial verification device adjusts the white balance, brightness and / or gamma values ​​of the facial area so that the white balance, brightness and / or gamma values ​​of the facial area are included in the range.

[0100] In operation 350, when the characteristic information is adjusted in operation 340, the face verification device performs liveness detection on the adjusted facial region detected in operation 310, for example, using the adjusted characteristic information. In response to determining in operation 330 that the characteristic information satisfies the condition, the face verification device does not adjust the characteristic information, and performs liveness detection based on the facial region detected in operation 310.

[0101] The activity detection model may be, for example, a neural network configured to output a value calculated according to an internal parameter of the neural network based on input data. The activity detection model may provide an activity value based on the input data, the activity value indicating a value, a probability value, or a feature value indicating whether the facial object as the test object is a real face or a fake face. That is, the activity value may be a value that will be used as a reference for determining the activity of the object. As a non-limiting example, the activity detection model may be based on a deep convolutional neural network (DCNN) model. For example, the DCNN model may include one or more convolutional layers, one or more pooling layers, and a fully connected layer. The DCNN model may provide information that will be used to determine the activity based on the image information input to the activity detection model through the respective operations performed by each layer. The image information may be a pixel value (as a non-limiting example, a color value and / or a brightness value) of a pixel included in the image. However, the DCNN model is provided only as an example, so the activity detection model may be based on a neural network model of another architecture that is different from the architecture of the DCNN model.

[0102] In one example, a facial verification device determines a first liveness value based on a first image corresponding to a detected facial region. When image information of the first image is input to a first liveness detection model, a first liveness value is obtained, and the first liveness detection model outputs a first liveness value corresponding to the image information of the first image. The first image may include shape information of the entire area of ​​the detected facial region. The first liveness value determined based on the first image may be associated with features or characteristics of light reflection and shape deformation, and the features or characteristics of light reflection and shape deformation are used to identify attempts to obtain false acceptance using, for example, a counterfeit image.

[0103] The facial verification device determines a second activity value based on a second image corresponding to a partial area of ​​the detected facial area. The partial area may be obtained by extracting a smaller area from the detected facial area. The partial area may be randomly extracted from the detected facial area, or an area of ​​a preset size based on the center of the detected facial area may be extracted as the partial facial area.

[0104] When the image information of the second image is input to the second liveness detection model, a second liveness value is obtained, and the second liveness detection model outputs a second liveness value corresponding to the image information of the second image. The second image may include texture information of the local facial area. The second liveness value determined based on the second image may be associated with subtle differences in texture, which are used to identify attempts to obtain false acceptance using, for example, a fake image.

[0105] The facial verification device determines a third liveness value based on a third image corresponding to the entire area of ​​the input image. When image information of the third image is input to a third liveness detection model, a third liveness value is obtained, and the third liveness detection model outputs a third liveness detection value corresponding to the entire area of ​​the input image. The third liveness value determined based on the input image can be associated with a contextual feature or characteristic for identifying an attempt to obtain a false acceptance using, for example, a fake image.

[0106] In this example described in the foregoing, the first activity value, the second activity value, and the third activity value may be reference values ​​to be used to determine the activity of the test object, and may be determined by respective activity detection models. In addition, when the characteristic information of the facial region is adjusted, the first activity value, the second activity value, and the third activity value may be determined based on the first image, the second image, and the third image of the facial region to which the adjusted characteristic information is applied, respectively.

[0107] Optionally, the first activity value, the second activity value, and the third activity value may be determined by a single activity detection model. As a non-limiting example, image information of the first image, image information of the second image, and image information of the entire area of ​​the input image may be input to the single activity detection model, and the single activity detection model may output the first activity value, the second activity value, and the third activity value from at least one output layer.

[0108] The facial verification device determines whether the test object is alive based on the first activity value, the second activity value, and the third activity value. In one example, the facial verification device determines the final activity value based on all the first activity values, the second activity values, and the third activity values. As a non-limiting example, the facial verification device may determine the sum or average of the first activity value, the second activity value, and the third activity value as the final activity value. For another example, the facial verification device may apply a weight to at least one of the first activity value, the second activity value, and the third activity value, and determine the final activity value based on the applied result (e.g., a weighted sum). As the weight, a preset constant may be used, or different weights may be applied to the first activity value, the second activity value, and the third activity value. Optionally, the weight may be a conditional weight determined based on a condition (e.g., a distance to the test object, image quality, the size of the facial area, the facial posture in the facial area, the position of the facial area, the presence or absence of an occlusion area in the facial area, and the lighting state of the facial area).

[0109] In another example, the facial verification device determines a final activity value based on at least one of the first activity value, the second activity value, and the third activity value. As a non-limiting example, the final activity value may be determined based on the first activity value, the second activity value, the third activity value, or any combination thereof.

[0110] The facial verification device determines whether the test object is alive based on whether the final liveness value satisfies a preset condition. As a non-limiting example, in response to the final liveness value being greater than a preset threshold, the facial verification device determines the test object as alive or real. Conversely, in response to the final liveness value being less than or equal to the threshold, the facial verification device determines the test object as inanimate or fake.

[0111] In one example, the face verification device may perform a control operation in response to a result of performing a liveness test on the object. In response to finally determining that the test object is a living object, the face verification device may generate a control signal for requesting execution of a user verification process. However, in response to finally determining that the test object is a fake object that is not a living, real object, the face verification device may generate a control signal for blocking access by the user without requesting execution of a user verification process.

[0112] Figure 4is a flowchart showing an example of a face authentication method. The face authentication method to be described below may be performed by a face authentication device.

[0113] Reference Figure 4 In operation 410, the facial verification device extracts a current facial image of the facial region from the input image. Figure 3 As described in operation 310 of , the face verification apparatus may detect a face region in an input image and obtain a current face image that is an image of the detected face region.

[0114] In operation 420, the facial verification device calculates a current image feature value of the current facial image. The current image feature value may correspond to characteristic information of the current facial image. In one example, the facial verification device calculates an image feature value (e.g., a hue value of the current facial image, a facial tilt indicating a degree of tilt of a face in the current facial image, and / or a white balance of a facial region in the current facial image).

[0115] In operation 430, the face verification device determines whether the current image feature value satisfies a defined condition. In one example, the face verification device compares the current image feature value with a range of corresponding image feature values ​​based on a plurality of face images used to train the neural network, and determines whether the current image feature value is included in the range. The neural network may be a neural network for verifying a face object in an input image, or a neural network for performing a liveness test on a face object.

[0116] The range of image feature values ​​based on the facial images used for training may correspond to the above reference Figure 3 The feature value range described herein can be defined as an absolute range or a statistical range according to the example. For the absolute range, the size of the range can be determined by the minimum image feature value and the maximum image feature value among the image feature values ​​of the facial images used for training. For the statistical range, the size of the range can be determined based on the mean value and the standard deviation of the image feature values ​​of the facial images used for training. For example, the statistical range can be defined as a range of N standard deviations from the mean value as the center on both sides, where N represents a positive integer.

[0117] In operation 440, when the current image feature value does not satisfy the defined condition, or when the current image feature value is out of range, the facial verification device adjusts the current facial image so that the image feature value of the current facial image is included in the range. For example, when the hue value of the current facial image is out of range, the facial verification device adjusts the hue value of the current facial image so that the hue value of the current facial image is included in the range. For another example, when the facial tilt of the facial object in the current facial image is out of range, the facial verification device rotates the facial area of ​​the current facial image so that the facial tilt is included in the range. For another example, when the white balance of the current facial image is out of range, the facial verification device adjusts the white balance of the current facial image so that the white balance of the current facial image is included in the range.

[0118] In operation 450, the facial verification device determines a facial verification result. When the current facial image is adjusted in operation 440, the facial verification device inputs the adjusted current facial image to the neural network. However, when operation 440 is not performed, the facial verification device inputs the current facial image extracted in operation 410 to the neural network. The neural network outputs a feature value corresponding to the input information. The facial verification device determines the similarity between the feature value obtained from the neural network and the feature value pre-registered by the valid user, and determines that the facial verification is successful when the similarity is greater than a threshold. However, when the similarity is less than or equal to the threshold, the facial verification device determines that the facial verification is unsuccessful. The registered feature value of the valid user can also be determined by the neural network.

[0119] Figure 5 is a diagram illustrating an example of the operation of a face authentication device.

[0120] Reference Figure 5 , the face verification device 500 may perform liveness detection and / or face verification. The face verification device 500 includes a face region detector 510, a preprocessor 520, and a verifier 550. The face region detector 510 detects a face region in an input image. The face region detector 510 detects a face region using a cascaded adaboost classifier and a Viola-Jones detector as a non-limiting example.

[0121] The preprocessor 520 performs a preprocessing process to improve the accuracy of the result of the liveness detection. The preprocessor 520 includes a feature value measurer 530 and a feature value adjuster 540. The feature value measurer 530 measures characteristic information or image feature values ​​(hereinafter, feature values) (e.g., hue, white balance, brightness and gamma values ​​of a face region, facial tilt indicating the degree of tilt of a face included in the face region, etc.).

[0122] The feature value adjuster 540 determines whether the feature value measured by the feature value measurer 530 satisfies a preset condition. In response to the measured feature value not satisfying the condition, the feature value adjuster 540 adjusts the feature value of the facial region so that the feature value satisfies the condition. In one example, the feature value adjuster 540 determines whether the measured feature value is included in the range of feature values ​​of the facial image used for training the neural network. The feature value adjuster 540 adjusts the feature value of the facial region so that the feature value measured from the facial region in the input image is included in the range of feature values ​​of the facial image used for training the neural network.

[0123] For example, when a feature value associated with, for example, the hue, white balance, brightness, and / or gamma value of a facial region is not included in a preset range, the feature value adjuster 540 adjusts the feature value so that the feature value is included in the range. For another example, when the facial tilt is not included in the preset range, the feature value adjuster 540 rotates the facial region so that the facial tilt is included in the range. When the feature value measurer 530 measures a plurality of feature values, and one or more of the measured feature values ​​do not satisfy a condition, the feature value adjuster 540 adjusts the one or more feature values ​​that do not satisfy the condition.

[0124] When the feature values ​​are adjusted, the verifier 550 performs liveness detection and / or face verification on the test subject based on the facial region having the adjusted feature values, and provides a result of the liveness detection and / or face verification.

[0125] For example, the verifier 550 uses a neural network-based liveness detection model to determine a liveness value, and determines whether the test object is alive based on the determined liveness value. In this example, information (e.g., pixel values) of a facial region with adjusted feature values ​​is input to the liveness detection model, and the liveness detection model outputs a liveness value corresponding to the input information. The verifier 550 compares the output liveness value with a threshold value and determines a result of liveness detection on the test object. When the feature value measured by the feature value measurer 530 satisfies a condition (e.g., when the measured feature value is included in a defined feature value range), the feature value adjuster 540 does not adjust the feature value, and the verifier 550 immediately performs liveness detection based on the facial region detected by the facial region detector 510.

[0126] In one example, when the test object is determined to be a real object as a result of liveness detection, face verification is performed. In this example, when the feature value measured by the feature value measurer 530 does not satisfy the condition (for example, when the measured feature value is not included in the defined feature value range), the verifier 550 inputs the facial region having the feature value adjusted by the feature value adjuster 540 to the neural network to extract the feature value. When the feature value measured by the feature value measurer 530 satisfies the condition (for example, when the measured feature value is included in the defined feature value range), the verifier 550 inputs the facial region for which the feature value is not adjusted to the neural network to extract the feature value.

[0127] The verifier 550 compares the feature value obtained or output from the neural network with the feature value of the pre-registered valid user, and determines that the face verification is successful in response to the similarity between the two feature values ​​being greater than a threshold value. However, in response to the similarity being less than or equal to the threshold value, the verifier 550 determines that the face verification is unsuccessful.

[0128] The preprocessor 520 transforms the corner cases that have not been learned or trained or not fully learned or trained in the training process of the neural network for liveness detection and / or face verification into fully trained cases through preprocessing operations, thereby improving the accuracy of processing corner cases. Here, because there is no training data corresponding to the corner cases, or there is a small amount of training data, it may not be easy to train the neural network. However, the preprocessing operation of the preprocessor 520 can achieve robust execution of liveness detection and / or face verification for such corner cases while maintaining the basic recognition performance of the neural network.

[0129] Figure 6 is a diagram showing examples of absolute ranges and statistical ranges.

[0130] A predefined range may be used to determine whether a facial region in an input image is suitable or desirable for liveness detection or facial verification. The range may correspond to a range of image feature values ​​of a plurality of facial images (hereinafter, training images) used to train a neural network for obtaining liveness values ​​or feature values. Each training image may have an image feature value, and the range may be determined based on a distribution of corresponding image feature values ​​of the training images. For example, when the image feature value of the input image is outside the range (e.g., when the image feature value is not included in the range), a preprocessing operation may be performed to adjust the input image so that the image feature value of the input image is included in the range.

[0131] In one example, the range may be an absolute range defined as a range between a minimum image feature value and a maximum image feature value among the image feature values ​​of the training images. In another example, the range may be a statistical range determined based on distribution characteristics (e.g., mean and standard deviation) of the image feature values ​​of the training images.

[0132] Reference Figure 6 , curve 600 shows the distribution of image feature values ​​of the training image. In curve 600, the absolute range corresponds to a range R5 between the minimum image feature value and the maximum image feature value among the image feature values. According to the example, the statistical range can be determined in various ways. For example, the statistical range corresponds to a range R1, a range R2, a range R3, or a range R4, and the range R1 indicates a range of one standard deviation σ from the mean value m as the center on both sides of the image feature values ​​of the training image, the range R2 indicates a range of two standard deviations from the mean value m on both sides, the range R3 indicates a range of three standard deviations from the mean value m on both sides, and the range R4 indicates a range of four standard deviations from the mean value m on both sides.

[0133] Figure 7 is a diagram illustrating an example of a preprocessing operation for adjusting color tone.

[0134] Reference Figure 7 , the face region 710 in the input image has a hue having an abnormal value. The feature value measurer 530 measures the hue of the face region 710. For example, the feature value measurer 530 measures a feature value corresponding to the hue based on a ratio of a blue B color element to a green G color element of the face region 710. The feature value adjuster 540 determines whether the measured feature value is within a preset range or a threshold range.

[0135] In response to the feature value not being included in the threshold range, the feature value adjuster 540 performs preprocessing on the face region 710. The feature value adjuster 540 obtains the face region 720 through the preprocessing, through which the feature value adjuster 540 adjusts the hue of the face region 710 to a hue included in the threshold range by applying the hue adjustment model to the face region 710. The verifier 550 performs liveness detection and / or face verification based on the face region 720 obtained through the preprocessing.

[0136] Figure 8 is a diagram illustrating an example of a preprocessing operation for adjusting a facial tilt.

[0137] Reference Figure 8, the facial region 810 in the input image includes a face having an abnormal tilt value or a facial tilt having an abnormal value. The face of the facial region 810 is tilted clockwise at a certain angle. When the facial region 810 is input, the feature value measurer 530 measures the facial tilt of the facial region 810. For example, the feature value measurer 530 detects feature points corresponding to the left eye, the right eye, and both ends of the mouth in the facial region 810, and measures the facial tilt based on an angle formed between a reference line (e.g., a vertical line or a horizontal line) and a determined straight line, the determined straight line passing through a determined first center point between the feature points corresponding to the left eye and the right eye and a determined second center point between the feature points corresponding to both ends of the mouth.

[0138] The feature value adjuster 540 determines whether the measured facial inclination is included in a preset range or a threshold range. In response to the facial inclination not being included in the range, the feature value adjuster 540 adjusts the facial inclination of the facial region 810. The feature value adjuster 540 obtains the facial region 820 through preprocessing, through which the feature value adjuster 540 rotates the facial region 810 by a certain angle so that the facial inclination is included in the range. The verifier 550 performs liveness detection and / or face verification based on the facial region 820 obtained through preprocessing.

[0139] Fig.9A and Fig. 9B is a diagram illustrating an example of a liveness detection method.

[0140] Reference Fig.9A , an entire facial region 920 is detected in an input image 910 including a test object, and a local facial region 930 including a portion of the entire facial region 920 is determined.

[0141] In one example, image information of a first image 925 corresponding to the entire facial region 920 is input to a first liveness detection model 940, and the first liveness detection model 940 outputs a first liveness value corresponding to the first image 925. The first image 925 may be an image obtained by performing image normalization (as a non-limiting example, resizing and affine transformation) on the entire facial region 920. In addition, image information of a second image 935 corresponding to a partial facial region 930 is input to a second liveness detection model 950, and the second liveness detection model 950 outputs a second liveness value corresponding to the second image 935. The second image 935 may be an image obtained by performing image normalization on the partial facial region 930. In addition, image information of a third image 915 corresponding to the entire region of the input image 910 is input to a third liveness detection model 960, and the third liveness detection model 960 outputs a third liveness value corresponding to the third image 915. The third image 915 may be the same as the input image 910, or may be an image obtained by performing image normalization on the input image 910.

[0142] In this example, a preprocessing operation may be performed to adjust characteristic information or feature values ​​of the facial area so that the feature values ​​of the facial area meet preset conditions, and activity values ​​(e.g., first activity value, second activity value, and third activity value) may be determined based on images corresponding to the preprocessed images (e.g., first image, second image, and third image).

[0143] The activity determiner 970 determines a final activity value based on the first activity value, the second activity value, and the third activity value, and determines whether the test object is alive based on whether the determined final activity value satisfies a preset condition. As a non-limiting example, the final activity value may be a sum, an average, or a weighted sum of the first activity value, the second activity value, and the third activity value.

[0144] Fig.9A The first activity detection model 940, the second activity detection model 950, and the third activity detection model 960 shown in FIG. Fig. 9B The single liveness detection model 980 shown in the example of FIG. Fig. 9BIn the example of , image information of the first image 925, image information of the second image 935, and image information of the third image 915 are input to at least one input layer of the activity detection model 980. For example, the image information of the first image 925, the image information of the second image 935, and the image information of the third image 915 are input to different input layers of the activity detection model 980, or are combined with each other to be input to a single input layer of the activity detection model 980. Then, the activity detection model 980 outputs activity values ​​corresponding to the first image 925, the second image 935, and the third image 915, respectively, and the activity determiner 990 determines whether the test object is alive based on whether the activity values ​​satisfy a preset condition.

[0145] Fig.10 is a diagram illustrating an example of a face authentication device.

[0146] The facial verification device 1000 may perform facial verification on an object included in an input image. The facial verification device 1000 may perform liveness detection during facial verification, and provide a result of facial verification based on a result of liveness detection. The facial verification device 1000 may perform one or more or all operations or stages or methods described herein with respect to liveness detection or facial verification, and provide a result of liveness detection or a result of facial verification to a user.

[0147] Reference Fig.10 , the face verification device 1000 includes a processor 1010 and a memory 1020. The memory 1020 may be connected to the processor 1010, and stores instructions executable by the processor 1010, data to be processed by the processor 1010, and / or data processed by the processor 1010.

[0148] The processor 1010 may execute the above reference Figures 1 to 9B One or more or all of the operations or stages or methods described with respect to liveness detection and / or face verification. In this example, for liveness detection and / or face verification, processor 1010 may use a network trained using a plurality of facial images, each facial image having image feature values. In this example, a neural network may be used as a network that performs liveness detection or verification of facial objects in an input image.

[0149] For liveness detection, the processor 1010 may detect a face region as a test subject for liveness detection in an input image, and measure characteristic information (or, hereinafter, feature values) of the detected face region. The processor 1010 may measure feature values ​​(as non-limiting examples, hue, white balance, brightness, and gamma values ​​of the face region, and facial tilt of a face included in the face region).

[0150] The processor 1010 may determine whether the measured feature value satisfies a condition. In response to determining that the feature value does not satisfy the condition, the processor 1010 may perform preprocessing to adjust the feature value. For example, when a feature value (such as a feature value of hue, white balance, brightness, and / or gamma value of a facial region) is not included in a preset range, the processor 1010 may adjust the feature value of the facial region so that the feature value is included in the range. For another example, when a feature value (such as facial tilt) is not included in a preset range, the processor 1010 may rotate the facial region so that the facial tilt is included in the range. The processor 1010 may then perform activity detection based on the facial region with the adjusted feature value.

[0151] Furthermore, in response to determining that the feature value of the facial region satisfies the condition, the processor 1010 may perform liveness detection based on the facial region detected in the input image without adjusting the feature value.

[0152] In one example, the processor 1010 may use an activity detection model based on a trained neural network to obtain an activity value, compare the obtained activity value with a threshold, and determine the result of the activity detection based on the comparison result. For example, in response to the activity value being greater than the threshold, the processor 1010 may determine that the test object is a real object. In this example, in response to the activity value being less than or equal to the threshold, the processor 1010 may determine that the test object is a fake object.

[0153] For face verification, the processor 1010 may extract a current face image of a face region from the input image and calculate a current image feature value of the current face image. The processor 1010 may compare the current image feature value with a range of corresponding image feature values ​​of face images used to train a neural network and determine whether the current image feature value is included in the range.

[0154] In one example, the range of the image feature value may be an absolute range determined based on the minimum image feature value and the maximum image feature value among the image feature values ​​of the facial image used to train the neural network. In the absolute range, the minimum value may be the minimum image feature value, and the maximum value may be the maximum image feature value.

[0155] In another example, the range of the image feature values ​​may be a statistical range to be determined based on the distribution characteristics of the image feature values ​​of the facial images used to train the neural network. The distribution characteristics may include, for example, the mean and standard deviation of the image feature values.

[0156] When the current image feature value is outside the range, the processor 1010 may adjust the current facial image so that the image feature value of the current facial image is included in the range, and input the adjusted current facial image to the neural network. Here, the neural network may output a feature value corresponding to the input information.

[0157] For example, in order to adjust the current facial image, the processor 1010 calculates the hue value of the current facial image. In this example, when the calculated hue value is outside the range, the processor 1010 adjusts the hue value of the current facial image so that the hue value of the current facial image is included in the range. For another example, the processor 1010 calculates a facial tilt indicating the degree of tilt of the face in the current facial image. In this example, when the calculated facial tilt is outside the range, the processor 1010 rotates the facial area of ​​the current facial image so that the facial tilt is included in the range. For another example, the processor 1010 calculates the white balance of the facial area in the current facial image. In this example, when the calculated white balance is outside the range, the processor 1010 adjusts the calculated white balance of the current facial image so that the white balance is included in the range.

[0158] The processor 1010 may compare the feature value obtained by the neural network with the pre-registered feature value, and determine the result of the facial verification based on the similarity between the two feature values. For example, in response to the similarity being greater than a threshold, the processor 1010 may determine that the facial verification is successful. In response to the similarity being less than or equal to the threshold, the processor 1010 may determine that the facial verification is unsuccessful.

[0159] The processor 1010 may generate a control signal based on the result of the liveness detection or the result of the face verification. For example, the processor 1010 may generate a control signal for determining whether to allow or block access to the object based on such a result, or a control signal for determining whether to reject or execute a request from the object (e.g., a request to execute a corresponding function).

[0160] Fig.11 is a diagram illustrating an example of an electronic device.

[0161] The electronic device 1100 may perform liveness detection and / or face verification on a test object included in an input image. Figure 1 The electronic device 120 shown in FIG. Fig.10 The functions or operations of the facial verification device 1000 are shown in FIG.

[0162] Reference Fig.11, the electronic device 1100 includes a processor 1110, a memory 1120, a camera 1130, a storage device 1140, an input device 1150, an output device 1160, and a network interface 1170. The processor 1110, the memory 1120, the camera 1130, the storage device 1140, the input device 1150, the output device 1160, and the network interface 1170 can communicate with each other through a communication bus 1180.

[0163] Processor 1110 may perform functions and execute instructions to perform liveness detection and / or face verification. For example, processor 1110 may process instructions stored in memory 1120 or storage device 1140. Processor 1110 may perform the above-referenced instructions. Figures 1 to 10 One or more or all of the operations or stages or methods described.

[0164] The memory 1120 may store information for performing liveness detection and / or facial verification. The memory 1120 may include a computer-readable storage medium or a computer-readable storage device. The memory 1120 may store instructions to be executed by the processor 1110 and information required to perform liveness detection and / or facial verification.

[0165] The camera 1130 may obtain still images, moving images, or video images, or both still images and moving images of a test subject including liveness detection and / or face verification. The image obtained by the camera 1130 may correspond to an input image described herein.

[0166] Storage device 1140 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1140 may store a larger amount of information than memory 1120 and may store information for a relatively long period of time. As non-limiting examples, storage device 1140 may include a magnetic hard disk, an optical disk, a flash memory, and a floppy disk.

[0167] The input device 1150 may receive input from a user (as non-limiting examples, tactile input, video input, audio input, and touch input). The input device 1150 may detect input from a keyboard, a mouse, a touch screen, a microphone, and a user as non-limiting examples, and include other devices configured to transmit the detected input to the electronic device 1100.

[0168] The output device 1160 may provide the user with an output of the electronic device 1100 through a visual channel, an audio channel, or a tactile channel. The output device 1160 may include, as non-limiting examples, a display, a touch screen, a speaker, a vibration generator, and other devices configured to provide output to the user. The network interface 1170 may communicate with an external device through a wired network or a wireless network.

[0169] Liveness detection device, face verification device 500, face verification device 1000, face region detector 510, preprocessor 520, feature value measurer 530, feature value adjuster 540, verifier 550, processor 1010, 1110, memory 1020, 1120, electronic device 120, electronic device 1100, camera 1130, storage device 1140, input device 1150, output device 1160, network interface 1170, communication bus 1180 and herein directed to Figures 1 to 11Other devices, devices, units, modules and other components described are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic modules, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware component can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used for the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware component can have any one or more of different processing configurations, and examples of different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0170] Figures 1 to 11The method for performing the operations described in the present application shown in the is performed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations can be performed by a single processor or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0171] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0172] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above, and any associated data, data files, and data structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0173] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of the features or aspects in each example will be considered applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0174] Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. A method for detecting activity, comprising: Detect facial regions in input images; measuring characteristic information of the detected facial region; in response to the measured characteristic information being determined not to satisfy the condition, generating an adjusted facial region by adjusting the characteristic information of the facial region; and Perform liveness detection on the adjusted facial regions using a neural network-based liveness detection model. wherein the measured characteristic information includes at least one type of characteristic information, each type of characteristic information is provided with a corresponding characteristic value range of the facial image for training the neural network, wherein the at least one type of characteristic information includes any one of a hue value, a brightness value, and a gamma value or any combination of any two or more thereof, and the corresponding characteristic value range includes an absolute range or a statistical range, Among them, the step of generating the adjusted facial area includes: in response to any type of characteristic information measured being determined to be outside the corresponding characteristic value range, the step of generating the adjusted facial area is performed by adjusting the measured any type of characteristic information to be included in the corresponding characteristic value range.

2. The activity detection method according to claim 1, wherein: The step of measuring characteristic information of the facial region includes measuring a hue value of the facial region, The step of generating the adjusted facial region includes: In response to determining that the measured tone value is not included in the preset tone value range, the step of generating the adjusted face region is performed by adjusting the tone value of the face region to a tone value included in the preset tone value range.

3. The activity detection method according to claim 2, wherein: The steps to adjust the tonal value of the facial area include: The tonal value of the facial region is adjusted by applying a tonal adjustment model to the facial region.

4. The activity detection method according to claim 1, wherein: The step of measuring characteristic information of the face region further includes measuring a face tilt value indicating a tilt degree of a face included in the face region, The step of generating the adjusted facial region includes: In response to determining that the measured facial inclination value is not included in the preset facial inclination value range, the step of generating the adjusted facial region is performed by adjusting the facial inclination value.

5. The activity detection method according to claim 4, wherein: The steps to measure facial inclination are: Detecting feature points corresponding to the left eye, the right eye, and both ends of the mouth in the face region; and The facial tilt value is measured based on the detected feature points.

6. The activity detection method according to claim 4, wherein: The steps to adjust the face tilt value include: The facial tilt value is adjusted by rotating the facial region based on the measured facial tilt value.

7. The activity detection method according to claim 1, wherein: The step of measuring characteristic information of the facial region further includes measuring a white balance value of the facial region, The step of generating the adjusted facial region includes: In response to determining that the measured white balance value is not included in the preset white balance value range, the step of generating the adjusted face region is performed by adjusting the white balance value of the face region to a white balance value included in the preset white balance value range.

8. The activity detection method according to claim 1, further comprising: In response to determining that the measured characteristic information satisfies the condition, liveness detection is performed based on the detected facial region without performing the step of generating the adjusted facial region.

9. The activity detection method according to claim 1, wherein: The steps of measuring characteristic information of the facial region include: At least one of a hue value, a facial tilt value, and a white balance value of a facial area is measured, wherein the hue value, the facial tilt value, and the white balance value are respectively provided with a preset hue value range, a preset facial tilt value range, and a preset white balance value range.

10. The activity detection method according to claim 1, wherein: The neural network is used to perform facial verification on the adjusted facial regions.

11. A facial verification method using a neural network, the facial verification method comprising: extracting a current facial image of a facial region from an input image; Calculate the current image feature value of the current facial image; comparing a current image feature value with a range of corresponding image feature values ​​of a facial training image used to train the neural network; In response to determining that the current image feature value is outside the range of the corresponding image feature value, generating an adjusted current facial image by adjusting the current image feature value of the current facial image to be included in the range of the corresponding image feature value; and Inputting the adjusted current facial image into the neural network, The current image feature value includes at least one type of feature information, each type of feature information is set with a corresponding image feature value range, wherein the at least one type of feature information includes any one of a hue value, a brightness value, and a gamma value or any combination of any two or more thereof, and the corresponding image feature value range includes an absolute range or a statistical range, Among them, the step of generating an adjusted current facial image includes: in response to any type of characteristic information measured being determined to be outside the range of the corresponding image characteristic value, the step of generating an adjusted current facial image is performed by adjusting the measured any type of characteristic information to be included in the range of the corresponding image characteristic value.

12. The facial verification method according to claim 11, wherein: The range of the corresponding image feature value is an absolute range determined based on the minimum image feature value and the maximum image feature value among the image feature values ​​of the face training images.

13. The facial verification method according to claim 11, wherein: The range of the corresponding image feature values ​​is a statistical range determined based on the distribution characteristics of the image feature values ​​of the facial training images, and The distribution characteristics include the mean and standard deviation of the image feature values.

14. The facial verification method according to claim 13, wherein: The statistical range is one of a range of 1 standard deviation on either side from the mean value as the center, a range of 2 standard deviations on either side from the mean value as the center, a range of 3 standard deviations on either side from the mean value as the center, and a range of 4 standard deviations on either side from the mean value as the center.

15. The facial verification method according to claim 11, wherein: The neural network is configured to perform liveness detection on facial objects in an input image.

16. The facial verification method according to claim 11, wherein: The neural network is configured to verify facial objects in an input image.

17. The facial verification method according to claim 11, wherein: The steps of calculating the current image feature value include: Calculate the hue value of the current facial image, The step of adjusting the current facial image includes: In response to determining that the calculated tone value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by adjusting the tone value of the current facial image to be included in the range of the corresponding image feature value.

18. The facial verification method according to claim 11, wherein: The step of calculating the current image feature value also includes: calculating a facial inclination value indicating a degree of inclination of a face in a current facial image, The step of adjusting the current facial image includes: In response to determining that the calculated facial tilt value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by rotating the facial region of the current facial image so that the facial tilt value of the adjusted current facial image is included in the range of the corresponding image feature value.

19. The facial verification method according to claim 11, wherein: The step of calculating the current image feature value also includes: Calculate the white balance value of the face area in the current face image, The step of adjusting the current facial image includes: In response to determining that the calculated white balance value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by adjusting the white balance value of the current facial image so that the adjusted white balance value of the current facial image is included in the range of the corresponding image feature value.

20. A device for activity detection, comprising: One or more processors configured to: Detect facial regions in input images; measuring characteristic information of the detected facial region; in response to the measured characteristic information being determined not to satisfy the condition, generating an adjusted facial region by adjusting the characteristic information of the facial region; and Perform liveness detection on the adjusted facial regions using a neural network-based liveness detection model. wherein the measured characteristic information includes at least one type of characteristic information, each type of characteristic information is provided with a corresponding characteristic value range of the facial image for training the neural network, wherein the at least one type of characteristic information includes any one of a hue value, a brightness value, and a gamma value or any combination of any two or more thereof, and the corresponding characteristic value range includes an absolute range or a statistical range, Among them, the step of generating the adjusted facial area includes: in response to any type of characteristic information measured being determined to be outside the corresponding characteristic value range, the step of generating the adjusted facial area is performed by adjusting the measured any type of characteristic information to be included in the corresponding characteristic value range.

21. The apparatus according to claim 20, wherein: The one or more processors are further configured to: Measuring the tonal values ​​of facial areas; and In response to determining that the measured tone value is not included in the preset tone value range, the step of generating the adjusted face region is performed by adjusting the tone value of the face region to a tone value included in the preset tone value range.

22. The apparatus of claim 20, wherein: The one or more processors are further configured to: measuring a facial inclination value indicating a degree of inclination of a face included in the facial region; and In response to determining that the measured facial inclination value is not included in the preset facial inclination value range, the step of generating the adjusted facial region is performed by adjusting the facial inclination value.

23. The apparatus of claim 20, wherein: The one or more processors are further configured to: Measure the white balance value of the facial area; and In response to determining that the measured white balance value is not included in a preset white balance value range, the step of generating an adjusted facial region is performed by adjusting the white balance value of the facial region.

24. The apparatus of claim 20, wherein: The one or more processors are further configured to: In response to determining that the measured characteristic information satisfies the condition, liveness detection is performed based on the detected facial region without performing the step of generating the adjusted facial region.

25. The apparatus of claim 20, wherein: The device is any one of the following: a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant, a set-top box, a home appliance, a biometric door lock, a security device, and a vehicle starting device.

26. A device using a neural network, the device comprising: One or more processors configured to: extracting a current facial image of a facial region from an input image; Calculate the current image feature value of the current facial image; comparing a current image feature value with a range of corresponding image feature values ​​of a facial training image used to train the neural network; In response to determining that the current image feature value is outside the range of the corresponding image feature value, generating an adjusted current facial image by adjusting the current facial image so that the current image feature value of the current facial image is included in the range of the corresponding image feature value; and Inputting the adjusted current facial image into the neural network, The current image feature value includes at least one type of feature information, each type of feature information is set with a corresponding image feature value range, wherein the at least one type of feature information includes any one of a hue value, a brightness value, and a gamma value or any combination of any two or more thereof, and the corresponding image feature value range includes an absolute range or a statistical range, Among them, the step of generating an adjusted current facial image includes: in response to any type of characteristic information measured being determined to be outside the range of the corresponding image characteristic value, the step of generating an adjusted current facial image is performed by adjusting the measured any type of characteristic information to be included in the range of the corresponding image characteristic value.

27. The apparatus of claim 26, wherein: The range of the corresponding image feature value is an absolute range determined based on the minimum image feature value and the maximum image feature value among the image feature values ​​of the face training images used to train the neural network.

28. The apparatus of claim 26, wherein: The range of the corresponding image feature values ​​is a statistical range determined based on the distribution characteristics of the image feature values ​​of the facial training images used to train the neural network, and The distribution characteristics include the mean and standard deviation of the image feature values.

29. The apparatus of claim 26, wherein: The neural network is used to verify facial objects in an input image.

30. The apparatus of claim 26, wherein: The one or more processors are configured to: Calculating the hue value of the current face image; and In response to determining that the calculated tone value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by adjusting the tone value of the current facial image to be included in the range of the corresponding image feature value.

31. The apparatus of claim 26, wherein: The one or more processors are configured to: calculating a facial inclination value indicating a degree of inclination of a face in a current facial image; and In response to determining that the calculated facial tilt value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by rotating the facial region of the current facial image so that the facial tilt value of the adjusted current facial image is included in the range of the corresponding image feature value.

32. The apparatus of claim 26, wherein: The one or more processors are configured to: Calculate the white balance value of the face area in the current face image; and In response to determining that the calculated white balance value is outside the range of the corresponding image feature value, an adjusted current facial image is generated by adjusting the white balance value of the current facial image to be included in the range of the corresponding image feature value.

33. The apparatus of claim 26, wherein: The one or more processors are configured to: calculating any one or any combination of any two or more of a hue value, a facial tilt value, and a white balance value of the current facial image; and In response to any one or any combination of any two or more of the calculated hue value, the calculated facial tilt value, and the calculated white balance value being determined to be outside the range of the corresponding image feature value, an adjusted current facial image is generated by adjusting the image feature values ​​outside the range to be within the range of the corresponding image feature value.

34. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors, configure the one or more processors to perform the method of any one of claims 1 to 19.

Images