Method and apparatus for activity detection

Abstract

Methods and apparatuses for activity detection are disclosed. The processor-implemented method with activity detection includes receiving a plurality of phase images of different phases; generating a plurality of pre-processed phase images by performing a pre-processing including edge enhancement processing on the plurality of phase images of different phases; generating a plurality of difference images based on the plurality of pre-processed phase images; generating a plurality of low-resolution difference images having a lower resolution than the plurality of difference images based on the plurality of difference images; generating a minimum map image based on the plurality of low-resolution difference images; and performing activity detection on an object in the plurality of phase images based on the minimum map image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0093137, filed on July 27, 2020, with the Korean Intellectual Property Office, and Korean Patent Application No. 10-2020-0178770, filed on December 18, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for activity detection. Background Technology

[0003] Biometric identification can be used for security authentication by using a user's fingerprint, iris, facial features, blood vessels, or other biometric characteristics. Because the underlying biometric characteristics used in such authentication rarely change throughout a user's lifetime, they are designed to be unique for each user. Such biometrics also pose a low risk of theft or forgery, generally providing reliable security authentication. Facial recognition or authentication, as a type of biometric identification, can be used to determine whether a user is a valid user based on a face in a still image or moving image (or video). The advantages of facial recognition or authentication include that the target of facial recognition or authentication can be verified without physical contact between the target and the device. Due to its convenience and effectiveness, facial recognition or authentication can be used in various technical fields, including, for example, security authentication, mobile authentication, and multimedia data retrieval. Summary of the Invention

[0004] The present invention is provided in a simplified form to describe the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method for activity detection includes: receiving a plurality of phase images of different phases; generating a plurality of preprocessed phase images by performing preprocessing, including edge enhancement processing, on the plurality of phase images of different phases; generating a plurality of difference images based on the plurality of preprocessed phase images; generating a plurality of low-resolution difference images having a lower resolution than the plurality of difference images based on the plurality of difference images; generating a minimum image based on the plurality of low-resolution difference images; and performing activity detection on objects in the plurality of phase images based on the minimum image.

[0006] The preprocessing steps may include removing noise from the plurality of phase images, and the edge enhancement process may include enhancing the edge regions of objects in each of the plurality of phase images.

[0007] The steps of performing edge enhancement processing may include applying any one or more of the Sobel filter, anisotropic filter, Laplacian filter, Canny edge filter, and neural network processing to each of the plurality of phase images.

[0008] The step of generating the plurality of low-resolution difference images may include: extracting the maximum pixel value of each of a plurality of block regions in the difference images from the difference images; and generating a low-resolution difference image in the plurality of low-resolution difference images based on the extracted maximum pixel value.

[0009] The step of generating the plurality of low-resolution difference images may include: determining the average value of pixel values ​​in pixel regions of each of the plurality of block regions in the plurality of difference images; and generating the low-resolution difference image based on the average value corresponding to the plurality of block regions.

[0010] The plurality of preprocessed phase images may include a first preprocessed phase image and a second preprocessed phase image, and the step of generating the plurality of difference images may include: generating a plurality of shifted second phase images by shifting the second preprocessed phase image by different shift displacements; and generating the plurality of difference images, the plurality of difference images including: a plurality of difference images each indicating the difference between a corresponding pixel value in the plurality of shifted second phase images and the first preprocessed phase image, and a difference image indicating the difference between the pixel values ​​in the second preprocessed phase image and the first preprocessed phase image.

[0011] The steps of generating the minimum image may include: identifying the minimum difference among the differences between corresponding regions of the plurality of low-resolution differential images; and determining the pixel values ​​of the minimum image based on the identified minimum difference.

[0012] The pixel value of the minimum image may correspond to any one of the following: the minimum difference; and the index of the shift displacement corresponding to the low-resolution difference image including the minimum difference.

[0013] The steps for performing activity detection may include obtaining the activity detection result by applying the minimum image and the plurality of phase images to a neural network-based activity detection model.

[0014] The resolution of the multiple phase images applied to the activity detection model can be adjusted to correspond to the resolution of the minimum image.

[0015] The multiple phase images can be obtained using a multiphase detection sensor.

[0016] The method may include performing the step of obtaining the plurality of phase images using a multiphase detection sensor.

[0017] The method may include controlling access to one or more functions of an electronic device based on the results of an active detection performed.

[0018] A non-transitory computer-readable storage medium stores instructions that, when executed by a processor, configure the processor to perform the method.

[0019] In another general aspect, an apparatus with activity detection includes: a processor configured to: generate a plurality of preprocessed phase images by performing preprocessing, including edge enhancement processing, on a plurality of phase images of different phases; generate a plurality of difference images based on the plurality of preprocessed phase images; generate a plurality of low-resolution difference images having a lower resolution than the plurality of difference images based on the plurality of difference images; generate a minimum image based on the plurality of low-resolution difference images; and perform activity detection on objects in the plurality of phase images based on the generated minimum image.

[0020] The processor can be configured to: remove noise from the plurality of phase images in order to perform preprocessing, and enhance the edge regions of objects in each of the plurality of phase images in order to perform edge enhancement processing.

[0021] To generate the plurality of low-resolution differential images, the processor may be configured to: extract the maximum pixel value of each of a plurality of block regions in the differential images from the differential images; and generate a low-resolution differential image from the plurality of low-resolution differential images based on the extracted maximum pixel value.

[0022] To generate the plurality of low-resolution differential images, the processor may be configured to: determine the average value of pixel values ​​in pixel regions of each of a plurality of block regions in the plurality of differential images; and generate the low-resolution differential images based on the average value corresponding to the plurality of block regions.

[0023] To generate a minimum image, the processor can be configured to: identify the minimum difference among the differences between corresponding regions of the plurality of low-resolution differential images; and determine the pixel values ​​of the minimum image based on the identified minimum difference.

[0024] To perform liveness detection, the processor can be configured to obtain liveness detection results by applying a minimum image and the plurality of phase images to a neural network-based liveness detection model, wherein the resolution of the plurality of phase images applied to the liveness detection model is adjusted to correspond to the resolution of the minimum image.

[0025] The device may include: a memory storing instructions that, when executed by a processor, configure the processor to perform the following steps: generating the plurality of preprocessed phase images, generating the plurality of differential images, generating the plurality of low-resolution differential images, generating a minimum image, and performing liveness detection.

[0026] The device may be an electronic device including a multiphase detection sensor configured to acquire the multiple phase images using multiple sensor pixels.

[0027] In another general aspect, an electronic device includes: a multiphase detection sensor configured to acquire a plurality of phase images of different phases; and a processor configured to: generate a plurality of preprocessed phase images by performing preprocessing, including edge enhancement processing, on the plurality of phase images; generate a plurality of low-resolution differential images having a lower resolution than the plurality of preprocessed phase images based on the plurality of preprocessed phase images; generate a minimum image based on the plurality of low-resolution differential images; and perform liveness detection on objects in the plurality of phase images based on the minimum image.

[0028] The processor can be configured to: remove noise from the plurality of phase images for preprocessing; enhance edge regions of objects in each of the plurality of phase images for edge enhancement processing; and generate a plurality of difference images based on the plurality of preprocessed phase images for generating the plurality of low-resolution difference images; extract the maximum pixel value of each of a plurality of block regions in the difference images from the difference images, and generate a low-resolution difference image from the plurality of low-resolution difference images based on the extracted maximum pixel value.

[0029] To generate the plurality of low-resolution differential images, the processor may be configured to: generate a low-resolution preprocessed phase image based on the maximum or average pixel value in the pixel region of each of the plurality of block regions in the preprocessed phase image; and generate a low-resolution differential image of the plurality of low-resolution differential images based on the generated low-resolution preprocessed phase image.

[0030] In another general aspect, a processor-implemented method for liveness detection includes: preprocessing phase images of different phases based on edge regions of objects in the phase images; reducing the resolution of a difference image generated based on the preprocessed phase images; generating a minimum image based on the reduced-resolution difference image; and performing liveness detection on the objects based on the minimum image.

[0031] The preprocessing steps may include: removing noise from the phase image, excluding edge regions; and enhancing the edge regions of the noise-removed phase image.

[0032] Steps to reduce resolution may include performing pooling operations on the difference image.

[0033] Pooling operations can be either max pooling or average pooling.

[0034] In another general aspect, a processor-implemented method for activity detection includes: preprocessing a first phase image and a second phase image based on edge regions of an object in the phase image; generating a shifted image by shifting the second preprocessed phase image by a corresponding shift displacement; generating a difference image based on the difference between the first preprocessed phase image and the corresponding shifted image; reducing the resolution of the difference image; generating a minimum image based on the reduced-resolution difference image; and performing activity detection on the object based on the minimum image.

[0035] The steps for generating a shifted image may include generating a shifted region by shifting a base region in a second phase image by a corresponding shift displacement along one or more directions.

[0036] The steps for generating a difference image may include: determining the difference between the base region and the shifted region in the first preprocessed phase image.

[0037] The method may include controlling access to one or more functions of an electronic device based on the results of an active detection performed.

[0038] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0039] Figure 1 and Figure 2 Examples of activity detection and biomarker verification are shown.

[0040] Figure 3 An example of a multiphase detection sensor is shown.

[0041] Figure 4 An example is shown showing the difference between a two-dimensional (2D) image and a three-dimensional (3D) image that can be detected by phase imaging.

[0042] Figure 5 An example of an activity detection method is shown.

[0043] Figure 6 An example of an activity detection method is shown.

[0044] Figure 7 An example of edge enhancement processing is shown.

[0045] Figure 8 and Figure 9 This shows an example of generating a minimum map image.

[0046] Figure 10A and Figure 10B An example of shifting a phase image is shown.

[0047] Figure 11A and Figure 11B This shows an example of determining the pixel values ​​of a minimum image.

[0048] Figure 12 An example of a device with activity detection is shown.

[0049] Figure 13 An example of a device with activity detection is shown.

[0050] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0051] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known upon understanding this disclosure may be omitted.

[0052] The features described herein may be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0053] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items. As used herein, the terms "comprising," "including," and "having" indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof. As used herein, the use of the term "may" in relation to examples or embodiments (e.g., what an example or embodiment may include or implement) indicates the presence of at least one example or embodiment that includes or implements such a feature, while all examples are not limited thereto.

[0054] Furthermore, terms such as first, second, A, B, (a), (b), etc., may be used herein to describe components according to the example embodiments. Each of these terms is not intended to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from one or more other components. Although the terms "first" or "second" are used herein to describe various elements, components, regions, layers, or portions, these elements, components, regions, layers, or portions should not be limited by these terms. Rather, these terms are only used to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Thus, without departing from the teachings of the examples, the first element, first component, first region, first layer, or first portion referred to in the examples described herein may also be referred to as a second element, second component, second region, second layer, or second portion.

[0055] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" said other element, "connected to," or "bonded to" said other element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, there may be no other elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent" and "closely adjacent" may be interpreted as described above.

[0056] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, and as commonly understood after understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the relevant field and in the context of this disclosure, and shall not be interpreted in an idealized or overly formalized sense.

[0057] Furthermore, in the description of the exemplary embodiments, such descriptions may be omitted when it is believed that a detailed description of the structure or function known therefrom after understanding the disclosure of this application may lead to a vague interpretation of the exemplary embodiments.

[0058] In the following description, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements.

[0059] Figure 1 and Figure 2 Examples of activity (also known as liveness) detection and biometric verification are shown.

[0060] Biometrics is one type of user authentication method that uses personal biometric information, including, for example, an individual's fingerprints, irises, face, veins, and skin. Among user authentication methods, facial verification can be used to determine whether a user is a valid user and / or not when the user attempts to perform facial verification to unlock a device, log in to a device, perform payment services, and perform access control operations (as various examples).

[0061] Reference Figure 1 Electronic equipment 120 (e.g., Figure 13 A device with activity detection (or activity detection device) 1300 can determine whether to verify (or authenticate) an object 110 (e.g., a user) attempting to connect to electronic device 120 via biometric verification (e.g., facial verification). Electronic device 120 can use image sensor 130 (e.g., image sensor 130). Figure 3 Two-phase detection (2PD) sensor 310 or Figure 12The multiphase detection sensor 1240 acquires image data of the object 110, analyzes the acquired image data, and determines the verification result. Such biometric verification may include extracting features from the image data, comparing the extracted features with the registration features of a valid object, and determining whether the verification was successful based on the comparison result. For example, when the verification of the object 110 is determined to be successful while the electronic device 120 is in a locked state, the locked state of the electronic device 120 may be canceled and / or changed to an unlocked state. Conversely, when the verification of the object 110 is determined to be unsuccessful, the electronic device 120 may continue to operate in the locked state. As a non-limiting example, the electronic device 120 may be a smartphone, wearable device, tablet computer, netbook, laptop computer, desktop computer, personal digital assistant (PDA), set-top box, home appliance, biometric door lock, security device, and / or vehicle starting device.

[0062] In one example, an active user can pre-register his / her biometrics in electronic device 120 through a registration process, and electronic device 120 can store the information to be used to identify the active user in a storage device or cloud storage device. In this example, the active user's facial image or facial features extracted from a facial image can be stored as the active user's registered biometrics.

[0063] In such biometric verification, an activity detection or test may be performed. The activity detection may be performed to determine whether the test subject (e.g., subject 110), as the target of the activity detection, is alive. That is, the activity detection may be performed to determine whether the method or biometric characteristic used for verification is genuine. For example, an activity detection may be performed to determine whether a face in an image captured by image sensor 130 is a real face or a fake face. The activity detection may be performed to distinguish between inanimate objects (e.g., photographs, images, paper, videos, and copies as a form of forgery) and living objects (e.g., the face of a living human) using one or more captured images. Electronic device 120 may perform one of activity detection and biometric verification, or both.

[0064] Figure 2 Examples of a fake face 210 and a real face 220 are shown. The electronic device 120 can identify the real face 220 in image data via activity detection. Furthermore, the electronic device 120 can identify a fake face 210 in a test subject image obtained by capturing an image of a valid user's real face displayed on a smartphone screen or personal computer (PC) screen, a printed piece of paper, or an image of a copy mimicking the user's real face.

[0065] Invalid users may attempt to use deception techniques to gain false acceptance from the biometric verification system. For example, an invalid user could present an image, video, copy, etc., of an valid user's face to the image sensor 130 to gain false acceptance in facial verification. One or more embodiments of activity detection can be used to prevent such false acceptance by filtering out or blocking attempts to verify using such deception techniques that employ such alternatives. When the verification object is determined to be inanimate as a result of activity detection, the verification object may not be allowed to continue with user verification that compares it to a registered object to verify whether the verification object matches the registered object, or the user verification may be ultimately determined to be unsuccessful regardless of the result of the user verification.

[0066] Image sensor 130 can detect objects (e.g., Figure 1 The visual information of object 110 is represented as image data of multiple phases. Image sensor 130 can detect visual information of multiple phases and generate image data associated with visual information of each of the multiple phases. Image sensor 130 can be a multiphase detection sensor including, for example, a two-phase detection (2PD) sensor configured to detect two types of phases and a four-phase detection (QPD) sensor configured to detect four types of phases. However, the number of phases that image sensor 130 can detect or sense is not limited to the foregoing examples, and image sensor 130 can detect various numbers of different phases. Although the following examples will be described under the assumption that image sensor 130 corresponds to a 2PD sensor for ease of description, the examples are not limited thereto. The following description also applies to available examples where image sensor 130 is a multiphase detection sensor of a different type.

[0067] Figure 3 An example of a multiphase detection sensor is shown (e.g., the structure of a 2PD sensor 310 as an example of a multiphase detection sensor). See reference... Figure 3 The 2PD sensor 310 may include multiple sensor pixels 315 that sense red, green, and blue (RGB) color values ​​as two different phases. Figure 3 In the example, reference numeral 320 indicates a magnification of each sensor pixel 315. Sensor pixel 315 may include a plurality of sensor subpixels 332, 342, 336, and 346 for sensing green color values, a plurality of sensor subpixels 334 and 344 for sensing red color values, and a plurality of sensor subpixels 338 and 348 for sensing blue color values. In this example, sensor subpixels 332, 334, 336, and 338 may correspond to a first phase (e.g., left (L)), and sensor subpixels 342, 344, 346, and 348 may correspond to a second phase (e.g., right (R)). Figure 3The structure of the 2PD sensor 310 shown is provided as an example only, and the arrangement of the sensor pixels 315 is not limited to this. Figure 3 The example structure is shown in the figure.

[0068] Each sensor sub-pixel included in the 2PD sensor 310 can sense visual information of one of a first phase and a second phase. Each of the sensor sub-pixels 332, 334, 336, 338, 342, 344, 346, and 348 may include a photodiode that receives external light and outputs an electrical signal value based on the external light. By extracting the output values ​​of the photodiodes of the sensor sub-pixels 332, 334, 336, and 338 corresponding to the first phase, a first phase image 350 of the first phase can be obtained. By extracting the output values ​​of the photodiodes of the sensor sub-pixels 342, 344, 346, and 348 corresponding to the second phase, a second phase image 360 ​​of the second phase can be obtained.

[0069] Disparity may exist between the first phase image 350 and the second phase image 360. Such disparity can be used for activity detection of an object (e.g., object 110) in one or more embodiments, thereby improving the accuracy of activity detection. Disparity can represent stereoscopic (or three-dimensional (3D)) information of an object, and therefore, using disparity can effectively distinguish counterfeit or deceptive objects implemented in two dimensions using photographs or paper. Figure 4 This illustrates an example of the differences between 2D and 3D objects detectable through phase imaging. (See reference...) Figure 4 When capturing an image of a 2D object using the 2PD sensor 310, parallax may not be detectable using the first phase image and the second phase image. For example, when capturing a photograph with a face printed on it using the 2PD sensor 310, there may be no depth difference between the facial region and the background region, and there may be no depth difference between facial parts. Therefore, parallax may not be detectable using the first phase image and the second phase image. In contrast, when capturing an image of a 3D object using the 2PD sensor 310, parallax can be detected using both the first phase image and the second phase image. In this example, when capturing an image of an actual human face using the 2PD sensor 310, parallax in the facial contours and / or nose can be detected by comparing the first phase image with the second phase image.

[0070] Return to reference Figure 1Information related to the shape, color, texture, and context of the object 110, corresponding to 2D information, can be extracted from the phase image obtained via the image sensor 130. In addition to the phase image, the electronic device 120 of one or more embodiments can also use parallax information obtained from the phase image for activity detection, and can apply features of the 3D structure of the object 110 to improve the accuracy of activity detection. Parallax information can be obtained from multiple differential images, each indicating the difference in pixel values ​​between the phase images.

[0071] The computational cost and accuracy of activity detection can be contradictory. High levels of accuracy can be achieved by performing activity detection based on full-resolution information from a phase image. However, when the phase image is at full resolution (e.g., megapixel resolution), significant time and resources may be consumed in processing it. Therefore, processing full-resolution phase images may be difficult for mobile platforms with limited resources. Thus, to improve processing speed and reduce the amount of resources (e.g., memory) required for activity detection, one or more embodiments of the electronic device 120 may perform activity detection by converting the phase image to a lower resolution instead of using the original phase image obtained by the image sensor 130 without modification. Converting the phase image to a lower resolution may correspond to reducing the size of the phase image. When the electronic device 120 is a mobile device (such as a smartphone), there may be significant constraints on computational cost and resource consumption; therefore, mitigating activity detection allows it to be performed efficiently on the electronic device 120. Mitigation may include, for example, converting the phase image to a lower resolution. However, converting the phase image to a lower resolution can result in the loss of parallax-related information in the phase image, reducing the accuracy of such parallax information and / or reducing the accuracy of activity detection. Furthermore, the phase image obtained by the image sensor 130 may include a significant amount of noise caused by the movement of the electronic device 120. Some of this noise may hinder the acquisition of accurate parallax information, thereby potentially reducing the accuracy of activity detection.

[0072] By performing edge enhancement processing to enhance or strengthen the edge components or regions of an object in a phase image and by efficiently extracting the principal pixels from the difference image, the activity detection method of one or more embodiments described below enables accurate disparity information to be obtained even at low resolution. Accurate disparity information improves the accuracy of activity detection, thus the activity detection method of one or more embodiments effectively prevents or blocks erroneous acceptances that may be obtained through spoofing techniques. In the case of performing activity detection in a mobile platform (such as a smartphone), the activity detection method of one or more embodiments described herein reduces the computational or operational load and resource requirements for activity detection and enables real-time processing. Furthermore, the activity detection method of one or more embodiments improves accuracy by robustly performing activity detection against various spoofing attacks. The activity detection method will be described in more detail below.

[0073] Figure 5 An example of an activity detection method is shown.

[0074] Reference Figure 5 In operation 510, the activity detection device (e.g., Figure 12 The device 1200 with activity detection can receive multiple phase images of different phases (including, for example, a first phase image of a first phase and a second phase image of a second phase). A multiphase detection sensor (e.g., Figure 12 A multiphase detection sensor 1240 is used to obtain phase images.

[0075] In operation 520, the activity detection device can generate multiple preprocessed phase images (including, for example, a first preprocessed phase image and a second preprocessed phase image) by performing preprocessing, including edge enhancement processing, on the phase images. In the preprocessing, the activity detection device can remove noise from the phase images and perform edge enhancement processing to enhance (or strengthen) edge regions of objects in each phase image. For example, the activity detection device can perform edge enhancement processing by applying a Sobel filter to each phase image. However, the examples are not limited to the aforementioned example of using a Sobel filter to perform edge enhancement processing. For another example, edge enhancement processing can be performed by using any one or more of anisotropic filters, Laplacian filters, and Canny edge filters, or by using a neural network. In one example, edge enhancement processing can be performed by applying any one or more of a Sobel filter, anisotropic filter, Laplacian filter, Canny edge filter, and neural network processing to each phase image. To match phase images, the activity detection methods of one or more embodiments can perform comparisons of edge regions (e.g., facial contours) more easily and accurately than comparisons performed on flat regions (e.g., cheeks of a face and / or regions excluding edge regions) using typical activity detection methods. Therefore, information associated with edge regions can be important and can be used to obtain accurate parallax information. Edge enhancement processing can enhance or strengthen edge regions in the phase image and remove noise from flat regions of an object's shape.

[0076] For example, in operation 520, the activity detection device may perform gamma correction on the phase image before performing edge enhancement processing, and may perform edge enhancement processing on the phase image obtained by performing gamma correction. Gamma correction may use a nonlinear transfer function to correct the brightness values ​​of the phase image and may be performed with low computational cost. The activity detection device may also perform denoising to remove noise from the phase image before performing edge enhancement processing.

[0077] In operation 530, the activity detection device can generate multiple difference images based on a preprocessed phase image. The difference images described herein can represent images indicating the difference in pixel values ​​of corresponding pixel regions between phase images. For example, when the preprocessed phase image includes a first preprocessed phase image and a second preprocessed phase image, the activity detection device can generate multiple shifted second phase images by shifting the second preprocessed phase image by different shift displacements (e.g., where the shift displacement corresponds to shifting multiple pixels in one direction (e.g., to the right or left)). In this example, the activity detection device can generate difference images, each indicating the difference in pixel values ​​between a corresponding pixel in the shifted second phase image and the first preprocessed phase image. The activity detection device can calculate the difference in pixel values ​​of corresponding regions between the first preprocessed phase image and the shifted second phase image, which is obtained by shifting the second preprocessed phase image while the first preprocessed phase image is fixed. The activity detection device can also generate difference images indicating the difference in pixel values ​​between the first preprocessed phase image and the unshifted second preprocessed phase image.

[0078] In operation 540, the activity detection device can generate multiple low-resolution differential images with a lower resolution than the differential image. The activity detection device generates each of the low-resolution differential images, each with a reduced size, by extracting principal pixels from corresponding differential images in the differential images. Principal pixels, as described herein, represent pixels that contain significant information or core signals among the pixels in each differential image. In one example, significant information or core signals include information that is more certain of the activity of an object in the image compared to signals or information from pixels that are not principal pixels. For example, the activity detection device can extract the maximum pixel value from each patch region of the differential image using max pooling and generate a low-resolution differential image based on the extracted maximum pixel value. In this example, the low-resolution differential image may consist of the maximum pixel value extracted from the patch region. For another example, the activity detection device can determine the average pixel value in the pixel region of each patch region in the differential image using average pooling and generate a low-resolution differential image based on the determined average value corresponding to each patch region. In this example, the low-resolution differential image may consist of the average value determined for each patch region. However, the examples are not limited to those using max pooling and average pooling, and other methods may also be applicable. In one example, operation 540 may be performed before operation 530. That is, the activity detection device may generate a low-resolution preprocessed phase image based on the maximum or average pixel value in each pixel region of multiple block regions in each preprocessed phase image, and then the activity detection device may generate multiple difference images based on the generated low-resolution preprocessed phase image.

[0079] In operation 550, the activity detection device can generate a minimummap image based on low-resolution differential images. Similar to a disparity map, the minimummap image described herein can represent an image containing 3D information about an object (e.g., a 3D cue signal). The activity detection device can identify the minimum difference among the differences between corresponding regions of the low-resolution differential images. The activity detection device can determine the pixel values ​​of the minimummap image based on the identified minimum difference. The pixel values ​​of corresponding regions in the minimummap image can correspond to the selected minimum difference, or to the index of the shift displacement corresponding to the low-resolution differential image containing the minimum difference. The index of the shift displacement can represent the displacement value by which the phase image is shifted. For example, in the case of shifting two pixels to the right, the index of the shift displacement can be +2, and in the case of shifting one pixel to the right, the index of the shift displacement can be +1. When there is no shift, the index of the shift displacement can be 0. Furthermore, in the case of shifting two pixels to the left, the index of the shift displacement can be -2, and in the case of shifting one pixel to the left, the index of the shift displacement can be -1.

[0080] In operation 560, the activity detection device can perform activity detection on objects in a phase image based on a minimum image. The activity detection device can achieve this by applying the minimum image and the phase image to a neural network-based activity detection model (e.g., ...). Figure 6 An activity detection model (670) is used to obtain the activity detection results. Here, the resolution of the phase image to be applied to the activity detection model can be adjusted to correspond to the resolution of the minimum image. Minimum images and phase images of the same resolution can form the input block of the activity detection model. Minimum images and phase images can be concatenated to form the input channel of the activity detection model.

[0081] The phase image included in the input block may correspond to the entire region of the original phase image or a portion of the region of interest (RoI) (e.g., the facial region).

[0082] An activity detection model may include at least one neural network pre-trained to detect the activity of an object based on an input block. The neural network may output a value computed by internal parameters in response to input data. At least a portion of the neural network may be implemented by hardware including a neural processor, or a combination of software and hardware. The neural network may be a deep neural network (DNN), including, for example, fully connected networks, deep convolutional networks, and / or recurrent neural networks. A DNN may include multiple layers. The multiple layers may include an input layer, one or more hidden layers, and an output layer. The neural network may be trained to perform a given operation by mapping input and output data in a non-linear relationship based on deep learning. Deep learning may be a machine learning method applied to solve a given problem on a large dataset. Deep learning may be an optimization process of the neural network used to find the point where energy is minimized while being trained using provided training data.

[0083] Neural networks can output a liveness score in response to input data. The liveness score can be represented as a reference value for determining whether an object is alive. It can be expressed as a value, probability value, or feature value indicating whether the object is real or fake. Liveness detection devices can determine an object's liveness based on whether the liveness score meets preset conditions. For example, when the liveness score is greater than a preset threshold, the liveness detection device can identify the object as a real, living object. When the liveness score is less than or equal to the threshold, the liveness detection device can identify the object as a lifeless, fake object.

[0084] As described above, for activity detection, by generating and using a low-resolution phase image instead of the raw phase image obtained through a multiphase detection sensor, the activity detection apparatus of one or more embodiments can reduce the computational complexity and resource consumption for activity detection, thereby enabling real-time processing. Furthermore, by employing preprocessing (e.g., including edge enhancement) and generating a low-resolution difference image (e.g., including pooling) as presented herein in activity detection, the activity detection apparatus of one or more embodiments can improve the accuracy (or confidence) of the minimum image by reducing the loss of 3D information while generating the low-resolution minimum image. For example, the activity detection apparatus of one or more embodiments can improve the accuracy of activity detection by effectively filtering out 2D spoofing attacks.

[0085] Furthermore, the activity detection device can perform control operations in response to the results of activity detection of an object. For example, when it is determined that the object is genuine, the activity detection device can request the execution of a user authentication process. However, when it is determined that the object is counterfeit, the activity detection device can block access by the user without requesting the execution of a user authentication process.

[0086] Figure 6 An example of an activity detection method is shown.

[0087] Reference Figure 6 A first phase image 350 and a second phase image 360 ​​can be obtained from a multiphase detection sensor (such as a 2PD sensor 310). Activity detection devices (e.g., Figure 12 A device 1200 with activity detection can generate a first preprocessed phase image 610 and a second preprocessed phase image 620 by performing preprocessing, including edge enhancement processing, on the first phase image 350 and the second phase image 360, respectively. Edge enhancement processing may include applying filters (such as Sobel filters and / or anisotropic filters) to each of the first phase image 350 and the second phase image 360 ​​to enhance or strengthen edge regions of an object. According to an example, preprocessing may include gamma correction and / or denoising.

[0088] The activity detection device can generate multiple difference images 630 based on a first preprocessed phase image 610 and a second preprocessed phase image 620. The activity detection device can generate a shifted second phase image by shifting the second preprocessed phase image 620 by different shift displacements, and can generate multiple difference images 630, each indicating the difference in pixel values ​​between a corresponding one of the first preprocessed phase image 610 and the shifted second phase image. The difference images 630 may also include a difference image indicating the difference in pixel values ​​between the first preprocessed phase image 610 and the unshifted second preprocessed phase image 620.

[0089] The activity detection device can generate multiple low-resolution difference images 640 by reducing the size of the difference image 630. For example, the activity detection device can generate low-resolution difference images 640 by extracting the principal pixels from each difference image 630 using max pooling.

[0090] The activity detection device can generate a minimum image 650 based on the low-resolution difference image 640. The activity detection device can identify the minimum difference among the differences between corresponding regions of the low-resolution difference images 640 and determine the pixel value of the minimum image 650 based on the minimum difference. The activity detection device can determine the minimum difference as the pixel value of the minimum image 650, or determine the pixel value of the minimum image 650 as the shift index of the low-resolution difference image with the minimum difference.

[0091] An activity detection device can generate a first low-resolution phase image 660 and a second low-resolution phase image 665 by reducing the resolution of the first phase image 350 and the second phase image 360, respectively. The activity detection device can then apply the first low-resolution phase image 660, the second low-resolution phase image 665, and the minimum image 650 to a neural network-based activity detection model 670 to perform activity detection. Instead of analyzing the full-resolution first phase image 350 and the second phase image 360 ​​obtained from the 2PD sensor 310, the reduced-resolution first low-resolution phase image 660, the second low-resolution phase image 665, and the minimum image 650 can be analyzed. Therefore, by generating and using images with reduced resolution, the activity detection device of one or more embodiments can use a mitigated neural network as the activity detection model 670.

[0092] The first low-resolution phase image 660, the second low-resolution phase image 665, and the minimum image 650, which are to be input into the activity detection model 670, may have the same resolution. The first low-resolution phase image 660, the second low-resolution phase image 665, and the minimum image 650 may form an input channel that will be input into the activity detection model 670 through the input layer. In another example, the image size of the minimum image 650 may be reduced such that the minimum image 650 has a defined or predetermined resolution when input into the activity detection model 670. In such an example, the image sizes of the first phase image 350 and the second phase image 360 ​​may also be reduced such that they have the same resolution as the minimum image 650 when input into the activity detection model 670.

[0093] The activity detection model 670 can output an activity score in response to input data. When the activity score meets a predetermined condition (e.g., greater than a threshold), the activity detection device can identify the object as a genuine object. When the activity score does not meet the condition (e.g., less than or equal to the threshold), the activity detection device can identify the object as a fake object.

[0094] Figure 7 An example of edge enhancement processing (e.g., preprocessing) is shown.

[0095] Reference Figure 7 Activity detection equipment (e.g., Figure 12 An activity detection device 1200 can perform edge enhancement processing on each of a first phase image 350 and a second phase image 360. The activity detection device can perform preprocessing by applying a Sobel filter 710 to the first phase image 350 to enhance the edge regions of objects in the first phase image 350. Similarly, the activity detection device can perform preprocessing by applying a Sobel filter 710 to the second phase image 360 ​​to enhance the edge regions of objects in the second phase image 360. Through such preprocessing, a first preprocessed phase image 610 and a second preprocessed phase image 620 with reduced noise and enhanced edge regions are obtained. According to examples, other edge enhancement filters (such as anisotropic filters, Laplacian filters, and / or Canny edge filters) can be used. The edge enhancement processing of one or more embodiments can enable the detection of more accurate parallax information in the edge regions of objects.

[0096] Figure 8 An example of generating a minimum graph image is shown.

[0097] Reference Figure 8 When performing preprocessing including edge enhancement, a difference image 630 can be generated based on the first preprocessed phase image 610 and the second preprocessed phase image 620. In one example, an activity detection device (e.g., Figure 12The device 1200 with activity detection can generate a difference image 630, each indicating the difference between a corresponding pixel value in a first preprocessed phase image 610 and a corresponding pixel value in a shifted second phase image obtained by shifting a second preprocessed phase image 620 by a different shift displacement. In this example, as shown, the difference image 630 includes a first difference image 812, a second difference image 814, a third difference image 816, a fourth difference image 818, and a fifth difference image 820. In one example, a first difference image 812 may indicate the difference in pixel values ​​between a first preprocessed phase image 610 and a shifted second phase image obtained by shifting the second preprocessed phase image 620 two pixels to the left; a second difference image 814 may indicate the difference in pixel values ​​between a first preprocessed phase image 610 and a shifted second phase image obtained by shifting the second preprocessed phase image 620 one pixel to the left; a third difference image 816 may indicate the difference in pixel values ​​between a first preprocessed phase image 610 and an unshifted second preprocessed phase image 620; a fourth difference image 818 may indicate the difference in pixel values ​​between a first preprocessed phase image 610 and a shifted second phase image obtained by shifting the second preprocessed phase image 620 one pixel to the right; and a fifth difference image 820 may indicate the difference in pixel values ​​between a first preprocessed phase image 610 and a shifted second phase image obtained by shifting the second preprocessed phase image 620 two pixels to the right. However, the examples of generating the difference image 630 are not limited to the aforementioned example of shifting the second preprocessed phase image 620. In another example, the shifted first phase image can be generated by shifting the first preprocessed phase image 610 by different shift displacements, and difference images can be generated, each indicating the difference in pixel values ​​between the second preprocessed phase image 620 and each of the shifted first phase images.

[0098] When the difference image 630 is generated, the activity detection device can generate a low-resolution difference image 640 with a lower resolution than the difference image 630. In this example, the activity detection device can generate a first low-resolution difference image 832, a second low-resolution difference image 834, a third low-resolution difference image 836, a fourth low-resolution difference image 838, and a fifth low-resolution difference image 840 by applying max pooling to the first difference image 812, the second difference image 814, the third difference image 816, the fourth difference image 818, and the fifth difference image 820, respectively. Then, the activity detection device can generate a minimum image 650 based on the minimum difference among the differences in pixel values ​​of corresponding regions between the low-resolution difference images 640. The minimum image 650 may have an index that is the minimum difference in pixel values ​​or a shift displacement of the low-resolution difference image including the minimum difference.

[0099] Figure 9 An example of generating a minimum graph image is shown.

[0100] Reference Figure 9 In operation 910, the activity detection device can shift the phase image. The activity detection device can shift the XN phase image (e.g., the second preprocessed phase image) while the first phase image (e.g., the first preprocessed phase image) is fixed. Figure 9 In the example, the number in each pixel of each phase image indicates the pixel value. In an XN phase image, X indicates the phase characteristic in the horizontal direction, and N indicates the number of phases. For example, in the case of using a first phase image and a second phase image generated by a 2PD sensor, the second phase image can be indicated as an X2 phase image. This example will be described below assuming that the XN phase image corresponds to the second phase image. In this example, the activity detection device may set a basic region in the first phase image and one or more shifted regions in the second phase image. For example, the activity detection device may set shifted regions (e.g., x-1, x0, and x+1) in the second phase image. In this example, x0 indicates the basic region that has not been shifted.

[0101] The basic region in the first phase image can also be referred to as the first basic region, and the basic region in the second phase image can also be referred to as the second basic region. The first basic region and the second basic region can correspond to each other in terms of position. In x-1 and x+1, - and + indicate the respective shift directions, and 1 indicates the reference shift value. The shift region can be set based on the reference shift value. For example, when the reference shift value is r, the shift region obtained by shifting the basic region along a preset direction by r can be set.

[0102] The activity detection device can set one or more shifted regions (e.g., shifted regions x-1 and x+1) by shifting a second basic region (e.g., shifted region x0) along a shift direction by a reference shift value (e.g., 1). The reference shift value can be set to various values, and the number of shifted regions corresponding to the reference shift value can be set. The number of shifted regions can be determined based on the reference shift value and the number of shift directions.

[0103] For example, with a reference shift value of 1 and two shift directions (e.g., left and right), there can be three shift regions (i.e., 2 × 1 + 1 = 3). These three shift regions can include x-1, x0, and x+1. For another example, with a reference shift value of 5 and two shift directions (e.g., left and right), there can be 11 shift regions (i.e., 2 × 5 + 1 = 11). These 11 shift regions can include x-5 to x-1, x0, and x+1 to x+5. For yet another example, with a reference shift value of 1 and four shift directions (e.g., left, right, up, and down), there can be five shift regions (i.e., 2 × 1 + 2 × 1 + 1 = 5). These five shift regions can include x-1, y-1, xy0, x+1, and y+1.

[0104] When using multiphase detection sensors (such as QPD sensors), phase characteristics in directions other than the horizontal direction can be identified. For example, as Figure 10A As shown, the activity detection device can shift the phase image along the horizontal and vertical directions of the QPD sensor, and determine the shifted area of ​​the phase image respectively. In the case of XN phase images, this can be achieved through references as described above. Figure 9 The image shift 910 describes a horizontal shift to determine the shift region (e.g., x-1, x0, and x+1). In the case of a YN phase image, the shift region (e.g., y-1, y0, and y+1) can be determined by a vertical shift.

[0105] In the aforementioned case, in XN and YN, X indicates the phase characteristic in the horizontal direction, Y indicates the phase characteristic in the vertical direction, and N indicates the number of phases. Although the same number of phases is used in both the horizontal and vertical directions, different numbers of phases can also be used in each of the horizontal and vertical directions. N can be determined based on the number of phases that can be identified by the sensor. In the case of a QPD sensor, N can be 2 (N = 2). Figure 10A In the example, there may be a first phase image, an X2 phase image, and a Y2 phase image.

[0106] For another example, such as Figure 10BAs shown, the activity detection device can shift the phase image along the horizontal, vertical, and diagonal directions of the QPD sensor, and determine the shifted regions of the phase image respectively. In the case of an XN phase image, the shifted regions (e.g., x-1, x0, and x+1) can be determined by shifting along the horizontal direction. In the case of a YN phase image, the shifted regions (y-1, y0, and y+1) can be determined by shifting along the vertical direction. In the case of a ZN phase image, the shifted regions (e.g., z-1, z0, and z+1) can be determined by shifting along the diagonal direction. In ZN, Z indicates the phase characteristic in the diagonal direction, and N indicates the number of phases. For example, when N is 2 (N=2), the shifted regions can be determined by shifting along the diagonal direction. Figure 10B The examples use a first phase image, an X2 phase image, a Y2 phase image, and a Z2 phase image.

[0107] Return to reference Figure 9 In operation 920, when the shifted regions are determined as described above, the activity detection device can calculate the difference between the image of the base region and the image of each shifted region. For example, the activity detection device can generate a difference image based on the difference between the fixed image (e.g., the image of the first base region) and each shifted image (e.g., the image of the shifted region), and then generate a minimum image based on the difference images. For example, the activity detection device can generate a first difference image based on the difference between the image of the first base region and the image of shifted region x-1, a second difference image based on the difference between the image of the first base region and the image of shifted region x0, and a third difference image based on the difference between the image of the first base region and the image of shifted region x+1. The activity detection device can assign index values ​​to each difference image. For example, the activity detection device can assign index values ​​in the order of x-1, x0, and x+1. Figure 9 In the example shown, index value 0 is assigned to the first difference image, index value 1 is assigned to the second difference image, and index value 2 is assigned to the third difference image. However, the example is not limited to the one shown, and index values ​​can be assigned in various orders in different examples.

[0108] The activity detection device can then generate a first low-resolution difference image, a second low-resolution difference image, and a third low-resolution difference image from the first difference image, the second difference image, and the third difference image, respectively, using a pooling method that selects only a portion of the pixels. In one example, the images at indices 0, 1, and 2 shown in operation 930 may correspond to the first low-resolution difference image, the second low-resolution difference image, and the third low-resolution difference image, respectively.

[0109] In operation 930, the activity detection device generates a minimum image. For example, the activity detection device can identify the minimum difference among corresponding differences in corresponding regions in a low-resolution differential image, and determine the pixel values ​​of the minimum image based on the identified minimum difference. Figure 9 In the example, the corresponding differences at position (1,1) are 1, 0, and 6. 0 is the minimum difference, so it can be selected. Furthermore, the corresponding differences at position (2,2) are 25, 33, and 30. Among these corresponding differences, 25 can be selected as the minimum difference. As described above, the minimum difference can be selected from the corresponding differences between low-resolution difference images, and the pixel value of the minimum image can be determined based on the selected minimum difference.

[0110] The pixel values ​​of the minimum difference image can correspond to the minimum difference or to the index of the low-resolution difference image that includes the minimum difference. The minimum difference image that includes the minimum difference can also be called the minimum difference image (e.g., 934), and the minimum difference image that includes the index of the low-resolution difference image containing the minimum difference can also be called the minimum index image (e.g., 932). In the foregoing example, 0 is selected as the minimum difference at position (1,1), and the index of the low-resolution difference image that includes 0 is 1. Therefore, the pixel value at (1,1) in the minimum difference image 934 is 0, and the pixel value at (1,1) in the minimum index image 932 is 1. Furthermore, 25 is selected as the minimum difference at position (2,2), and the index of the low-resolution difference image that includes 25 is 0. Therefore, the pixel value at (2,2) in the minimum difference image 934 is 25, and the pixel value at (2,2) in the minimum index image 932 is 0.

[0111] A set of difference images can be generated for each phase image. Figure 10A and Figure 10B In the example, when phase images exist in different directions, a minimum image for each phase image can be generated based on a set of difference images for each phase image. Figure 10A In the example, a minimum image can be generated for each of the XN-phase and YN-phase images. Figure 10B In the example, a minimum image can be generated for each of the XN phase image, YN phase image, and ZN phase image.

[0112] Figure 11A and Figure 11B This shows an example of determining the pixel values ​​of a minimum image.

[0113] Reference Figure 11ARegion 1132 in the first low-resolution difference image 832, region 1134 in the second low-resolution difference image 834, region 1136 in the third low-resolution difference image 836, region 1138 in the fourth low-resolution difference image 838, and region 1140 in the fifth low-resolution difference image 840 are considered as corresponding regions. The activity detection device can identify the smallest difference among the differences indicated by regions 1132, 1134, 1136, 1138, and 1140. For example, if the difference in region 1134 is less than the individual differences in the other regions 1132, 1136, 1138, and 1140, the activity detection device can determine the difference in region 1134 as the pixel value of region 1150 in the smallest image 650 corresponding to region 1134. Optionally, the activity detection device may determine the shift index (e.g., -1) corresponding to the second low-resolution differential image 834 including region 1134 as the pixel value of region 1150 in the minimum image 650.

[0114] Reference Figure 11B Region 1162 in the first low-resolution differential image 832, region 1164 in the second low-resolution differential image 834, region 1166 in the third low-resolution differential image 836, region 1168 in the fourth low-resolution differential image 838, and region 1170 in the fifth low-resolution differential image 840 are the corresponding regions. The activity detection device can refer to the above. Figure 11A A similar method is described to identify the smallest difference among the differences indicated by regions 1162, 1164, 1166, 1168, and 1170. For example, if the difference in region 1168 is less than the respective differences in the other regions 1162, 1164, 1166, and 1170, the activity detection device may determine the difference in region 1168 as the pixel value of region 1180 in the minimum image 650 corresponding to region 1168. Alternatively, the activity detection device may determine the shift index (e.g., +1) corresponding to the fourth low-resolution difference image 838 including region 1168 as the pixel value of region 1180 in the minimum image 650.

[0115] The activity detection device can determine the minimum image 650 by performing the aforementioned operation on all corresponding regions in the low-resolution difference image 640. By determining the pixel values ​​of the minimum image 650 using the minimum difference, a minimum difference image can be generated (e.g., Figure 9 The minimum difference image 934). By determining the pixel values ​​of the minimum image 650 using a shift index, a minimum indexed image can be generated (e.g., Figure 9The minimum index image 932). When the minimum image 650 is determined, the minimum image 650 and a phase image with the same resolution as the minimum image 650 can be input into the activity detection model.

[0116] Figure 12 An example of a device with activity detection is shown.

[0117] Reference Figure 12 The device 1200 with activity detection may include a processor 1210 (e.g., one or more processors) and a memory 1220 (e.g., one or more memories). The memory 1220 may be connected to the processor 1210 to store instructions executable by the processor 1210, data to be processed by the processor 1210, and data processed by the processor 1210. The memory 1220 may include high-speed random access memory (RAM) and / or non-volatile computer-readable storage media (e.g., flash memory).

[0118] Processor 1210 is executable for performing the above-mentioned references. Figures 1 to 11B The instructions described are one or more of the following: In one example, processor 1210 may receive multiple phase images of different phases obtained by multiphase detection sensor 1240, and generate multiple preprocessed phase images by performing preprocessing on the phase images, including edge enhancement processing. For example, processor 1210 may remove noise from the phase images and enhance edge regions of objects in each phase image. Processor 1210 may generate multiple difference images based on the preprocessed phase images, and generate multiple low-resolution difference images, each having a lower resolution than the individual difference images. For example, processor 1210 may extract the maximum pixel value for each block region from the difference images and generate a low-resolution difference image based on the extracted maximum pixel value. In another example, processor 1210 may determine the average pixel value in the pixel regions of each block region in the difference images and generate a low-resolution difference image based on the determined average value corresponding to each block region.

[0119] Processor 1210 can generate a minimum image based on low-resolution difference images. For example, processor 1210 can identify the minimum difference among the differences between corresponding regions in the low-resolution difference images and determine the pixel values ​​of the minimum image based on the identified minimum difference. The pixel values ​​of the minimum image may correspond to the selected minimum difference or to the index of a shift displacement corresponding to a low-resolution difference image in the difference images that includes the minimum difference. Processor 1210 can perform activity detection on objects in the phase image based on the minimum image. Processor 1210 can apply the minimum image and the phase image to a neural network-based activity detection model (e.g., Figure 6 The activity detection model 670 is used to obtain the activity detection result. The resolution of the phase image to be applied to the activity detection model can be adjusted to correspond to the resolution of the minimum image. The activity detection model can output an activity score as the activity detection result. Then, the processor 1210 can compare the activity score with a threshold and determine whether the object is a real object or a fake object based on the comparison result.

[0120] Figure 13 An illustration showing an example of an electronic device.

[0121] Device 1300 with activity detection (e.g., Figure 1 The electronic device 120 can generate an input image obtained by capturing an image of an object and perform activity detection on the object in the input image. The device 1300 with activity detection can perform biometric verification (e.g., facial verification and iris verification) based on the result of the activity detection performed on the object. The device 1300 with activity detection can perform... Figure 12 The operation or function of the device 1200 with activity detection.

[0122] Reference Figure 13 The device 1300 with activity detection may include a processor 1310 (e.g., one or more processors), a memory 1320 (e.g., one or more memories), an image sensor 1330, a storage device 1350, an input device 1360, an output device 1370, and a communication device 1380. The processor 1310, memory 1320, image sensor 1330, storage device 1350, input device 1360, output device 1370, and communication device 1380 can communicate with each other via a communication bus 1390. In a non-limiting example, the device 1300 with activity detection may include... Figure 12 The device 1200 with activity detection includes a processor 1310, a memory 1320, and an image sensor 1330.

[0123] Processor 1310 can execute functions and instructions to be executed in device 1300 with activity detection. Processor 1310 can process instructions stored in memory 1320 or storage device 1350. Processor 1310 can execute the above-mentioned instructions. Figures 1 to 12One or more of the described operations or methods. In one example, processor 1310 may generate a preprocessed phase image by performing preprocessing, including edge enhancement processing, on phase images of different phases obtained by image sensor 1330, and generate a difference image based on the preprocessed phase image. Processor 1310 may generate a low-resolution difference image with a lower resolution than the difference image, and generate a minimum image based on the low-resolution difference image. Processor 1310 may then perform activity detection on objects in the phase image based on the minimum image. Processor 1310 may apply the minimum image and the phase image to a neural network-based activity detection model (e.g., Figure 6 The activity detection model 670 is used to obtain the activity score, and the result of the activity detection is determined based on the comparison of the activity score with a threshold.

[0124] The memory 1320 may store data for performing activity detection. The memory 1320 may store instructions to be executed by the processor 1310, as well as information for performing activity detection and / or biometric verification.

[0125] Image sensor 1330 can generate an image by capturing an image of an object. Image sensor 1330 may include a multiphase detection sensor (e.g., a 2PD sensor, a QPD sensor, etc.) configured to acquire phase images of different phases.

[0126] Storage device 1350 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1350 may store a larger amount of information than memory 1320 and store the information for a relatively long period of time. As a non-limiting example, storage device 1350 may include magnetic hard disks, optical disks, flash memory, and floppy disks.

[0127] As a non-limiting example, input device 1360 may receive input from a user: haptic input, video input, audio input, and touch input. As a non-limiting example, input device 1360 may include a keyboard, mouse, touchscreen, microphone, and other means configured to detect input from the user and transmit the detected input to device 1300 with activity detection.

[0128] Output device 1370 can provide the user with the output of device 1300 having activity detection via a visual channel, audio channel, or tactile channel. As a non-limiting example, output device 1370 may include a display, touchscreen, speaker, vibration generator, and other devices configured to provide output to the user. Communication device 1380 can communicate with external devices via a wired or wireless network.

[0129] In this regard Figures 1 to 13The described electronic devices, image sensors, 2PD sensors, sensor pixels, sensor subpixels, devices with activity detection, processors, memory, multiphase detection sensors, storage devices, input devices, output devices, communication devices, communication buses, electronic device 120, image sensor 130, 2PD sensor 310, sensor pixel 315, sensor subpixels 332, 334, 336, 338, 342, 344, 346, 348, device with activity detection 1200, processor 1210, memory 1220, multiphase detection sensor 1240, device with activity detection 1300, processor 1310, memory 1320, image sensor 1330, storage device 1350, input device 1360, output device 1370, communication device 1380, communication bus 1390, and other devices, apparatuses, units, modules, and components are implemented or represented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers.One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components. The hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, single instruction single data (SISD) multiple processing, single instruction multiple data (SIMD) multiple processing, multiple instruction single data (MISD) multiple processing, and multiple instruction multiple data (MIMD) multiple processing.

[0130] Figures 1 to 13 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0131] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0132] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or may be recorded, stored, or fixed on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling one or more processors or computers to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions or software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0133] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0134] Therefore, the scope of the disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A method for activity detection, comprising: Receive multiple phase images of different phases; Multiple preprocessed phase images are generated by performing preprocessing, including edge enhancement, on the multiple phase images of different phases; A difference image is generated based on the multiple preprocessed phase images; Generate low-resolution difference images with lower resolution than the difference images based on the difference images; Generate a minimum image based on a low-resolution difference image; Activity detection is performed on objects in the plurality of phase images based on the minimum image. The step of generating the minimum image includes: identifying the minimum difference among the differences between corresponding regions of the low-resolution difference images; and determining the pixel value of the minimum image based on the identified minimum difference.

2. The method according to claim 1, wherein, Edge enhancement processing includes: enhancing the edge regions of objects in each of the plurality of phase images. The preprocessing steps include: Noise is removed from the plurality of phase images, and Edge enhancement processing is performed based on the noise-removed multiple phase images.

3. The method according to claim 1, wherein, The steps of performing edge enhancement processing include applying any one or more of the Sobel filter, anisotropic filter, Laplacian filter, Canny edge filter, and neural network processing to each of the plurality of phase images.

4. The method according to claim 1, wherein, The steps to generate a low-resolution difference image include: A low-resolution difference image corresponding to the difference image is generated based on the average pixel value in the pixel region of each of the multiple block regions in the difference image, or the maximum pixel value of each of the multiple block regions in the difference image.

5. The method according to claim 1, wherein, The plurality of preprocessed phase images include a first preprocessed phase image and a second preprocessed phase image, and The steps to generate a difference image include: Multiple shifted second phase images are generated by shifting the preprocessed phase image by different shift displacements; and Multiple difference images are generated, including: multiple difference images that each indicate the difference between a corresponding pixel value in the multiple shifted second phase images and the first preprocessed phase image, and a difference image that indicates the difference between the pixel values ​​in the second preprocessed phase image and the first preprocessed phase image.

6. The method according to claim 1, wherein, The pixel value of the minimum image corresponds to any of the following: The minimum difference; and The index of the shift displacement corresponding to the low-resolution difference image including the minimum difference.

7. The method according to claim 1, wherein, The steps for performing activity detection include: obtaining the activity detection result by applying the minimum image and the plurality of phase images to a neural network-based activity detection model.

8. The method according to claim 7, wherein, The resolution of the multiple phase images applied to the activity detection model is adjusted to correspond to the resolution of the minimum image.

9. The method according to any one of claims 1 to 8, wherein, The multiple phase images are obtained using a multiphase detection sensor.

10. The method of claim 9, further comprising: Perform the step of obtaining the multiple phase images using a multiphase detection sensor.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method according to any one of claims 1 to 10.

12. An apparatus for activity detection, comprising: The processor is configured as follows: Multiple preprocessed phase images are generated by performing preprocessing, including edge enhancement, on multiple phase images of different phases; A difference image is generated based on the multiple preprocessed phase images; Generate low-resolution difference images with lower resolution than the difference images based on the difference images; Generate a minimum image based on a low-resolution difference image; and Activity detection is performed on objects in the plurality of phase images based on the minimum image. In order to generate the minimum image, the processor is configured to: identify the minimum difference among the differences between corresponding regions of the low-resolution differential images; and determine the pixel values ​​of the minimum image based on the identified minimum difference.

13. The device according to claim 12, wherein, The processor is configured to enhance the edge regions of objects in each of the plurality of phase images for edge enhancement processing. The processor is configured as follows: To perform preprocessing, noise is removed from the multiple phase images, and Edge enhancement processing is performed based on the noise-removed multiple phase images.

14. The device according to claim 12, wherein, To generate low-resolution difference images, the processor is configured as follows: A low-resolution difference image corresponding to the difference image is generated based on the average pixel value in the pixel region of each of the multiple block regions in the difference image, or the maximum pixel value of each of the multiple block regions in the difference image.

15. The device according to claim 12, wherein, In order to perform liveness detection, the processor is configured as follows: The activity detection result is obtained by applying the minimum image and the multiple phase images to a neural network-based activity detection model, and The resolution of the multiple phase images applied to the activity detection model is adjusted to correspond to the resolution of the minimum image.

16. The device according to any one of claims 12 to 15, wherein, The device is an electronic device that includes a multiphase detection sensor configured to acquire the multiple phase images using multiple sensor pixels.

17. An electronic device comprising: A multiphase detection sensor is configured to acquire multiple phase images of different phases; and The processor is configured as follows: Multiple preprocessed phase images are generated by performing preprocessing, including edge enhancement, on the multiple phase images of different phases; A difference image is generated based on the multiple preprocessed phase images; Generate low-resolution difference images with lower resolution than the difference images based on the difference images; Generate a minimum image based on a low-resolution difference image; and Activity detection is performed on objects in the plurality of phase images based on the minimum image. In order to generate the minimum image, the processor is configured to: identify the minimum difference among the differences between corresponding regions of the low-resolution differential images; and determine the pixel values ​​of the minimum image based on the identified minimum difference.

18. The electronic device according to claim 17, wherein, The processor is configured to enhance the edge regions of objects in each of the plurality of phase images for edge enhancement processing. The processor is configured as follows: To perform preprocessing, noise is removed from the multiple phase images. Edge enhancement processing is performed based on the noise-removed multiple phase images.

19. The electronic device according to claim 17, wherein, To generate low-resolution difference images, the processor is configured as follows: A low-resolution difference image corresponding to the difference image is generated based on the average pixel value in the pixel region of each of the multiple block regions in the difference image, or the maximum pixel value of each of the multiple block regions in the difference image.

Images