Method and apparatus for activity detection

Abstract

A method and apparatus for activity detection are disclosed. The method includes generating a first phase image based on first visual information of a first phase, generating a second phase image based on second visual information of a second phase, generating a min-map based on a disparity between the first phase image and the second phase image, and detecting activity based on the min-map.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0022858, filed February 25, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to a method and apparatus for detecting activity based on phase difference. BACKGROUND

[0003] Biometric authentication technology is used to authenticate a user based on, for example, a fingerprint, an iris, a voice, a face, or a blood vessel. Such a biometric characteristic used for authentication differs from individual to individual, rarely changes throughout life, and has a low risk of being stolen or imitated. Furthermore, an individual does not need to intentionally carry such a characteristic at all times. A face verification technology, which is one type of biometric authentication technology, is an authentication technology that determines whether a user is a valid user based on a face appearing in a still image or a moving image. The face verification technology can identify a target person to be authenticated without physical contact with the target person. Due to the convenience and efficiency of the face verification technology, the face verification technology is being widely applied to various application fields (e.g., a security system, mobile verification, or multimedia data search). SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is neither intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in determining the scope of the claimed subject matter.

[0005] In one general aspect, an activity detection method includes generating a first phase image based on first visual information of a first phase sensed by a first pixel group of an image sensor; generating a second phase image based on second visual information of a second phase sensed by a second pixel group of the image sensor; generating a minimum map based on a disparity between the first phase image and the second phase image; and detecting an activity based on the minimum map.

[0006] The generating of the minimum map can include setting a first base region in the first phase image; setting a second base region corresponding to the first base region in the second phase image; setting at least one shifted region by shifting the second base region by a reference shift value; generating a difference image based on a difference between an image of the first base region and an image of the second base region and a difference between the image of the first base region and at least one image of the at least one shifted region; and generating the minimum map based on the difference image.

[0007] The step of generating the minimum image based on the difference image can include selecting a minimum value among corresponding difference values at positions corresponding to each other in the difference image, and determining a pixel value of the minimum image based on the minimum value. The pixel value of the minimum image can correspond to the minimum value or to an index of a difference image including the minimum value among the difference images.

[0008] The step of detecting the activity can include inputting input data including at least one patch based on the minimum image to at least one activity detection model, and detecting the activity based on an output of the at least one activity detection model. The at least one activity detection model can include at least one neural network, and the at least one neural network can be pre-trained to detect the activity of the object.

[0009] The activity detection method can further include generating a reference image by concatenating the first phase image, the second phase image, and the minimum image. The step of detecting the activity can further include generating the at least one patch by cropping the reference image based on a region of interest (ROI). The at least one patch can include a plurality of patches having different characteristics of the object. The at least one activity detection model can include a plurality of activity detection models processing input data including the plurality of patches. The step of detecting the activity based on the output of the at least one activity detection model can include detecting the activity by fusing outputs of the plurality of activity detection models in response to input of the input data.

[0010] The activity detection method can further include generating a reference image by concatenating the first phase image, the second phase image, and the minimum image. The step of detecting the activity can include detecting the activity based on the reference image. The activity detection method can further include performing pre-processing of the first phase image and the second phase image. The step of performing the pre-processing can include applying any one or any combination of down-sizing, lens shading correction, gamma correction, histogram matching, and de-noising to the first phase image and the second phase image.

[0011] The first pixels of the first pixel group and the second pixels of the second pixel group can be disposed adjacent to each other. The activity detection method can further include generating a third phase image based on third visual information of a third phase sensed by a third pixel group of the image sensor, and generating a fourth phase image based on fourth visual information of a fourth phase sensed by a fourth pixel group of the image sensor. When the minimum image is generated, a disparity between the first phase image and the third phase image and a disparity between the first phase image and the fourth phase image can also be used.

[0012] In another general aspect, an activity detection device includes a processor, and a memory including instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to generate a first phase image based on first visual information of a first phase sensed by a first pixel group of an image sensor, generate a second phase image based on second visual information of a second phase sensed by a second pixel group of the image sensor, generate a minima map based on a disparity between the first phase image and the second phase image, and detect an activity based on the minima map.

[0013] In another general aspect, an electronic device includes an image sensor configured to sense first visual information of a first phase using a first pixel group and to sense second visual information of a second phase using a second pixel group, and a processor configured to generate a first phase image based on the first visual information, to generate a second phase image based on the second visual information, to generate a minima map based on a disparity between the first phase image and the second phase image, and to detect an activity based on the minima map.

[0014] In another general aspect, an activity detection device includes a multi-phase detection sensor configured to generate a first phase image by sensing first visual information of a first phase using a first pixel group and to generate a second phase image by sensing second visual information of a second phase using a second pixel group, a multi-phase block generator configured to generate a minima map based on a disparity between the first phase image and the second phase image, and an activity detector configured to detect an activity based on the minima map.

[0015] In another general aspect, a device includes one or more processors, and at least one memory storing instructions executable by the one or more processors. In response to the instructions being executed by the one or more processors, the one or more processors are configured to input an image including an object, generate disparity data based on a disparity between a first phase image corresponding to the object and a second phase image corresponding to the object, generate a reference image based on the first phase image, the second phase image, and the disparity data, generate input data based on the reference image, input the input data to a detection model including a neural network, and authenticate the object based on output data of the detection model.

[0016] The one or more processors can be configured to authenticate the object by determining an activity of the object based on the output data.

[0017] The one or more processors can be configured to generate the reference image by concatenating the first phase image, the second phase image, and the disparity data.

[0018] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 An example showing operation of an activity detection device.

[0020] Figure 2 An example showing a quadrature phase detection (QPD) image sensor.

[0021] Figure 3 An example showing differences between a two-dimensional (2D) object and a three-dimensional (3D) object that can be detected based on phase images.

[0022] Figure 4 An example showing a method of detecting activity based on phase difference detection.

[0023] Figure 5 An example showing phase characteristics in each direction in an input image.

[0024] Figure 6 An example showing generation of a minimum map.

[0025] Figure 7A And Figure 7B An example showing shifting of a phase image.

[0026] Figure 8 An example showing detecting activity using reference information and an activity detection model.

[0027] Figure 9 An example showing generation of a reference image.

[0028] Figure 10 An example showing generation of output data using an activity detection model.

[0029] Figure 11 An example showing generation of output data using multiple activity detection models.

[0030] Figure 12A And Figure 12B is a block diagram showing an example of an activity detection device.

[0031] Figure 13 is a block diagram showing an example of an electronic device.

[0032] Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing references will be understood to represent the same elements, features, and structures. The drawings can not be to scale, and the relative dimensions, proportions, and depiction of the elements in the drawings can be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0033] The following detailed description is presented to aid the reader in gaining a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, various changes, modifications, and equivalents can be used, and thus particular embodiments described herein are not intended as being exhaustive of what the present disclosure can provide. For example, although sequences of operations can be described, the sequence is an example and is not limited by the orders not specified in that sequence. For example, operations described sequentially can be rearranged or omitted or operations described contemporaneously can be ordered sequentially, except operations that by their nature must precede other operations. Moreover, descriptions of features in terms of "coupled" or "connected" refer to electrical or physical connections made by wires, buses, or other means that are not necessarily direct electrical connections through intervening electrical components.

[0034] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, these examples are provided as illustrative of a number of ways in which the methods, apparatuses, and / or systems described herein can be implemented, which will be apparent to one skilled in the art after understanding the present disclosure.

[0035] The following detailed description of structure or function is exemplary to describe only examples and the scope of examples is not limited to the description provided in this specification. Various changes and modifications can be made thereto by those of ordinary skill in the art.

[0036] Although the terms "first" or "second" are used to explain various components, the components are not limited by the terms. The terms should be used only to distinguish one component from another component. For example, a "first" component can be referred to as a "second" component, or similarly, a "second" component can be referred to as a "first" component, within the scope of the rights of the disclosed concept.

[0037] As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0038] Unless defined otherwise herein, all terms used herein, including technical terms and scientific terms, have the same meanings as those generally understood by those with ordinary knowledge in the art. Unless defined otherwise herein, terms defined in a general dictionary should be interpreted as having meanings matching the contextual meanings in the relevant field, and should not be interpreted as having idealized or overly formal meanings.

[0039] Hereinafter, examples will be described in detail with reference to the accompanying drawings, in which the same reference numerals denote the same elements throughout.

[0040] Figure 1 An example of the operation of the activity (or called, living body) detection apparatus 100 is shown. Referring toFigure 1 The activity detection device 100 generates a detection result 120 based on the visual information of the object 110. The detection result 120 can include information about the activity. For example, the detection result 120 can indicate whether the object 110 corresponds to a real user or an attacker (such as an image acquired by taking a picture of a user). The detection result 120 can be used in image-based biometric authentication (e.g., face verification or iris authentication).

[0041] The visual information of the object 110 can be represented by a plurality of phases. The image sensor 130 can sense the visual information of the plurality of phases and can generate sensor data associated with the visual information of each phase. The image sensor 130 can correspond to a multi-phase detection sensor. For example, the image sensor 130 can be a two-phase detection (2PD) sensor for sensing two types of phases, or a quadrature phase detection (QPD) sensor for sensing four types of phases. However, the number of phases sensed by the image sensor 130 is not limited thereto, and the image sensor 130 can sense various numbers of phases. The image sensor 130 corresponds to a 2PD sensor as shown in FIG. 1B, and an example in which the image sensor 130 corresponds to a 2PD sensor will be described below. However, this is for convenience of description only, and the description below also applies to an example in which the image sensor 130 corresponds to another multi-phase detection sensor (such as a QPD sensor). Figure 1

[0042] The plurality of pixels included in the image sensor 130 can belong to one of a first group 1 and a second group 2. First pixels of the first group 1 can sense first visual information of a first phase and generate first sensor data, and second pixels of the second group 2 can sense second visual information of a second phase and generate second sensor data. The first pixels and the second pixels can be disposed adjacent to each other. The first pixels and the second pixels disposed adjacent to each other can indicate any one or any combination of an example in which there are no pixels between the first pixels and the second pixels in a direction that distinguishes a phase characteristic, an example in which the first pixels are discontinuously arranged, and an example in which the second pixels are discontinuously arranged. The distinction of the phase characteristic will be further described below with reference to Figure 5

[0043] Figure 2 An example of a QPD image sensor is shown. Referring to FIG. 1C, Figure 2 The image sensor 210 can sense four types of phases in the form of a grid by distinguishing the phases. For example, in the image sensor 210, first pixels of a first group 1 can sense first visual information of a first phase, second pixels of a second group 2 can sense second visual information of a second phase, third pixels of a third group 3 can sense third visual information of a third phase, and fourth pixels of a fourth group 4 can sense fourth visual information of a fourth phase. ​​

[0044] Referring back to Figure 1 , the activity detection device 100 can generate a first phase image 141 based on the first sensor data, and can generate a second phase image 142 based on the second sensor data. Due to a characteristic of the image sensor 130, a disparity can exist between the first phase image 141 and the second phase image 142, and the disparity can be used to detect the activity of the object 110. For example, Figure 3 An example showing a difference between a two-dimensional (2D) object and a three-dimensional (3D) object that can be detected based on phase images is shown. When a 2D object is photographed, a disparity can not be detected based on the first phase image and the second phase image. When a 3D object is photographed, a disparity can be detected based on the first phase image and the second phase image. For example, a disparity can be detected from a stereoscopic structure (e.g., a user's nose).

[0045] The activity detection device 100 can generate a minimum map 150 and a reference image 160 based on the first phase image 141 and the second phase image 142, and can detect the activity of the object 110 based on the minimum map 150 and the reference image 160. When the object 110 corresponds to a real user, a disparity corresponding to a difference between the first phase image 141 and the second phase image 142 can exist. Due to a structural characteristic of the image sensor 130 having a narrow gap between the first pixel of the first group 1 and the second pixel of the second group 2, the disparity can not be relatively large. The activity detection device 100 can analyze the above fine disparity based on the minimum map 150 and the reference image 160, and can effectively detect the activity of the object 110 based on the analysis result.

[0046] The activity detection device 100 can shift (or move) one of the first phase image 141 and the second phase image 142 at least once in a state of fixing the other, and can generate the minimum map 150 based on a difference between the fixed image and the shifted image. For example, the activity detection device 100 can set a first basic region in the first phase image 141, can set a second basic region corresponding to the first basic region in the second phase image 142, and can generate at least one shifted region by shifting the second basic region by a reference shift value. The activity detection device 100 can generate a difference image based on a difference between an image of the first basic region and an image of the second basic region and a difference between the image of the first basic region and an image of the at least one shifted region.

[0047] The activity detection device 100 can generate the minimum map 150 based on the difference image. For example, the activity detection device 100 can select a minimum value among corresponding difference values located at corresponding coordinates corresponding to each other in the difference image, and can determine a pixel value of the minimum map 150 based on the minimum value. Based on the above scheme, each pixel value of the minimum map 150 can be determined. The pixel value of the minimum map 150 can correspond to the minimum value or an index of the difference image including the minimum value among the difference images. The minimum map 150 can include the minimum value or the index.

[0048] The activity detection device 100 can generate the reference image 160 by combining the first phase image 141, the second phase image 142, and the minimum map 150, and can detect the activity of the object 110 based on the reference image 160. For example, the activity detection device 100 can generate the reference image 160 by concatenating the first phase image 141, the second phase image 142, and the minimum map 150, and can detect the activity of the object 110 based on the reference image 160.

[0049] The activity detection device 100 can detect the activity of the object 110 using at least one activity detection model. Each activity detection model can include at least one neural network. The activity detection device 100 can generate input data of the activity detection model based on the reference image 160, can input the input data to the activity detection model, and can detect the activity of the object 110 based on output data of the activity detection model. At least a part of the neural network can be implemented as software, hardware including a neural processor, or a combination of software and hardware.

[0050] For example, the neural network can correspond to a deep neural network (DNN) (e.g., a fully connected network, a deep convolutional neural network (CNN), or a recurrent neural network (RNN)). The DNN can include a plurality of layers. The plurality of layers can include an input layer, at least one hidden layer, and an output layer.

[0051] The neural network can be trained to perform a given operation by mapping input data and output data in a non-linear relationship based on deep learning. Deep learning can be a machine learning scheme for solving a given problem according to a large data set. Deep learning can be understood as a process of solving an optimization problem by searching for a point that minimizes energy while training a neural network based on prepared training data. Through supervised learning or unsupervised learning by deep learning, a structure of the neural network or a weight corresponding to the model can be obtained, and the input data and the output data can be mapped to each other through the weight.

[0052] A neural network can be trained based on training data in a training process, and can perform an inference operation (e.g., classification, recognition, or detection) of input data in an inference process. The neural network of the activity detection model can be pre-trained to detect an activity of an object in input data. The term "pre" indicates a state before the neural network is "launched". A "launched" neural network indicates that the neural network can be ready for inference. For example, the "launching" of the neural network can include loading the neural network into a memory, or inputting input data for inference to the neural network after the neural network is loaded into the memory.

[0053] Figure 4 An example of a method of detecting an activity based on a phase difference is illustrated. Referring to FIG. 4, an activity detection device detects an activity of an object in an input image in operation 410. Figure 4 In operation 410, the activity detection device generates a phase image based on visual information of a plurality of phases. For example, the activity detection device can receive sensor data from a pixel group that senses visual information of different phases, and can generate a phase image based on the sensor data. Hereinafter, an example in which the phase image representatively includes a first phase image and a second phase image will be representatively described.

[0054] In operation 420, the activity detection device performs pre-processing of the phase image. When detecting an activity from a 2D image, pre-processing (such as distortion correction) is generally performed. However, in pre-processing according to an example, pre-processing (such as distortion correction) can not be performed. This is because it can be desirable to maintain the shape of the object to detect a fine parallax, but pre-processing (such as distortion correction) can change the shape of the object. In one example, instead of distortion correction, pre-processing including any one or any combination of downsize, lens shading correction, gamma correction, histogram matching, and denoising can be performed. In another example, pre-processing can not be performed.

[0055] In one example, the activity detection device can apply downsize to the phase image, and can perform pre-processing (such as lens shading correction) of the phase image to which the downsize is applied. By downsize, the amount of calculation can be reduced. For example, downsize can be performed in a direction in which a phase characteristic is not distinguished. Since information associated with a parallax is mainly included in a direction in which a phase characteristic is distinguished, a loss of information can be minimized in the downsize process. Further, noise can be removed through pre-processing (such as lens shading correction or gamma correction), and thus, the accuracy of image information can be enhanced. Hereinafter, an example of downsize operation will be described with reference to FIG. 5. Figure 5 An example of downsize operation is further described.

[0056] Figure 5 An example of a phase characteristic in each direction in an input image is illustrated. Referring to FIG. 3, an input image 300 is illustrated. The input image 300 can include a first phase image 310 and a second phase image 320. The first phase image 310 and the second phase image 320 can be generated based on visual information of a plurality of phases. The first phase image 310 and the second phase image 320 can be generated based on sensor data of a pixel group that senses visual information of different phases. Figure 5The pixels of the first group 1 and the pixels of the second group 2 are alternately arranged in the horizontal direction of the image sensor 510. Accordingly, the phase characteristics can be considered to be reflected in the horizontal direction. In other words, the phase characteristics can be distinguished based on the pixel values in the horizontal direction. Since the image sensor 510 corresponds to a 2PD sensor, the phase characteristics are not distinguished in the vertical direction. Accordingly, in order to maintain the phase characteristics, the activity detection device can perform the down-sizing in the direction in which the phase characteristics are not distinguished. For example, the activity detection device can down-size each of the first phase image 521 and the second phase image 522 in the vertical direction.

[0057] In this example, the activity detection device can remove the sensing data of a predetermined row based on a predetermined down-sizing ratio or can perform statistical processing (e.g., averaging) of the sensing data of a plurality of rows based on a predetermined down-sizing ratio to perform the down-sizing. In one example, the activity detection device can perform, for each column, averaging of the sensor data of every two adjacent rows in a certain order (e.g., from the first row to the last row) so that the size of the phase image is down-sized to 1 / 2 of the original phase image size. For example, the activity detection device can perform, for each column, averaging of the sensor data of the first row and the sensor data of the second row adjacent to the first row, and can down-size the phase image to 1 / 2.

[0058] Referring back to Figure 4 In operation 430, the activity detection device generates a minimum map based on the disparity between the phase images. As described above, the activity detection device can shift one of the first phase image and the second phase image at least once in a state in which the other is fixed, and can generate the minimum map based on the difference between the fixed image and the shifted image. Examples of generating the minimum map will be further described below with reference to Figure 6 , Figure 7A and Figure 7B .

[0059] In operation 440, the activity detection device detects the activity based on the minimum map. In one example, the activity detection device can generate a reference image by combining the phase images and the minimum map, can input input data corresponding to the reference image to the activity detection model, and can detect the activity based on the output data of the activity detection model. For example, the activity detection device can generate at least one patch by cropping the reference image based on a region of interest (ROI), and the input data of the activity detection model can be generated based on the at least one patch. Examples of detecting the activity will be further described below with reference to Figure 8

[0060] Figure 6 Examples of generating the minimum map are illustrated. Referring to Figure 6 ​In operation 610, the activity detection device performs phase image shifting. As described above, the activity detection device can shift one of the first phase image and the second phase image at least once in a state in which the other is fixed. Figure 6 An example in which the first phase image is fixed and the XNth phase image is shifted is shown. In Figure 6 In the XN, X indicates that the phase characteristics are distinguished in the horizontal direction, and N represents the number of phases. For example, when the first phase image and the second phase image generated by the 2PD sensor are used, the second phase image can be represented as the X2nd phase image. Hereinafter, an example in which the XNth phase image corresponds to the second phase image is described. The activity detection device can set a base region in the first phase image, and can set at least one shifted region in the second phase image. For example, the activity detection device can set shifted regions x-1, x0, and x+1 in the second phase image. In this example, x0 represents a base region in which no shifting is performed.

[0061] The base region of the first phase image can be referred to as a "first base region", the base region of the second phase image can be referred to as a "second base region", and the first base region and the second base region can correspond to each other in position. In "x-1" and "x+1", - and + represent a shifting direction, and "1" represents a reference shift value. The base region can be set based on the reference shift value. When the reference shift value is "r", a shifted region can be set by shifting the base region by "r" in a predetermined direction. Accordingly, the base region can be set in a range such that an available space for shifting is secured.

[0062] The activity detection device can set at least one shifted region (for example, shifted regions x-1 and x+1) by shifting the second base region (for example, shifted region x0) by a reference shift value (for example, "1") in a shifting direction. The reference shift value can be set to various values, and the number of shifted regions corresponding to the reference shift value can be set. For example, the number of shifted regions can be determined based on the reference shift value and the number of shifting directions.

[0063] The activity detection device can set at least one shifted region (for example, shifted regions x-1 and x+1) by shifting the second base region (for example, shifted region x0) by a reference shift value (for example, "1") in a shifting direction. The reference shift value can be set to various values, and the number of shifted regions corresponding to the reference shift value can be set. For example, the number of shifted regions can be determined based on the reference shift value and the number of shifting directions.

[0064] In one example, when the reference shift value is "1" and when the number of shift directions is "2" indicating left and right directions, the number of shift regions can be "2x1+1=3". In this example, the three shift regions can include shift regions x-1, x0, and x+1. In another example, when the reference shift value is "5" and when the number of shift directions is "2" indicating left and right directions, the number of shift regions can be "2x5+1=11". In this example, the 11 shift regions can include shift regions x-5 to x-1, x0, and x+1 to x+5. In yet another example, when the reference shift value is "1" and when the number of shift directions is "4" indicating left, right, up, and down directions, the number of shift regions can be "2x1+2x1+1=5". In this example, the five shift regions can include shift regions x-1, y-1, xy0, x+1, and y+1.

[0065] When a multi-phase detection sensor such as a QPD sensor is used, phase characteristics can be distinguished in directions other than the horizontal direction. In one example, as shown in Figure 7A , the activity detection device can move phase images in the horizontal direction and the vertical direction of the QPD sensor, and can determine a movement region of each phase image. In the XNth phase image, similar to operation 610 of Figure 6 , shift regions (e.g., shift regions x-1, x0, and x+1) can be determined by shifting in the horizontal direction. In the YNth phase image, shift regions (e.g., shift regions y-1, y0, and y+1) can be determined by shifting in the vertical direction.

[0066] In XN and YN, X indicates that phase characteristics are distinguished in the horizontal direction, and Y indicates that phase characteristics are distinguished in the vertical direction. Further, N denotes the number of phases. While the same number of phases is used in the vertical direction and the horizontal direction as described above, the number of phases used in the vertical direction and the number of phases used in the horizontal direction can be different from each other. For example, N can be determined based on the number of phases that can be distinguished by the sensor. In the example of the QPD sensor, N can be "2", and there can be a first phase image, an X2nd phase image, and a Y2nd phase image in the example of Figure 7A

[0067] In another example, as shown in Figure 7B ​As shown, the activity detection device can move the phase image of the QPD sensor in the horizontal, vertical, and diagonal directions, and can determine the shift region of each phase image. In the XNth phase image, the shift region (e.g., shift regions x-1, x0, and x+1) can be determined by the horizontal shift. In the YNth phase image, the shift region (e.g., shift regions y-1, y0, and y+1) can be determined by the vertical shift. In the ZNth phase image, the shift region (e.g., shift regions z-1, z0, and z+1) can be determined by the diagonal shift. In ZN, Z indicates that the phase characteristics are distinguished in the diagonal direction, and N represents the number of phases. For example, when N is "2", in Figure 7B The example can use the first phase image, the X2 phase image, the Y2 phase image, and the Z2 phase image.

[0068] When the shifted regions are determined as described above, the difference between the image of the base region and the image of each shifted region can be calculated in operation 620. The activity detection device can generate a difference image based on the difference between the fixed image (e.g., the image of the first base region) and the shifted image (e.g., the image of the shifted region), and can generate a minimum image based on the difference image. For example, the activity detection device can generate a first difference image based on the difference between the image of the first base region and the image of the shifted region x-1, a second difference image based on the difference between the image of the first base region and the image of the shifted region x0, and a third difference image based on the difference between the image of the first base region and the image of the shifted region x+1.

[0069] An activity detection device can assign index values ​​to each difference image. For example, an activity detection device can assign index values ​​in the order of x-1, x0, and x+1. Figure 6 As shown, index value "0" is assigned to the first difference image, index value "1" is assigned to the second difference image, and index value "2" is assigned to the third difference image. The index values ​​can be assigned in various orders.

[0070] A set of difference images, including the above difference images, can be generated for each phase image. For example, in Figure 7A In the example, a set of difference images of the XN-th phase image and a set of difference images of the YN-th phase image can be generated. Figure 7B In the example, a set of difference images for each of the XN-th phase image, the YN-th phase image, and the ZN-th phase image can be generated.

[0071] In operation 630, the activity detection device generates a minimum map. The activity detection device can select the minimum value among corresponding difference values ​​at corresponding positions in the difference images of the difference image set, and can determine the pixel values ​​of the minimum map based on the minimum value. In one example, in Figure 6In the above example, the corresponding difference values located at (1, 1) are "1", "0", and "6". Among "1", "0", and "6", "0" can be selected as the minimum value. In another example, the corresponding difference values located at (2, 2) are "25", "33", and "30". Among "25", "33", and "30", "25" can be selected as the minimum value. As described above, the minimum value can be selected among the corresponding difference values, and the pixel of the minimum map is determined based on the minimum value.

[0072] The pixel value of the minimum map can correspond to the minimum value or an index of the difference image including the minimum value among the difference images. The minimum map including the minimum value can be referred to as a "minimum value map", and the minimum map including the minimum index can be referred to as a "minimum index map". In the above example, "0" can be selected as the minimum value at (1, 1), and the index of the difference image including "0" can be "1". Accordingly, the pixel value at (1, 1) is "0" in the minimum value map, and the pixel value at (1, 1) is "1" in the minimum index map. Further, "25" can be selected as the minimum value at (2, 2), and the index of the difference image including "25" can be "0". Accordingly, the pixel value at (2, 2) is "25" in the minimum value map, and the pixel value at (2, 2) is "0" in the minimum index map.

[0073] As described above, the difference image set of each phase image can be generated. When there are phase images associated with a plurality of directions as in the example of Figure 7A and Figure 7B When there are phase images associated with a plurality of directions as in the example of Figure 7A , the minimum map of each of the XNth phase image and the YNth phase image can be generated. In the example of Figure 7B , the minimum map of each of the XNth phase image, the YNth phase image, and the ZNth phase image can be generated.

[0074] Figure 8 An example of operation 440 of Figure 4 , i.e., detecting activity using reference information and an activity detection model, is shown. Referring to Figure 8 , in operation 810, the activity detection device generates a reference image by concatenating the phase image and the minimum map. For example, data A and data B are concatenated (e.g., merged by concatenation) by a concatenation operation between data A and data B. As a result, the size of the data changes. For example, if each size of data A and data B is WxH, the size of the operation result can be 2WxH, Wx2H, WxHx2, etc. Concatenation can correspond to an example of combination. Hereinafter, an example of generating a reference image is further described with reference to Figure 9 ​

[0075] Figure 9 An example of generating a reference image is shown. Referring to Figure 9 When the phase characteristics are distinguished in the horizontal direction, the reference image can be generated by concatenating the first phase image, the Xth phase image (e.g., the second phase image), and the minimum map. To fit the size of each image, an image of the first basic region can be used instead of the first phase image, and an image of the second basic region can be used instead of the Xth phase image.

[0076] In operation 910, when the phase characteristics are distinguished in the horizontal direction and the vertical direction, additional phase images and additional minimum maps can be further concatenated. For example, in the example of Figure 7A the reference image can be generated by concatenating the first phase image, the Xth phase image, the Yth phase image, the first minimum map, and the second minimum map. In the example of Figure 7B the reference image can be generated by further concatenating the Zth phase image and the third minimum map. The first minimum map can be generated based on the first phase image and the Xth phase image, the second minimum map can be generated based on the first phase image and the Yth phase image. In addition, the third minimum map can be generated based on the first phase image and the Zth phase image.

[0077] In addition, to fit the size of each image, an image of the first basic region can be used instead of the first phase image, an image of the second basic region can be used instead of the Xth phase image, an image of the third basic region can be used instead of the Yth phase image, and an image of the fourth basic region can be used instead of the Zth phase image. The image of the third basic region can represent a region corresponding to the basic region in the Yth phase image, and the image of the fourth basic region can represent a region corresponding to the basic region in the Zth phase image.

[0078] Referring back to Figure 8 In operation 820, the activity detection device inputs input data corresponding to the reference image to the activity detection model. For example, the input data can correspond to the reference image, or to an image obtained by cropping the reference image. The reference image can be cropped in various versions based on the ROI.

[0079] For example, the ROI can correspond to a face box. In this example, a cropped image corresponding to the face box can be denoted by It, and a cropped image corresponding to m times the face box can be denoted by mx t (e.g., 2t in the case of a cropped image corresponding to twice the face box). The full-size reference image can be denoted by a "reduced reference image". In one example, the input data can include the cropped images It and 2t and the reduced reference image. Examples of the activity detection model will be further described below with reference to Figure 10 and Figure 11 Examples of the activity detection model will be further described below with reference to

[0080] In operation 830, the activity detection device detects the activity based on the output data of the activity detection model. The output data can include an activity score. The activity detection device can compare the activity score with a predetermined threshold to detect the activity of the object. The detection result can indicate whether the object corresponds to a real user or an attacker (such as an image acquired by photographing a user). In one example, when the activity score is greater than or equal to the predetermined threshold, the detection result can indicate that the object corresponds to a real user. When the activity score is less than the predetermined threshold, the detection result can indicate that the object corresponds to an attacker (such as an image acquired by photographing a user).

[0081] Figure 10 An example of generating output data using an activity detection model is illustrated. Referring to Figure 10 , the activity detection model generates input data 1030 based on a reference image 1010 and ROI information 1020. The ROI information can include information about a face box and can be generated by a face detector. The activity detection model can generate a patch by cropping the reference image 1010 based on the ROI information 1020. The input data 1030 can include the patch. When the reference image 1010 includes a plurality of concatenated images, the activity detection device can generate a patch by cropping each image based on the ROI information, and can generate the input data 1030 by concatenating the patches.

[0082] The activity detection model 1040 can include at least one neural network, and the at least one neural network can be pre-trained to detect the activity of the object included in the input data. The training data can include the input data and a label. In one example, when the input data corresponds to a real user, the label can have a relatively high activity score. When the input data corresponds to an attacker (such as an image), the label can have a relatively low activity score. The neural network can be trained based on the above training data to output the activity score of the input data. Figure 10 A state in which the training of the activity detection model 1040 is completed is illustrated.

[0083] The activity detection device can input the input data 1030 to the activity detection model 1040. In response to the input of the input data 1030, the activity detection model 1040 can output output data 1050. The output data 1050 can include an activity score. The activity detection device can detect the activity of the object by comparing the activity score with a predetermined threshold.

[0084] Figure 11 An example of generating output data using a plurality of activity detection models is illustrated. Referring to Figure 11The activity detection model generates input data 1130 based on the reference image 1110 and the ROI information 1120. The ROI information can include information about the face box. The activity detection model can generate a plurality of patches (e.g., patch 1t, patch 2t, and a down-scaled patch) by cropping the reference image 1110 based on the ROI information 1120.

[0085] For example, the activity detection model can generate patch 1t corresponding to the face box and patch 2t which is expanded to twice the face box. The down-scaled patch can represent the full size of the reference image 1110. A patch which is expanded to three times the face box (which can be denoted by 3t) can also be used instead of the down-scaled patch. The patch 1t, the patch 2t, and the down-scaled patch can include different characteristics of the object. For example, the patch 1t can include characteristics of the face, the patch 2t can include characteristics of the portion around the face, and the down-scaled patch can include characteristics associated with the background or context. The input data 1130 can include a plurality of patches.

[0086] The activity detection model 1140 can output output data 1150 associated with each patch in response to input of the input data 1130. For example, the activity detection model 1140 can include a first activity detection model, a second activity detection model, and a third activity detection model. The first activity detection model can output output data 1150 associated with patch 1t, the second activity detection model can output output data 1150 associated with patch 2t, and the third activity detection model can output output data 1150 associated with the down-scaled patch.

[0087] The output data 1150 can include an activity score associated with each patch. The activity detection device can perform a statistical operation (e.g., an averaging operation) based on the activity score associated with each patch, can compare the result of the operation with a predetermined threshold, and can detect the activity of the object. In addition, with reference to Figure 10 The description provided can be applicable to Figure 11 examples.

[0088] Figure 12A is a block diagram illustrating an example of an activity detection device 1200. With reference to Figure 12A , the activity detection device 1200 includes a processor 1210 and a memory 1220. The memory 1220 is connected to the processor 1210 and can store instructions executable by the processor 1210, data to be processed by the processor 1210, or data processed by the processor 1210. The memory 1220 can include, for example, a non-transitory computer readable storage medium (e.g., a high-speed random access memory (RAM)) and / or a non-volatile computer readable storage medium (e.g., at least one disk storage, a flash memory device, or other non-volatile solid-state memory device).

[0089] The processor 1210 can execute instructions to perform the above-described operations with reference to Figures 1 to 11At least one of the described operations. For example, the processor 1210 can generate a first phase image based on first visual information of a first phase sensed by a first pixel group of the image sensor, can generate a second phase image based on second visual information of a second phase sensed by a second pixel group of the image sensor, can generate a minimum map based on a disparity between the first phase image and the second phase image, and can detect an activity based on the minimum map.

[0090] Figure 12B is a block diagram illustrating an example of an activity detection device 1250. Referring to Figure 12B , the activity detection device 1250 includes a multi-phase detection sensor 1251, a multi-phase image pre-processor 1252, an ROI detector 1253, a multi-phase block generator 1254, and an activity detector 1255. The multi-phase detection sensor 1251, the multi-phase image pre-processor 1252, the ROI detector 1253, the multi-phase block generator 1254, and the activity detector 1255 can be implemented as at least one hardware module, at least one software module, and / or a combination thereof.

[0091] Operations related to activity detection will be described below with respect to each of the multi-phase detection sensor 1251, the multi-phase image pre-processor 1252, the ROI detector 1253, the multi-phase block generator 1254, and the activity detector 1255, however, the operations need not be performed by separate components such as the multi-phase detection sensor 1251, the multi-phase image pre-processor 1252, the ROI detector 1253, the multi-phase block generator 1254, and the activity detector 1255. For example, operations described as being performed by one component can be performed by another component, or the above operations can be performed by a single integrated component (e.g., the activity detection device 1250).

[0092] The multi-phase detection sensor 1251 can sense visual information of a plurality of phases, and can generate sensor data associated with visual information of each phase. For example, the multi-phase detection sensor 1251 can be a 2PD sensor for sensing two types of phases, a QPD sensor for sensing four types of phases, or a sensor for sensing various types of phases. The multi-phase detection sensor 1251 can sense visual information having different phase characteristics using sensing pixels disposed adjacent to each other, and can generate sensor data based on the sensed visual information. Based on the sensor data, a phase image corresponding to each phase characteristic can be generated.

[0093] The multi-phase image preprocessor 1252 can perform pre-processing of the phase images. For example, the multi-phase image preprocessor 1252 can perform pre-processing including any one or any combination of down-sizing, lens shading correction, gamma correction, histogram matching, and de-noising. In one example, the multi-phase image preprocessor 1252 can not perform pre-processing such as distortion correction. This is because it can be desirable to maintain the shape of the object to detect fine parallax, but pre-processing such as distortion correction can change the shape of the object.

[0094] The ROI detector 1253 can detect an ROI in the phase images. For example, the ROI can correspond to a face box in each of the phase images. The ROI detector 1253 can specify the ROI based on coordinate information and / or size information. In one example, the phase images can be resized to fit the input size of the ROI detector 1253 and can be input to the ROI detector 1253.

[0095] The multi-phase block generator 1254 can generate a minimum map based on the phase images (e.g., the phase images to which pre-processing is applied), and can generate a reference image based on the minimum map. For example, the multi-phase block generator 1254 can shift at least one of the other phase images at least once in the state of one of the fixed phase images, and can generate at least one minimum map based on the difference between the fixed image and the shifted image. The multi-phase block generator 1254 can generate a reference image by concatenating the phase images and the at least one minimum map.

[0096] The multi-phase block generator 1254 can generate at least one block by cropping the reference image based on the ROI. The at least one block can be used to generate an input image of the activity detector 1255. For example, the multi-phase block generator 1254 can generate a block 1t corresponding to the face box and a block 2t extended to twice the face box by cropping the reference image based on the ROI. Also, the multi-phase block generator 1254 can prepare a down-sized block corresponding to the full size of the reference image. The multi-phase block generator 1254 can generate input data based on the block 1t, the block 2t, and the down-sized block. For example, the multi-phase block generator 1254 can concatenate each block, and can perform resizing to fit the input size of the activity detector 1255.

[0097] The activity detector 1255 can detect the activity of the object based on the input data. For example, the activity detector 1255 can include at least one neural network pre-trained to detect the activity of the object in the input data. The at least one neural network can output output data including an activity score in response to the input of the input data. The activity detector 1255 can compare the activity score with a threshold to detect the activity of the object.

[0098] Figure 13 is a block diagram illustrating an example of an electronic device 1300. Refer to FIG. 12.Figure 13 The electronic device 1300 can generate an input image including an object, and can detect the liveliness of the object in the input image. Further, the electronic device 1300 can perform a biometric authentication (e.g., an image-based biometric authentication such as a face authentication or an iris authentication) based on the liveliness of the object. The electronic device 1300 can structurally or functionally include Figure 1 the liveliness detection device 100 of FIG. 1, Figure 12A the liveliness detection device 1200 of FIG. 12, and / or Figure 12B the liveliness detection device 1250 of FIG. 13.

[0099] The electronic device 1300 includes a processor 1310, a memory 1320, a camera 1330, a storage 1340, an input device 1350, an output device 1360, and a network interface 1370. The processor 1310, the memory 1320, the camera 1330, the storage 1340, the input device 1350, the output device 1360, and the network interface 1370 can communicate with each other via a communication bus 1380. For example, the electronic device 1300 can be implemented as at least a part of, for example, a mobile device (such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, or a laptop computer), a wearable device (such as a smart watch, a smart band, or smart glasses), a computing device (such as a desktop computer or a server), a home appliance (such as a television (TV), a smart TV, or a refrigerator), a security device (such as a door lock), and a vehicle (such as a smart vehicle).

[0100] The processor 1310 can execute instructions and functions in the electronic device 1300. For example, the processor 1310 can process instructions stored in the memory 1320 or the storage 1340. The processor 1310 can perform at least one of the operations described above with reference to Figures 1 to 12B FIGS. 1 to 13.

[0101] The memory 1320 can store data for detecting liveliness. The memory 1320 can include a non-transitory computer readable storage medium or a non-transitory computer readable storage device. The memory 1320 can store instructions to be executed by the processor 1310, and also store information associated with software and / or applications when the software and / or applications are executed by the electronic device 1300.

[0102] The camera 1330 can photograph still images, videos, or both. For example, the camera 1330 can photograph a face image including a face of a user. In one example, the camera 1330 can provide a 3D image including depth information of an object. The camera 1330 can include an image sensor that detects a plurality of phases (e.g., a 2PD sensor or a QPD sensor).

[0103] The storage 1340 can include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The storage 1340 can store data used by the activity detection process or various models such as an activity detection model or a face detector. In one example, the storage 1340 can store a larger amount of information than the memory 1320 for a relatively long period of time. For example, the storage 1340 can include a magnetic hard disk, a compact disk, a flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0104] The input device 1350 can receive input from a user through a conventional input scheme using a keyboard and a mouse as well as through a new input scheme such as a touch input, a voice input, or an image input. For example, the input device 1350 can detect input from a keyboard, a mouse, a touch screen, a microphone, or a user, and can include any other device configured to transfer the detected input to the electronic device 1300.

[0105] The output device 1360 can provide output of the electronic device 1300 to the user through a visual, auditory, or tactile channel. The output device 1360 can include, for example, a display, a touch screen, a speaker, a vibration generator, or other devices configured to provide output to the user. The network interface 1370 communicates with external devices via wired or wireless networks.

[0106] The devices, units, modules, apparatuses, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to implement the hardware components described in this application, depending on the context, include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic module units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to carry out the operations described in this application. In other examples, one or more of the hardware components that carry out the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic gates arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. The one or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components. The hardware components can have any one or more of various processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessor, single-instruction multiple-data (SIMD) multiprocessor, multiple-instruction single-data (MISD) multiprocessor, and multiple-instruction multiple-data (MIMD) multiprocessor.

[0107] The methods that perform the operations described in this application are performed by computing hardware (e.g., by one or more processors or computers) that is implemented in hardware as described above to execute instructions or software to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations can be performed by a single processor or by two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. A processor or processors, or a processor and a controller, can perform a single operation or two or more operations.

[0108] The instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above can be written as a computer program, a code segment, instructions, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as produced by a compiler) directly executable by a processor or computer. In another example, the instructions or software include higher-level code (which is partially or fully interpreted) that is executed by a processor or computer using an interpreter. Programmers of ordinary skill in the art can easily write the instructions or software based on the flow charts and block diagrams illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0109] The instructions or software for controlling the processor or computer-implemented hardware components and performing the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), a card-type memory such as a multimedia card or a micro card (e.g., a secure digital (SD) or extreme digital (XD)), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0110] While the present disclosure includes certain examples, it will be clear to one of ordinary skill in the art that various changes in form and details can be made without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example should be considered as being applicable to other examples in the absence of contrary text. If the described technology is performed in a different order, and / or if the components described in the described system, architecture, device, or circuit are combined in a different manner, or replaced or supplemented by other components or their equivalents, appropriate results can be obtained. Accordingly, the scope of the disclosure is not limited by the specific embodiments described herein, but only by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

Claims

1. An activity detection method, comprising: generating a first phase image based on first visual information sensed by a first group of pixels of an image sensor at a first phase; generating a second phase image based on second visual information sensed by a second group of pixels of the image sensor at a second phase; generating a min map based on a disparity between the first phase image and the second phase image; and detecting an activity based on the min map, wherein the generating the min map comprises: shifting one of the first phase image and the second phase image while fixing the other; and generating the min map based on a difference between the shifted image and the fixed image. The generating the min map comprises:

2. The activity detection method according to claim 1, wherein, setting a first base region in the first phase image; setting a second base region corresponding to the first base region in the second phase image; setting at least one shifted region by shifting the second base region by a reference shift value; generating a plurality of difference images based on a difference between an image of the first base region and an image of the second base region and a difference between the image of the first base region and at least one image of the at least one shifted region; and generating the min map based on the plurality of difference images. The generating the min map based on the plurality of difference images comprises:

3. The activity detection method according to claim 2, wherein, selecting a minimum value among corresponding difference values at positions corresponding to each other in the plurality of difference images; and determining a pixel value of the min map based on the minimum value. The pixel value of the min map corresponds to the minimum value or to an index of a difference image including the minimum value among the plurality of difference images.

4. The activity detection method according to claim 3, wherein, The detecting the activity comprises:

5. The activity detection method according to any one of claims 1 to 4, wherein, inputting input data including at least one patch based on the min map to at least one activity detection model; and detecting the activity based on an output of the at least one activity detection model, wherein the at least one activity detection model comprises at least one neural network, and wherein the at least one neural network is pre-trained to detect an activity of an object. 6.The activity detection method of claim 5, further comprising: generating a reference image by concatenating the first phase image, the second phase image, and the min map, wherein the detecting the activity comprises generating the at least one patch by cropping the reference image based on a region of interest. 7.The activity detection method of claim 6, wherein: the at least one patch comprises a plurality of patches having different characteristics of the object, the at least one activity detection model comprises a plurality of activity detection models processing input data including the plurality of patches, and the detecting the activity based on the output of the at least one activity detection model comprises detecting the activity by fusing outputs of the plurality of activity detection models in response to an input of the input data. 8.The activity detection method of any one of claims 1 to 4, further comprising: generating a reference image by concatenating the first phase image, the second phase image, and the min map; and detecting the activity based on the reference image. 9.The activity detection method of any one of claims 1 to 4, further comprising: performing a pre-processing of the first phase image and the second phase image, ​ ​ The step of performing pre-processing includes applying any one or any combination of down-sizing, lens shading correction, gamma correction, histogram matching, and de-noising to the first phase image and the second phase image.

10. The activity detection method according to any one of claims 1 to 4, wherein, The first pixels of the first pixel group and the second pixels of the second pixel group are disposed adjacent to each other.

11. The activity detection method of any one of claims 1 to 4, further comprising: generating a third phase image based on third visual information of a third phase sensed by a third pixel group of the image sensor; generating a fourth phase image based on fourth visual information of a fourth phase sensed by a fourth pixel group of the image sensor, generating a minimum map using a disparity between the first phase image and the third phase image and a disparity between the first phase image and the fourth phase image.

12. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the activity detection method of any one of claims 1 to 11.

13. An activity detection device comprising: a processor; and a memory including instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to: generate a first phase image based on first visual information of a first phase sensed by a first pixel group of the image sensor; generate a second phase image based on second visual information of a second phase sensed by a second pixel group of the image sensor; generate a minimum map based on a disparity between the first phase image and the second phase image; and detect an activity based on the minimum map, wherein the processor is configured to: shift one of the first phase image and the second phase image while fixing the other; and generate the minimum map based on a difference between the shifted image and the fixed image. The processor is configured to:

14. The activity detection device of claim 13, wherein, set a first base region in the first phase image; set a second base region corresponding to the first base region in the second phase image; set at least one shifted region by shifting the second base region by a reference shift value; generate a plurality of difference images based on a difference between an image of the first base region and an image of the second base region and a difference between the image of the first base region and at least one image of the at least one shifted region; and generate the minimum map based on the plurality of difference images. The processor is configured to select a minimum value among corresponding difference values at positions corresponding to each other in the plurality of difference images and determine a pixel value of the minimum map based on the minimum value. The processor is configured to generate a reference image by concatenating the first phase image, the second phase image, and the minimum map and detect the activity based on the reference image.

15. The activity detection device of claim 14, wherein, The first pixels of the first pixel group and the second pixels of the second pixel group are disposed adjacent to each other.

16. The activity detection device according to any one of claims 13 to 15, wherein, 18. An electronic device comprising:

17. The activity detection device according to any one of claims 13 to 15, wherein, an image sensor configured to sense first visual information of a first phase using a first pixel group and to sense second visual information of a second phase using a second pixel group; and a processor configured to generate a first phase image based on the first visual information, to generate a second phase image based on the second visual information, to generate a minimum map based on a disparity between the first phase image and the second phase image, and to detect an activity based on the minimum map. ​ ​ the processor is configured to generate a first phase image based on first visual information, generate a second phase image based on second visual information, generate a minimum map based on a disparity between the first phase image and the second phase image, and detect an activity based on the minimum map, wherein the processor is configured to: shift one of the first phase image and the second phase image while fixing the other; and generate the minimum map based on a difference between the shifted image and the fixed image.

19. The electronic device of claim 18, wherein, the processor is configured to: set a first base region in the first phase image; set a second base region corresponding to the first base region in the second phase image; set at least one shifted region by shifting the second base region by a reference shift value; generate a plurality of difference images based on a difference between an image of the first base region and an image of the second base region and a difference between the image of the first base region and at least one image of the at least one shifted region; and generate the minimum map based on the plurality of difference images. the processor is configured to select a minimum value among corresponding difference values at positions corresponding to each other in the plurality of difference images, and determine a pixel value of the minimum map based on the minimum value.

20. The electronic device of claim 19, wherein, 21. An activity detection apparatus comprising: a multi-phase detection sensor configured to generate a first phase image by sensing first visual information of a first phase using a first pixel group and generate a second phase image by sensing second visual information of a second phase using a second pixel group; a multi-phase block generator configured to generate a minimum map based on a disparity between the first phase image and the second phase image; and an activity detector configured to detect an activity based on the minimum map, wherein the multi-phase block generator is configured to: shift one of the first phase image and the second phase image while fixing the other; and generate the minimum map based on a difference between the shifted image and the fixed image. the multi-phase block generator is configured to:

22. The activity detection device of claim 21, wherein, set a first base region in the first phase image; set a second base region corresponding to the first base region in the second phase image; set at least one shifted region by shifting the second base region by a reference shift value; generate a plurality of difference images based on a difference between an image of the first base region and an image of the second base region and a difference between the image of the first base region and at least one image of the at least one shifted region; and generate the minimum map based on the plurality of difference images.

23. The activity detection apparatus according to claim 21 or 22, further comprising: a region of interest detector configured to detect a region of interest from the first phase image and the second phase image, wherein the multi-phase block generator is configured to generate a reference image by concatenating the first phase image, the second phase image, and the minimum map, and generate at least one block by cropping the reference image based on the region of interest, and wherein the activity detector is configured to detect the activity based on the at least one block.

24. The activity detection apparatus according to claim 21 or 22, further comprising: ​ ​ A multi-phase image pre-processor configured to apply any one or any combination of down-sizing, lens shading correction, gamma correction, histogram matching, and de-noising to the first phase image and the second phase image. 25.An electronic device comprising: one or more processors; and at least one memory storing instructions executable by the one or more processors, wherein, in response to the instructions being executed by the one or more processors, the one or more processors are configured to: input an image including a subject; generate disparity data based on disparity between a first phase image corresponding to the subject and a second phase image corresponding to the subject; generate a reference image based on the first phase image, the second phase image, and the disparity data; generate input data based on the reference image; input the input data to a detection model including a neural network; and authenticate the subject based on output data of the detection model, wherein the first phase image is generated based on first visual information of a first phase sensed by a first pixel group of an image sensor, and the second phase image is generated based on second visual information of a second phase sensed by a second pixel group of the image sensor, wherein the one or more processors are configured to: shift one of the first phase image and the second phase image while fixing a state of the other; and generate a minimum map based on a difference between the shifted image and the fixed image.

26. The electronic device of claim 25, wherein, the one or more processors are configured to authenticate the subject by determining liveliness of the subject based on the output data.

27. The electronic device of claim 25 or 26, wherein, the one or more processors are configured to generate the reference image by concatenating the first phase image, the second phase image, and the disparity data.

Images