Identification device and electronic device

Abstract

The present invention provides an identification device and an electronic device that can perform identification with high accuracy without being affected by changes in ambient light. The present invention provides an identification device, comprising: a direct reflection light information calculation unit that calculates direct reflection light information related to direct reflection light from an object based on sensor data obtained by a TOF sensor that irradiates light on an object and detects the light; an object detection unit that detects the object based on the direct reflection light information; and an object identification unit that identifies the object based on the direct reflection light information of the detected object.

Description

Technical Field

[0001] The present disclosure relates to an identification device and an electronic device. Background Art

[0002] In recent years, electronic devices such as smart phones are often equipped with recognition devices to ensure their safety. Specifically, the recognition device takes a facial image of a person who wants to use the electronic device and compares it with the facial image. Only the person who is recognized as the user of the electronic device is allowed to use the electronic device.

[0003] For example, the following patent document 1 discloses a recognition device that pre-registers a user's facial image and compares the registered facial image with a newly captured facial image. In the patent document 1, lighting is controlled so that a new facial image is captured under the same lighting conditions as those used when the registered facial image was captured, thereby improving recognition accuracy.

[0004] Prior art literature

[0005] Patent Document 1: Japanese Patent Publication No. 2017-27492

[0006] When the recognition accuracy is to be improved, it is desirable to capture a new face image under the same ambient light as that when the registered face image was captured, as described above. However, the ambient light is easily variable, and in some cases it is difficult to reproduce the same ambient light as that when the registered face image was captured for capturing. Summary of the invention

[0007] Therefore, the present invention provides a novel and improved recognition device and electronic device that are not affected by changes in ambient light and can perform high-precision recognition.

[0008] According to the present invention, there is provided an identification device, comprising: a direct reflected light information calculation unit, which calculates direct reflected light information related to direct reflected light from an object based on sensor data obtained by a TOF sensor that irradiates light onto an object and detects the light; an object detection unit, which detects the object based on the direct reflected light information; and an object identification unit, which identifies the object based on the direct reflected light information of the detected object.

[0009] Furthermore, according to the present invention, there is provided an identification device, comprising: a distance information calculation unit, which calculates the distance information of the object based on sensor data obtained by a TOF sensor that irradiates light onto the object and detects the light; an object detection unit, which detects the object based on the distance information; and an object identification unit, which identifies the object based on the distance information of the detected object.

[0010] In addition, according to the present invention, there is provided an electronic device equipped with a recognition device, the recognition device comprising: a direct reflected light information calculation unit, which calculates direct reflected light information related to direct reflected light from the object based on sensor data obtained by a TOF sensor that irradiates light on the object and detects the light; an object detection unit, which detects the object based on the direct reflected light information; and an object recognition unit, which recognizes the object based on the direct reflected light information of the detected object.

[0011] As described above, according to the present invention, it is possible to provide a recognition device and an electronic device that are not affected by changes in ambient light and can perform high-precision recognition.

[0012] In addition, the above-mentioned effects are not limitative effects, and any of the effects described in this specification or other effects that can be understood from this specification may be achieved together with or instead of the above-mentioned effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is an explanatory diagram for explaining a recognition method performed by a recognition device of a comparative example.

[0014] Figure 2 This is an explanatory diagram for explaining an image used in the recognition device according to the embodiment of the present invention.

[0015] Figure 3 This is an explanatory diagram for explaining the difference between the recognition method according to the embodiment of the present invention and the recognition method of the comparative example.

[0016] Figure 4 It is a block diagram showing a configuration example of the recognition system 10 according to the first embodiment of the present invention.

[0017] Figure 5 : is a block diagram showing a configuration example of the TOF sensor 100 according to this embodiment.

[0018] Figure 6 This is an explanatory diagram for explaining the principle of a method for calculating distance information.

[0019] Figure 7 It is an explanatory diagram for explaining a method of calculating distance information using the TOF sensor 100 according to this embodiment.

[0020] Figure 8 It is an explanatory diagram schematically showing elimination of ambient light (indirect reflected light) in this embodiment.

[0021] Fig. 9This is an explanatory diagram for explaining an example of normalization in this embodiment.

[0022] Fig.10 Detailed description of the flowchart of the registration phase of the identification method according to the present embodiment.

[0023] Fig.11 Detailed description of the flowchart of the recognition phase of the recognition method according to the embodiment.

[0024] Fig.12 This is a flowchart for explaining the registration phase of the identification method according to the second embodiment of the present invention.

[0025] Fig.13 Detailed description of the flowchart of the recognition phase of the recognition method according to the embodiment.

[0026] Fig.14 It is a diagram showing a configuration example of a stacked image sensor 20 according to a third embodiment of the present invention.

[0027] Fig.15 2 is a block diagram showing a detailed configuration example of the stacked image sensor 20 according to this embodiment.

[0028] Fig.16 It is a diagram showing a configuration example of a stacked image sensor 20 a according to a modification of the embodiment.

[0029] Fig.17 It is a block diagram showing a detailed configuration example of a stacked image sensor 20 a according to a modification of the embodiment.

[0030] Fig.18 This is a block diagram showing an example of the hardware configuration of an electronic device 900 according to the fourth embodiment of the present invention.

[0031] Description of Reference Numerals

[0032] 10 Identification System

[0033] 20, 20a Stacked Image Sensor

[0034] 100, 980 TOF sensor

[0035] 102 Irradiation Department

[0036] 104, 104a, 104b light receiving part

[0037] 120 pixel area

[0038] 200 processing units

[0039] 202 Distance information calculation unit

[0040] 204 Direct reflected light calculation unit

[0041] 206 Subject detection unit

[0042] 208 3D Transformation Unit

[0043] 210 Subject Normalization Department

[0044] 212 Subject Recognition Unit

[0045] 220 Signal processing circuit area

[0046] 300 Storage

[0047] 320 memory area

[0048] 400 Display unit

[0049] 502, 502a: facial image for registration

[0050] 504, 504b, 504c, 504d: facial image for comparison

[0051] 600 Distance Image

[0052] 602, 602a: distance image for registration

[0053] 604, 604b, 604c, 604d: distance images for comparison

[0054] 700 Direct reflected light image

[0055] 702, 702a Direct reflected light image for registration

[0056] 704, 704b, 704c, 704d Comparison with direct reflected light images

[0057] 612, 620, 622, 710, 712 Subject area

[0058] 800a, 800b area

[0059] 900 Electronic equipment

[0060] 950 CPU

[0061] 952 ROM

[0062] 954 RAM

[0063] 956 Recording Media

[0064] 958 Input and Output Interface

[0065] 960 Operation Input Device

[0066] 962 Display devices

[0067] 968 Communication Interface

[0068] 970 Bus DETAILED DESCRIPTION

[0069] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in this specification and the accompanying drawings, components having substantially the same functional structure are denoted by the same reference numerals, thereby omitting repeated descriptions.

[0070] Furthermore, in this specification and the drawings, for multiple components having substantially the same or similar functional structures, different numbers are sometimes marked after the same reference numerals to distinguish them. However, in the case where there is no need to particularly distinguish each of the multiple components having substantially the same or similar functional structures, only the same reference numerals are marked. Furthermore, for similar components of different embodiments, different letters are sometimes marked after the same reference numerals to distinguish them. However, in the case where there is no need to particularly distinguish each of the similar components, only the same reference numerals are marked.

[0071] In addition, the description is given in the following order.

[0072] 1. Background of the invention

[0073] 2. Overview of Embodiments of the Invention

[0074] 3. First Implementation Method

[0075] 3.1 Overview of the Recognition System 10 According to the First Embodiment

[0076] 3.2 Detailed Structure of TOF Sensor 100

[0077] 3.3 Detailed Structure of Processing Unit 200

[0078] 3.4 Identification Method

[0079] 3.4.1 Registration stage

[0080] 3.4.2 Identification Phase

[0081] 4. Second Implementation Method

[0082] 4.1 Identification Method

[0083] 4.1.1 Registration stage

[0084] 4.2.2 Identification Phase

[0085] 5. Third Implementation Method

[0086] 6. Fourth Implementation Method

[0087] 7. Summary

[0088] 8. Supplement

[0089] <<1. Background of the invention of the present invention>>

[0090] Before describing the details of the embodiments of the present invention, Figure 1 The background in which the present inventors created the embodiments of the present invention will be described. Figure 1 It is an explanatory diagram for explaining a recognition method performed by a recognition device of a comparative example. Here, the comparative example means a recognition device or a recognition method that the inventors have continuously studied before creating the embodiment involved in the present invention.

[0091] In the recognition device involved in the comparative example, the facial image of a specific person is pre-registered, and the person whose facial image is newly captured is recognized by comparing the newly captured facial image with the pre-registered facial image. However, in the recognition device involved in the comparative example, sometimes the result of the above-mentioned comparison is that the person is recognized as another person even though it is actually the same person, and there is a limit to improving the recognition accuracy.

[0092] Reference Figure 1 The following specifically describes the case where the above-mentioned erroneous recognition is performed in the recognition device involved in the comparative example. Figure 1 At this time, the recognition device involved in the comparative example newly captures a facial image of a person, obtains a comparison facial image 504, and compares the newly obtained comparison facial image 504 with the above-mentioned registration facial image 502.

[0093] For example, when the recognition device according to the comparative example captures the same person as the registration facial image 502 under different lighting conditions than when the registration facial image 502 was captured, the recognition device may obtain Figure 1 Specifically, when the registration facial image 502 is taken, light is irradiated to the front of the person's face (front light), as shown in FIG. Figure 1 As shown on the left side of FIG. 5 , the registration face image 502 is a clear image covering the entire face. On the other hand, when the comparison face image 504b is taken, light is irradiated only to the left half of the person's face, as shown in FIG. Figure 1As shown in the upper right part of FIG. 504 , the difference in shadows in the comparison face image 504b is large. Specifically, half of the face is clear and the remaining half is unclear. In addition, when the comparison face image 504c was taken, the entire face of the person was not fully illuminated by light, as shown in FIG. Figure 1 As shown in the lower right part of , the comparison facial image 504c becomes an unclear image covering the entire face.

[0094] That is, the comparison facial images 504b and 504c are different from the registration facial image 502, and are images in which the whole or a part of the registration facial image is not clear. Therefore, when the registration facial image 502 is used to compare with the comparison facial images 504b and 504c, although these images 502, 504b, and 504c are facial images of the same person, the recognition device involved in the comparative example determines that they are facial images of different persons. Furthermore, as a result, the recognition device involved in the comparative example cannot recognize that they are the same person.

[0095] The reason why the recognition device according to the comparative example makes misrecognition is that the lighting conditions when the registration facial image 502 is taken are different from the lighting conditions when the comparison facial image 504 is taken. Therefore, although they are facial images of the same person, the facial images are taken in different states. That is, in the recognition device according to the comparative example, it can be said that its recognition accuracy is easily affected by the change of the lighting conditions (ambient light) when taking pictures.

[0096] Therefore, in the recognition device disclosed in the above-mentioned Patent Document 1 described above, the lighting is controlled so that the lighting conditions when the comparison facial image 504 is photographed are the same as the lighting conditions when the registration facial image 502 is photographed. By doing so, when the comparison facial image 504 is photographed, the lighting conditions can be the same as the lighting conditions when the registration facial image 502 is photographed, so if it is the same person, a substantially identical (substantially equivalent) facial image is photographed. Therefore, in the recognition device disclosed in the above-mentioned Patent Document 1, the probability of obtaining the recognition result that it is the same person can be increased, so the recognition accuracy can be improved.

[0097] Specifically, in the above-mentioned patent document 1, the lighting conditions when the registration facial image 502 is taken are estimated, and the lighting is controlled so that the lighting conditions when the comparison facial image 504 is taken are the same. However, in outdoor environments, the lighting conditions are affected by the fluctuation of sunlight, etc., so it is difficult to stably control the lighting conditions to the desired conditions. In addition, since the lighting conditions when the registration facial image 502 is taken are estimated and the lighting is controlled, in the above-mentioned patent document 1, it is difficult to avoid the processing time being prolonged or the power consumption being increased, and it is also difficult to avoid the complexity of the structure of the recognition device or the increase of the manufacturing cost.

[0098] Therefore, in view of the above-mentioned situation, the inventors of the present invention consider using distance information (2.5-dimensional information or 3-dimensional information) representing the depth information of the subject (object) for identification instead of using a 2-dimensional image (such as the above-mentioned registration facial image 502, the control facial image 504, and the like) that is easily affected by changes in ambient light. In addition, here, 2.5-dimensional information refers to information generated by associating the distance information (depth information) obtained for each pixel of the TOF sensor described later and the position information of the corresponding pixel. And, here, 3-dimensional information refers to 3-dimensional coordinate information in the actual space (to be specific, a collection of multiple 3-dimensional coordinate information) generated by transforming the position information of the pixel in the 2.5-dimensional information into coordinates in the actual space and associating it with the distance information corresponding to the transformed coordinates.

[0099] As one of the methods for obtaining distance information, a method using a stereo camera can be cited. A stereo camera uses two cameras to obtain images, and uses the parallax of these cameras to obtain distance information to the subject. However, in a stereo camera, since two cameras are used, it is difficult to avoid the structure of the device becoming larger. In addition, according to the research of the inventors, it is difficult for a stereo camera to obtain distance information for a uniform surface without patterns. For example, it is difficult to obtain distance information for skin areas with few patterns such as the face. In addition, the accuracy of the distance information obtained by the stereo camera is easily affected by changes in ambient light.

[0100] Furthermore, as another method of obtaining distance information, the structured light method can be cited. The structured light method is a method in which light having a predetermined pattern is projected onto the surface of the subject, and the deformation of the pattern of light projected onto the subject is analyzed to estimate the distance to the subject. The structured light method can be said to be a method that is less susceptible to changes in ambient light than the comparative example, but it is difficult to completely eliminate the influence of changes in ambient light. In addition, in the structured light method, an image of a subject in a state where a predetermined pattern is projected is captured. When such an image is used for the recognition of a person or the like, it is affected by the projected pattern and it is difficult to improve the recognition accuracy.

[0101] As a method for obtaining distance information, there is also a method of continuously photographing a subject while moving a camera around the subject, obtaining multiple photographing frames of the subject, and calculating the distance information of the subject based on the obtained multiple photographing frames. However, it is also difficult to eliminate the influence of changes in ambient light in this method. In addition, in this method, it takes time to obtain multiple photographing frames. In addition, when the subject moves or the shape of the subject changes, the distance information cannot be calculated, so it is difficult to apply to a recognition device for recognizing a person or the like.

[0102] Furthermore, although it is also conceivable to use both a camera that captures a two-dimensional image as in the comparative example and a camera that obtains the distance information as described above for recognition, since multiple cameras are used, it is inevitable that the structure of the recognition device becomes larger.

[0103] Therefore, based on the above research, the inventors have created a recognition device according to an embodiment of the present invention that is not affected by changes in ambient light and can perform high-precision recognition. The following describes the details of the embodiments of the present invention created by the inventors in sequence.

[0104] <<2. Overview of Embodiments of the Invention>>

[0105] First, refer to Figure 2 as well as Figure 3 An outline of embodiments of the present invention will be described. Figure 2 is an explanatory diagram for explaining an image used in a recognition device according to an embodiment of the present invention. Figure 3 This is an explanatory diagram for explaining the difference between the recognition method according to the embodiment of the present invention and the recognition method of the comparative example.

[0106] Based on the above-mentioned research, the inventors consider using a TOF (Time Of Flight) sensor to obtain distance information, etc., and to perform identification based on the obtained distance information, etc. The TOF sensor, for example, irradiates a subject with an irradiation light having a predetermined period, detects the light (reflected light) reflected by the subject, and detects the phase difference or time difference between the irradiation light and the reflected light, thereby being able to obtain the depth (distance information) of the subject. In addition, in an embodiment of the present invention created by the inventors, the TOF sensor is a sensor that can obtain the depth of the subject by detecting the phase difference between the irradiation light and the reflected light.

[0107] The TOF sensor can obtain distance information in the above manner. Therefore, for example, the TOF sensor involved in the present embodiment described below can obtain the following distance information. Figure 2 An image 600 (hereinafter referred to as distance image 600) based on distance information of a subject (here, a person's face) as shown on the right side of . The distance image 600 is an image obtained by projecting 2.5-dimensional information onto a plane (by assigning a color or brightness corresponding to the distance information), wherein the above-mentioned 2.5-dimensional information is obtained by associating distance information obtained based on reflected light reflected from a person's face with position information of pixels of the TOF sensor. Even if the ambient light changes, the distance information of a constant value can be obtained, so the distance information can be said to be information that is not affected by changes in the ambient light. In addition, the distance image 600 may also be an image obtained by projecting the above-mentioned 3-dimensional information onto a plane.

[0108] In addition, the TOF sensor can irradiate light (such as infrared light) toward the subject and detect the light reflected from the subject, so an image based on the detected reflected light (such as an infrared image) can also be obtained simultaneously with the distance image 600. Specifically, the TOF sensor involved in this embodiment can also obtain the following Figure 2 The image 700 (hereinafter referred to as the direct reflected light image 700) based on the direct reflected light information of the subject (here, the face of a person) is shown on the left side of FIG. The direct reflected light image 700 is an image based on the direct reflected light information obtained by irradiating the subject with irradiation light using the TOF sensor and detecting the reflected light directly reflected by the subject. More specifically, the direct reflected light image 700 is as shown in FIG. Figure 2 The left side of FIG. 1 shows an image that uses grayscale (gradation) to represent the intensity of light detected by the TOF sensor. However, in the embodiment of the present invention created by the inventor, the direct reflected light image 700 is different from the registration face image 502 and the comparison face image 504 (see FIG. 5 ) of the comparative example. Figure 1), but because it is an image obtained by processing the information of light detected by the TOF sensor (sensing data) in a manner that eliminates the influence of ambient light, it is very different from the comparative example. It is also clear from the fact that the influence of ambient light is eliminated that the direct reflected light image 700 can be said to be an image that is not affected by changes in ambient light. That is, the inventor independently came up with the idea of ​​using the direct reflected light image 700 for recognition, and the direct reflected light image 700 is based on the information (direct reflected light information) obtained by processing the sensing data of the TOF sensor in a manner that eliminates the influence of ambient light. In addition, the details of the elimination of the influence of ambient light in the embodiment of the present invention created by the inventor are described later.

[0109] In addition, the distance image 600 and the direct reflected light image 700 can be obtained simultaneously through a single shot by the TOF sensor involved in this embodiment. Therefore, in this embodiment, it is not necessary to shoot multiple image frames in order to obtain two images, etc., and the increase in the time involved in recognition can be avoided. In addition, the TOF sensor involved in this embodiment irradiates light to the subject, so in this embodiment, the subject can be recognized even in a dark place. In addition, Figure 2 The distance image 600 and the direct reflected light image 700 shown are merely examples, and the distance image 600 and the direct reflected light image 700 involved in this embodiment are not limited to the distance image 600 and the direct reflected light image 700. Figure 2 Example.

[0110] As described above, in an embodiment of the present invention created by the inventor, by using at least one of the distance information (distance image 600) that is not easily affected by changes in ambient light and the direct reflected light information (direct reflected light image 700) for identification, it will not be affected by changes in ambient light and can be identified with high precision.

[0111] Specifically, refer to Figure 3 , while comparing the recognition method involved in the embodiment of the present invention with the recognition method of the comparative example. Figure 3 The upper left portion of FIG. 5 shows a facial image 502a for registration in the recognition method according to the comparative example. Figure 3 The lower left side of shows the facial images 504b, 504c, and 504d for comparison in the recognition method according to the comparative example. Figure 3 The upper right portion of shows the registration distance image 602a and the registration direct reflected light image 702a in the recognition method according to this embodiment. Figure 3The lower right part of shows the comparison distance images 604b, 604c, 604d and the comparison direct reflected light images 704b, 704c, 704d in the recognition method according to this embodiment. Figure 3 The images shown are all of the same person’s face. Figure 3 As shown in the center of , it is assumed that the registration facial image 502a, the registration distance image 602a, and the registration direct reflected light image 702a are images captured under lighting condition A. Furthermore, it is assumed that the control facial image 504, the control distance image 604, and the control direct reflected light image 704 are images captured under lighting conditions B, C, and D that are different from the lighting condition A and are different from each other.

[0112] like Figure 3 As shown on the left side of , it can be seen that: in the comparative example, if the lighting conditions change, the comparison facial images 504b, 504c, and 504d become images different from the registration facial image 502a. Specifically, in the comparison facial image 504b, half of the face is clear and the remaining half is unclear. The comparison facial image 504c is an image in which the entire face is unclear, that is, it is even difficult to recognize that it is a facial image. Therefore, in the comparative example, if the lighting conditions, that is, the ambient light, change, the state of the image changes. Therefore, even if it is a facial image of the same person, since the comparison facial image 504 is an image that is greatly different from the registration facial image 502a, it is difficult to recognize that it is the same person. This is because: in the comparative example, when taking an image of the subject, not only the light directly reflected by the subject is detected, but also the indirect light (indirect reflected light) generated by lighting, etc., that is, the ambient light, is detected at the same time, so the image cannot avoid the influence of the ambient light.

[0113] On the other hand, in an embodiment of the present invention, Figure 3 As shown on the right side of , even if the lighting conditions change, the comparison distance images 604b, 604c, 604d and the comparison direct reflected light images 704b, 704c, 704d become images that are substantially the same (substantially equivalent) as the registration distance image 602a and the registration direct reflected light image 702a. That is, in this embodiment, even if the lighting conditions, i.e., the ambient light, change, the state of the image does not change. Therefore, in this embodiment, even if the lighting conditions, i.e., the ambient light, change, if it is the same person, the state of the image changes less, so it can be recognized that it is the same person.

[0114] That is, according to the embodiment of the present invention, it is not affected by the change of the ambient light and can be recognized with high precision. The details of such an embodiment of the present invention are described in sequence below. In addition, in the embodiment of the present invention described below, it is assumed that both the distance information and the direct reflected light information (specifically, the distance image 600 and the direct reflected light image 700) are used for recognition. However, in this embodiment, it is not limited to using both the distance information and the direct reflected light information, and it is also possible to use at least one of them for recognition.

[0115] <<3. First Implementation Method>>

[0116] <3.1 Overview of Recognition System 10 According to First Embodiment>

[0117] First, refer to Figure 4 An overview of a recognition system (recognition device) 10 according to a first embodiment of the present invention will be described. Figure 4 1 is a block diagram showing an example of the structure of the recognition system 10 according to the present embodiment. Figure 4 As shown, the recognition system 10 according to the present embodiment mainly includes a TOF sensor 100, a processing unit 200, a storage unit 300, and a display unit 400. Hereinafter, an overview of each device included in the recognition system 10 according to the present embodiment will be described.

[0118] (TOF sensor 100)

[0119] The TOF sensor 100 obtains distance information and direct reflected light information (specifically, Figure 2 The TOF sensor 100 outputs the sensing data obtained by irradiating a subject (object) with infrared light or other irradiating light and detecting direct reflected light reflected by the subject to the processing unit 200 described later. The processing unit 200 can obtain the distance information (depth) of the subject by calculating the phase difference between the irradiating light and the reflected light based on the sensing data. Furthermore, the processing unit 200 can obtain the direct reflected light information of the subject by processing the sensing data. In addition, the method of obtaining distance information by phase difference as described above is called an indirect TOF method. The details of the TOF sensor 100 will be described later.

[0120] In addition, in the present embodiment, since one TOF sensor 100 is used instead of a plurality of cameras, it is possible to avoid increasing the size or complexity of the structure of the recognition system 10 and, further, to avoid increasing the manufacturing cost of the recognition system 10 .

[0121] (Processing unit 200)

[0122] The processing unit 200 is mainly composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and can store a registration image (specifically, a registration distance image 602, a registration direct reflected light image 702, etc.) in a storage unit 300 described later, or perform human recognition using the registration image stored in the storage unit 300. The details of the processing unit 200 will be described later.

[0123] (Storage Unit 300)

[0124] The storage unit 300 is implemented by a ROM and a RAM, and as described above, stores the above-mentioned registration image used when performing recognition.

[0125] (Display unit 400)

[0126] The display unit 400 is a functional unit that outputs recognition results, etc. to the user, and is implemented by, for example, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, etc. For example, when the facial image of a newly captured person is consistent with the above-mentioned registration image stored in the above-mentioned storage unit 300, the display unit 400 displays information such as the name of the person associated with the above-mentioned registration image. On the other hand, when the facial image of a newly captured person is inconsistent with the above-mentioned registration image, the meaning of inconsistency is displayed.

[0127] In addition, in the present embodiment, part or all of the TOF sensor 100, the processing unit 200, the storage unit 300, and the display unit 400 may be integrated. For example, when the TOF sensor 100, the processing unit 200, the storage unit 300, and the display unit 400 are integrated, it is possible to form a device that performs the processing involved in the recognition as a single machine. In addition, the processing unit 200 described above may also be constructed by a system composed of multiple devices that is connected to a network as a prerequisite, such as cloud computing.

[0128] <3.2 Detailed Structure of TOF Sensor 100>

[0129] The above is a description of the outline of the recognition system 10 according to the present embodiment. Figure 5 The detailed structure of the TOF sensor 100 according to the present embodiment will be described. Figure 5 1 is a block diagram showing a configuration example of the TOF sensor 100 according to the present embodiment. Figure 5 As shown in FIG. 1 , the TOF sensor 100 mainly includes an irradiation unit 102 and a light receiving unit 104. The following describes in detail each functional unit of the TOF sensor 100.

[0130] (Irradiation part 102)

[0131] The irradiation unit 102 includes a laser light source (not shown) and an optical element (not shown). As the laser light source, for example, a laser diode can be used, and the wavelength of the irradiated light can be changed by appropriately selecting the laser diode. In addition, in the present embodiment, the irradiation unit 102 is described as irradiating infrared light having a wavelength of about 785 nm, but in the present embodiment, it is not limited to irradiating such infrared light.

[0132] (Light receiving unit 104)

[0133] The light receiving unit 104 includes a focusing lens (not shown) and a light receiving element (not shown). The focusing lens has the function of focusing the received light to the light receiving element. The light receiving element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor having a plurality of pixels, and generates a light receiving signal for each pixel based on the intensity of the received light, and outputs the generated light receiving signal to the processing unit 200.

[0134] In addition, the irradiation unit 102 and the light receiving unit 104 of the TOF sensor 100 may be controlled by the processing unit 200 , or may be controlled by a control unit (not shown) provided in the TOF sensor 100 .

[0135] Here, refer to Figure 6 The principle of the method of calculating distance information executed by the TOF sensor 100 will be described. Figure 6 This is an explanatory diagram for explaining the principle of a method for calculating distance information, and more specifically, schematically shows temporal changes in the intensity of irradiated light and reflected light in the TOF sensor 100 .

[0136] like Figure 6 As shown in FIG. 1 , the TOF sensor 100 irradiates the subject with light modulated so that the light intensity varies periodically from the irradiation unit 102. The irradiated light is reflected by the subject and detected as reflected light by the light receiving unit 104 of the TOF sensor 100. Figure 6 As shown, the detected reflected light ( Figure 6 The lower part) relative to the irradiation light ( Figure 6The TOF sensor 100 has a phase difference (in the upper part), and the phase difference becomes larger as the distance from the TOF sensor 100 to the subject is longer, and becomes smaller as the distance from the TOF sensor 100 to the subject is shorter.

[0137] Therefore, for example, the TOF sensor 100 senses the intensity of light at four phases (0 degrees, 90 degrees, 180 degrees, and 270 degrees) of the reflected light. 90 ,q 180 ,q 270 ) is input into the following formula (1) to calculate the phase difference (phase). In addition, by using the phase difference calculated in this way and the wavelength (range) of the light, the distance information (distance) to the subject can be obtained according to the following formula (1).

[0138]

Number 1

[0139] I=q0-q 180

[0140] Q=q g0 -q 270

[0141]

[0142]

[0143] Furthermore, since the distance information can be acquired for each pixel of the light receiving unit 104 , the 2.5-dimensional information described above can be acquired by associating the position information and distance information of the corresponding pixel.

[0144] In addition, the light receiving unit 104 according to this embodiment is Figure 5 The first and second light receiving parts 104a and 104b are shown as having a differential operation. Specifically, the first and second light receiving parts 104a and 104b involved in this embodiment are manufactured at the same time, and thus have substantially the same (substantially the same) characteristics. In addition, the first and second light receiving parts 104a and 104b operate during a period of the same length, but are assumed to operate in a manner having a phase difference of 180 degrees (see Figure 7 ). In addition, the TOF sensor 100 having such two light receiving parts is called a double-tap TOF sensor.

[0145] Below, refer to Figure 7 A method for calculating distance information in the double-click type TOF sensor 100 according to the present embodiment will be described. Figure 7 1 is an explanatory diagram for explaining a method of calculating distance information using the TOF sensor 100 according to the present embodiment. Figure 7 In the case of irradiation light ( Figure 7 1st level) and reflected light ( Figure 7 In order to facilitate understanding, the second level of the pulsed light is shown, and it is assumed that the phase difference between the incident light and the reflected light is φ. Figure 7 , the first light receiving unit 104a ( Figure 7 ) and the second light receiving unit 104b ( Figure 7 4) and assume that the light receiving parts 104a and 104b are in motion during the upward projection. Figure 7 As shown, the periods in which the first and second light receiving units 104a and 104b respectively operate do not overlap, and therefore it can be seen that the first and second light receiving units 104a and 104b operate differentially with each other.

[0146] like Figure 7 As shown in FIG. 7 , when the reflected light has a phase difference φ with respect to the irradiated light, the first light receiving unit 104a and the second light receiving unit 104b can detect the reflected light in the gray areas 800a and 800b shown in FIG. 7 . Specifically, by integrating the intensities of the light detected by the first and second light receiving units 104a and 104b, the intensity of the light detected by the first and second light receiving units 104a and 104b can be obtained. Figure 7 The area of ​​the region 800a and the area of ​​the region 800b are equivalent to the light receiving signal. Figure 7 It can be seen that the difference between the integral value in the first light receiving unit 104a and the integral value in the second light receiving unit 104b changes according to the phase difference φ of the reflected light. Therefore, in this embodiment, the difference between the integral values ​​of the first and second light receiving units 104a and 104b can be calculated, the phase difference φ is calculated based on the calculated difference, and the distance information is further calculated. In addition, in this embodiment, instead of calculating the difference in the integral value, the phase difference φ can be calculated using the ratio of the integral values, and the distance information can be calculated.

[0147] In fact, the first and second light receiving units 104a and 104b detect not only the reflected light directly reflected from the subject (direct reflected light), but also the indirect reflected light (ambient light) generated by lighting, etc. Figure 8 The light shown in the upper part, Figure 8 It is an explanatory diagram schematically showing elimination of ambient light (indirect reflected light) in this embodiment.

[0148] Therefore, if Figure 8As shown in the upper part of , when the above-mentioned indirect reflected light (ambient light) is regarded as light whose intensity does not change periodically during a specified period, unlike the direct reflected light, the first light receiving unit 104a and the second light receiving unit 104b both detect the indirect reflected light of the same intensity. Therefore, by calculating the difference between the integral value of the first light receiving unit 104a and the integral value of the second light receiving unit 104b, the integral component caused by the intensity of the indirect reflected light that is common to each other can be eliminated, so that only the integral component such as Figure 8 That is, in this embodiment, by calculating the difference between the integral value of the first light receiving unit 104a and the integral value of the second light receiving unit 104b, the influence of the indirect reflected light (ambient light) can be eliminated. Furthermore, according to this embodiment, by using the direct reflected light information after eliminating the indirect reflected light, a direct reflected light image 700 (see Figure 2 ). In addition, in the present embodiment, it is not limited to extracting the direct reflected light from which the influence of the indirect reflected light is eliminated by calculating the difference between the integral value of the first light receiving unit 104a and the integral value of the second light receiving unit 104b as described above. For example, in the present embodiment, the direct reflected light from which the influence of the indirect reflected light is eliminated may be extracted by using the ratio of the integral value of the first light receiving unit 104a to the integral value of the second light receiving unit 104b.

[0149] In addition, the above distance information is not easily affected by changes in ambient light, and therefore is not limited to calculation using the difference in integral values ​​as previously described. However, by using the difference in integral values ​​for calculation, the common noise signal inherent to the first and second light receiving units 104a and 104b can be removed from the distance information, so the distance information is preferably calculated using the difference in integral values.

[0150] Furthermore, in the present embodiment, the double-click TOF sensor 100 is not limited to a TOF sensor having two light receiving parts 104a and 104b. For example, the double-click TOF sensor 100 involved in the present embodiment may also be a sensor having one light receiving part 104 and two reading parts (a first reading part and a second reading part) (not shown), and the two reading parts read the light received by the one light receiving part 104 at different times. Even such a TOF sensor 100 having one light receiving part and two reading parts can obtain direct reflected light that eliminates the influence of indirect reflected light or a distance signal that removes noise signals, as described above.

[0151] <3.3 Detailed Configuration of Processing Unit 200>

[0152] The detailed structure of the TOF sensor 100 according to the present embodiment has been described above. Figure 4 The detailed structure of the processing unit 200 involved in this embodiment is described. Figure 4 As shown, the processing unit 200 mainly includes a distance information calculation unit 202, a direct reflected light calculation unit (direct reflected light information calculation unit) 204, a subject detection unit (object detection unit) 206, a 3D transformation unit (3D coordinate calculation unit) 208, a subject normalization unit (normalization processing unit) 210, and a subject recognition unit (object recognition unit) 212. The following describes the details of each functional unit of the processing unit 200.

[0153] (Distance Information Calculation Unit 202)

[0154] As described above, the distance information calculation unit 202 calculates the phase difference between the irradiated light and the reflected light based on the sensing data from the TOF sensor 100, and calculates the distance information of the subject (distance image 600) based on the phase difference. The distance information calculated by the distance information calculation unit 202 is information associated with the position information of the pixel of the light receiving unit 104 of the TOF sensor 100, so it can be said to be the above-mentioned 2.5-dimensional information. In addition, the distance information calculation unit 202 can output the calculated distance information to the subject detection unit 206, the 3D transformation unit 208, the subject normalization unit 210, etc. described later.

[0155] (Direct Reflected Light Calculation Unit 204)

[0156] The direct reflected light calculation unit 204 calculates the direct reflected light information of the subject (direct reflected light image 700) by eliminating the indirect reflected light (ambient light) using the above-mentioned method for the sensing data from the TOF sensor 100. In addition, the direct reflected light calculation unit 204 can output the calculated direct reflected light information to the subject detection unit 206, the subject normalization unit 210, etc. described later.

[0157] (Subject Detection Unit 206)

[0158] The subject detection unit 206 detects the area occupied by the subject (subject area) in the distance image 600 based on the distance information obtained by the distance information calculation unit 202 or the direct reflection light image 700 based on the direct reflection light information obtained by the direct reflection light calculation unit 204. When the subject detection unit 206 detects the area of ​​the face of a person as the subject from the distance image 600, for example, it can detect the area of ​​the face based on a predetermined contour line (contour of the face). In addition, when the subject detection unit 206 detects the area of ​​the face of a person from the direct reflection light image 700, for example, it can detect the area of ​​the face based on the positional relationship of predetermined feature points (eyes, nose, mouth), etc. In addition, when the subject is close to the TOF sensor 100, the subject area detected by the subject detection unit 206 is wide, and when the subject is far from the TOF sensor 100, the detected subject area is narrow. Furthermore, the subject detection unit 206 can output the result of the detected subject region to the three-dimensional transformation unit 208 , the subject normalization unit 210 , and the like, which will be described later.

[0159] Furthermore, the subject detection unit 206 can detect the subject region in either the distance image 600 or the direct reflected light image 700, and directly apply the detected subject region to the subject region of the other image. Specifically, the distance image 600 and the direct reflected light image 700 are images obtained from the same shooting from the TOF sensor 100, so there is no need to align the two, so the subject region of one image can be directly used as the subject region of the other.

[0160] In addition, the subject detection unit 206 may determine the final subject region using both the detection result of the subject region in the distance image 600 and the detection result of the subject region in the direct reflected light image 700. Specifically, in the direct reflected light image 700, for example, when the contrast between the face of a person and the background is small, it is difficult to detect the outline of the face of the person. On the other hand, in the distance image 600, even when the contrast is small as described above, it is easy to detect the outline of the face because the positions of the face of the person and the background are different. However, in the distance image 600, it is difficult to detect feature points with few bumps such as the eyes, nose, and mouth of the face. Therefore, by using both the distance image 600 and the direct reflected light image 700 to detect the subject region, the subject detection unit 206 can make the weaknesses of the detection of both complement each other, thereby further improving the accuracy of the detection of the subject region.

[0161] (3D Transformation Unit 208)

[0162] The 3D transformation unit 208 transforms the 2.5-dimensional information, i.e., the distance information, calculated by the distance information calculation unit 202 into coordinate values ​​of a 3D space of the actual space. In detail, the 3D transformation unit 208 transforms the above distance information into coordinates (X, Y, Z) of a 3D space by transforming the position information of the pixels of the light receiving unit 104 of the TOF sensor 100 into coordinate values ​​of the actual space, thereby generating 3D coordinate information (3D information). By performing such a transformation, the distance information can be processed as an actual distance in space. In addition, the 3D transformation unit 208 can output the above 3D coordinate information to the subject normalization unit 210 described later, etc. In addition, the 3D transformation unit 208 may not be provided in the processing unit 200.

[0163] (Subject Normalization Unit 210)

[0164] The subject normalization unit 210 normalizes the subject region in the distance image 600, the direct reflected light image 700, and the three-dimensional information (not shown). Examples of the normalization performed in the subject normalization unit 210 include normalization of direction, normalization of scale, normalization of brightness, etc. Furthermore, the subject normalization unit 210 outputs the normalized subject region to the subject recognition unit 212 or the storage unit 300 described later. As described above, the three-dimensional information refers to a collection of three-dimensional coordinate information (X, Y, Z coordinate information).

[0165] Reference Fig. 9 An example of normalization in the subject normalization unit 210 will be described. Fig. 9 1 is an explanatory diagram for explaining an example of normalization in the present embodiment, and more specifically, shows an example of normalization of the direction of a person's face. Fig. 9 The left side of the figure shows the subject area 710 of the direct reflected light image before normalization. Fig. 9 The right side of represents the subject area 712 of the normalized direct reflected light image. In the subject area 710 of the direct reflected light image before normalization, the face of the person is not facing the front but facing diagonally. Therefore, the subject normalization unit 210 detects the fact that the face of the person is facing diagonally based on the positional relationship or shape of the specified feature points (eyes, nose, mouth) in the subject area 710 of the direct reflected light image before normalization, and normalizes the subject area 710 of the direct reflected light image before normalization so that the face faces the front. As a result, the subject normalization unit 210 can obtain Fig. 9 The right side of FIG. 7 shows the normalized direct reflected light image of the subject region 712 .

[0166] In addition, in this embodiment, the subject normalization unit 210 is not limited to Fig. 9The normalization may be performed in a manner that is oriented toward the front direction as shown in the figure, or may be performed in a manner that is consistent with a predetermined direction (orientation), for example, the direction of an image stored in the storage unit 300, without particular limitation. For example, by normalizing in a manner that is consistent with the direction of an image stored in the storage unit 300, the direction of the image stored in the storage unit 300 is the same as the direction of the image to be compared, so that the accuracy of the comparison in the subject recognition unit 212 described later can be further improved.

[0167] Furthermore, in this embodiment, the subject normalization unit 210 is not limited to Fig. 9 The subject region 710 of the direct reflected light image is normalized as shown in the figure, and the subject region (not shown) of the distance image 600 or the three-dimensional information (not shown) can also be normalized. In this case, the subject normalization unit 210 can, for example, perform normalization based on the shape of a predetermined contour line (face contour) in the subject region of the distance image 600 so that the face of the person faces forward. That is, the subject normalization unit 210 can normalize at least one of the distance image 600, the direct reflected light image 700, and the subject region in the three-dimensional information.

[0168] However, the subject normalization unit 210 preferably normalizes two subject areas, namely, the subject area of ​​the distance image 600 or the three-dimensional information (not shown) and the subject area of ​​the direct reflected light image 700, so as to further improve the accuracy of normalization. In detail, the subject normalization unit 210 preferably uses both the detection result of the orientation of the face in the subject area of ​​the distance image 600 or the three-dimensional information and the detection result of the orientation of the face of the person in the subject area of ​​the direct reflected light image 700 to determine the parameters for normalization. As previously described, the distance image 600 or the three-dimensional information and the direct reflected light image 700 have different detection easiness or difficulty, respectively. Therefore, by using the detection results of both parties to determine the orientation of the face, the subject normalization unit 210 can make the weaknesses of the detection of both parties complement each other, thereby improving the accuracy of the detection of the orientation of the face.

[0169] In addition, in the present embodiment, the subject normalization unit 210 may also perform normalization in such a manner that the size (scale) of the subject region in the distance image 600, the direct reflected light image 700, and the three-dimensional information (not shown) is consistent with the prescribed size (scale). As previously described, when the distance from the TOF sensor 100 to the subject is long, the subject region is narrow, and when the distance from the TOF sensor 100 to the subject is short, the subject region is wide. Therefore, for example, the subject normalization unit 210 performs normalization by enlarging or reducing the size of the subject region in the distance image 600, the direct reflected light image 700, and the three-dimensional information so that they become the same size as the image stored in the storage unit 300. In other words, the subject normalization unit 210 can normalize the subject region by using the image stored in the storage unit 300 as a reference so as to obtain an image captured at the same distance (predetermined distance) as the distance between the subject and the TOF sensor 100 when the reference image was captured. By doing so, the accuracy (resolution) of the image stored in the storage unit 300 is the same as the accuracy of the image to be compared, so that the accuracy of the comparison in the subject recognition unit 212 described later can be further improved.

[0170] Furthermore, in the present embodiment, the subject normalization unit 210 may also normalize the subject region in the distance image 600, the direct reflected light image 700, and the three-dimensional information (not shown) with the image stored in the storage unit 300 as a reference so as to have the same brightness (brightness) or contrast as the reference image. By doing so, the brightness of the image or information stored in the storage unit 300 and the brightness of the image or information to be compared are the same, so that the accuracy of the comparison in the subject recognition unit 212 described later can be further improved.

[0171] As previously described, the subject normalization unit 210 may normalize at least one of the subject areas of the distance image 600, the direct reflected light image 700, and the three-dimensional information (not shown), or may normalize a part or all of them, and there is no particular limitation in the present embodiment. In addition, the subject normalization unit 210 may directly apply the normalization parameters (e.g., the parameters used when normalizing the direction) used in one of the subject areas of the distance image 600, the direct reflected light image 700, and the three-dimensional information to the normalization of the subject areas of other images. The distance image 600, the direct reflected light image 700, and the three-dimensional information are information obtained by the same shooting from the TOF sensor 100, so there is no need to align the two, and the above parameters can be directly applied.

[0172] (Subject Recognition Unit 212)

[0173] The subject recognition unit 212 reads the image stored in the storage unit 300 at the recognition stage described later, and compares the read image with the distance image 600, the direct reflected light image 700, and the subject area of ​​the three-dimensional information (not shown) obtained from the subject normalization unit 210. In detail, the subject recognition unit 212 can compare by comparing the feature points of each image or their positional relationship. At this time, if it is the subject area of ​​the distance image 600, the subject recognition unit 212 reads the distance image 600 from the storage unit 300 for comparison, and the subject recognition unit 212 uses the same type of image for comparison. Furthermore, when the read image and the subject area are consistent, the subject recognition unit 212 outputs information indicating consistency and information (name, etc.) stored in association with the read image to the display unit 400, etc. In addition, when the read image and the subject area are inconsistent, the subject recognition unit 212 outputs information indicating inconsistency to the display unit 400, etc. In addition, the subject recognition unit 212 may output the recognition result obtained by the above-mentioned comparison to a functional unit (not shown) other than the display unit 400, and the functional unit may further control other functional units based on the recognition result.

[0174] <3.4 Identification method>

[0175] The above is a description of the details of each device included in the recognition system 10 according to the present embodiment. Next, the recognition method according to the present embodiment is described. The recognition method according to the present embodiment is mainly divided into the following two stages: a registration stage until the image is stored in the storage unit 300; and a recognition stage for performing recognition using the image stored in the storage unit 300. In addition, in the recognition method described below, the processing unit 200 may not include the above-mentioned 3D transformation unit 208.

[0176] <3.4.1 Registration Phase>

[0177] First, refer to Fig.10 The registration phase of the identification method according to the present embodiment will be described. Fig.10 FIG. 1 is a flow chart illustrating the registration phase of the identification method according to the present embodiment. Fig.10 As shown, the registration phase according to the present embodiment includes a plurality of steps from step S101 to step S113. The following describes in detail each step included in the registration phase according to the present embodiment.

[0178] (Step S101)

[0179] First, the TOF sensor 100 irradiates light onto a subject (eg, a person's face), and outputs sensing data obtained by detecting reflected light to the processing unit 200 (sensing performed by the TOF sensor 100 ).

[0180] (Step S103)

[0181] The processing unit 200 performs processing to eliminate ambient light (indirect reflected light) from the sensing data as described above.

[0182] (Step S105)

[0183] The processing unit 200 calculates the distance information (distance image 600 ) of the object based on the sensing data from which the ambient light is eliminated in the above-mentioned step S103 .

[0184] (Step S107)

[0185] The processing unit 200 calculates the direct reflected light information (direct reflected light image 700 ) of the object based on the sensing data from which the ambient light has been eliminated in step S103 described above.

[0186] (Step S109)

[0187] The processing unit 200 detects a subject region (not shown) from the distance image 600 obtained in step S105. In addition, the processing unit 200 detects a subject region 710 from the direct reflected light image 700 obtained in step S107. Fig.10 In FIG. 6 , the detection of the subject region from the distance image 600 is omitted from illustration.

[0188] (Step S111)

[0189] The processing unit 200 normalizes the distance image 600 and the subject area 710 of the direct reflected light image 700 obtained in the above step S109, and obtains the normalized subject area 612 of the distance image and the normalized subject area 712 of the direct reflected light image. In addition, the normalization performed here can be normalization of direction, normalization of scale, normalization of brightness, etc., and is not particularly limited.

[0190] (Step S113)

[0191] The processing unit 200 stores the normalized subject region 612 and the subject region 712 obtained in the above-mentioned step S111 in the storage unit 300 .

[0192] <3.4.2 Identification Phase>

[0193] Below, refer to Fig.11The recognition phase of the recognition method according to the present embodiment will be described. Fig.11 is a flow chart illustrating the recognition phase of the recognition method according to the present embodiment. Fig.11 As shown, the recognition stage involved in this embodiment includes multiple steps from step S201 to step S213. In addition, Fig.11 From step S201 to step S211, Fig.10 The steps from step S101 to step S111 are the same, therefore, detailed descriptions of these steps are omitted here, and only step S213 is described. Fig.11 In the figure, the detection of the subject area from the distance image 600 is also omitted.

[0194] (Step S213)

[0195] The processing unit 200 reads an image stored in the storage unit 300 (for example, a facial image of a person stored in the above-mentioned registration stage), and compares the read image with the normalized subject area 612 and the subject area 712 (here, the facial image of a person). Furthermore, when the read image and the subject areas 612 and 712 are consistent, the processing unit 200 outputs information indicating the consistency and information (such as the name of a person) stored in association with the read image to the user. Furthermore, when the read image and the subject areas are inconsistent, the processing unit 200 outputs information indicating the inconsistency to the user.

[0196] As described above, in this embodiment, recognition is performed using the distance image 600 or the direct reflected light image 700 which is not easily affected by changes in ambient light, rather than a two-dimensional image which is easily affected by changes in ambient light. Therefore, recognition can be performed stably with high accuracy.

[0197] <4. Second Embodiment>>

[0198] In the first embodiment described above, 2.5-dimensional information, namely, the distance image 600 is used, but in the second embodiment described below, the 2.5-dimensional information is converted into 3-dimensional information (3-dimensional coordinate information) for use, which is different from the first embodiment described above. The details of such a second embodiment will be described below.

[0199] In addition, in the following description, only the points different from the first embodiment are described, and the description of the points common to the first embodiment is omitted. Specifically, in the present embodiment, the detailed configuration of the recognition system 10 or the device included in the recognition system 10 is the same as that of the first embodiment except that the processing unit 200 includes the three-dimensional transformation unit 208. Therefore, the description of the detailed configuration of the recognition system 10 or the device included in the recognition system 10 according to the present embodiment is omitted here.

[0200] <4.1 Identification method>

[0201] The recognition method according to the present embodiment is described below. The recognition method according to the present embodiment is also mainly divided into the following two stages, similarly to the first embodiment: a registration stage until the image is stored in the storage unit 300; and a recognition stage for performing recognition using the image stored in the storage unit 300.

[0202] <4.1.1 Registration Phase>

[0203] First, refer to Fig.12 The registration phase of the identification method according to the present embodiment will be described. Fig.12 FIG. 1 is a flow chart illustrating the registration phase of the identification method according to the present embodiment. Fig.12 As shown, the registration stage involved in this embodiment includes multiple steps from step S301 to step S315. In addition, Fig.12 From step S301 to step S309, Fig.10 The steps from step S101 to step S109 are the same as those of the first embodiment, and therefore, detailed descriptions of these steps are omitted here. Fig.12 In FIG. 6 , the detection of the subject region from the distance image 600 is also omitted.

[0204] (Step S311)

[0205] The processing unit 200 converts the subject region (not shown) of the distance image 600 obtained in step S309 into three-dimensional coordinate information, and obtains the subject region 620 of the three-dimensional information of the subject.

[0206] (Step S313)

[0207] The processing unit 200 and Fig.10Similarly to step S111 of the first embodiment, the subject region 620 of the three-dimensional information obtained in the above-mentioned step S311 and the subject region 710 of the direct reflected light image 700 obtained in step S309 are normalized. Then, the processing unit 200 obtains the subject region 622 of the normalized three-dimensional information and the subject region 712 of the normalized direct reflected light image.

[0208] (Step S315)

[0209] The processing unit 200 stores the normalized subject region 622 and the subject region 712 obtained in the above-mentioned step S313 in the storage unit 300 .

[0210] <4.1.2 Identification Phase>

[0211] Below, refer to Fig.13 The recognition phase of the recognition method according to the present embodiment will be described. Fig.13 is a flow chart illustrating the recognition phase of the recognition method according to the present embodiment. Fig.13 As shown, the registration stage involved in this embodiment includes multiple steps from step S401 to step S415. In addition, Fig.13 The steps from step S401 to step S413 are the same as those described above. Fig.12 The same as from step S301 to step S313, Fig.13 Step S415 and Fig.11 The steps are similar to step S213 of the first embodiment, and therefore, detailed description of these steps is omitted here. Fig.13 In FIG. 6 , the detection of the subject region from the distance image 600 is also omitted.

[0212] In this embodiment, as in the first embodiment, recognition is performed using three-dimensional information (not shown) or a direct reflected light image 700 that is not easily affected by changes in ambient light instead of a two-dimensional image that is easily affected by changes in ambient light. This enables stable recognition with high accuracy.

[0213] <<5. Third Implementation Method>>

[0214] In the first and second embodiments described above, the recognition system 10 is described as being composed of a plurality of devices, but at least a portion of the recognition system 10 may be constructed by a stacked image sensor. Fig.14 and Fig.15 A third embodiment in which the TOF sensor 100 and the processing unit 200 are configured using a stacked image sensor will be described. Fig.141 is a diagram showing a configuration example of a stacked image sensor 20 according to the present embodiment. Fig.15 It is a block diagram showing a detailed configuration example of the stacked image sensor 20 according to the present embodiment.

[0215] like Fig.14 As shown in FIG. 1 , in the stacked image sensor 20, the pixel region 120 and the signal processing circuit region 220 are stacked on each other. Fig.14 In FIG. 1 , the pixel region 120 is shown to be stacked above the signal processing circuit region 220. However, in the present embodiment, the pixel region 120 is not limited to the signal processing circuit region 220. Fig.14 The layers can be stacked in the order described above or in the reverse order.

[0216] In addition, if Fig.15 As shown, a distance information calculation unit 202, a direct reflected light calculation unit 204, a subject detection unit 206, a subject normalization unit 210, and a subject recognition unit 212 are provided in the signal processing circuit area 220. That is, in the stacked image sensor 20, the pixel area 120 functions as the above-mentioned TOF sensor 100, and outputs the sensing data to the signal processing circuit area 220. Furthermore, the signal processing circuit area 220 can function as the above-mentioned processing unit 200, and output the image obtained by processing the sensing data to the storage unit 300, or use the image stored in the storage unit 300 for recognition. In addition, in Fig.15 In the signal processing circuit area 220, the above-mentioned three-dimensional conversion unit 208 may be provided as needed, and other functional units not shown in the figure may also be provided. That is, in this embodiment, the recognition system 10 can be composed of a stacked image sensor 20 and a storage unit 300 provided outside the stacked image sensor 20.

[0217] Furthermore, the stacked image sensor may be a three-layer stacked image sensor. Fig.16 as well as Fig.17 An example of a three-layer stacked image sensor 20a will be described. Fig.16 1 is a diagram showing a configuration example of a stacked image sensor 20a according to a modification of the present embodiment. Fig.17 It is a block diagram showing a detailed configuration example of a stacked image sensor 20 a according to a modification of the present embodiment.

[0218] like Fig.16 As shown in FIG. 1 , in the three-layer stacked image sensor 20 a, in addition to the pixel region 120 and the signal processing circuit region 220, a memory region 320 is stacked. Fig.16, the pixel region 120 is stacked on the memory region 320, and the memory region 320 is stacked on the signal processing circuit region 220. However, in this embodiment, it is not limited to Fig.16 The layers may be stacked in that order or in a different order.

[0219] In this variation, if Fig.17 As shown, a storage unit 300 is provided in the memory area 320, and the storage unit 300 stores an image used for recognition. That is, in this modification, the recognition system 10 can be composed of one stacked image sensor 20a, in other words, the stacked image sensor 20a can be used for recognition. In addition, in this modification, the above-mentioned three-dimensional conversion unit 208 and the like can also be provided in the signal processing circuit area 220 as needed.

[0220] As described above, according to the present embodiment and the present variation, by configuring at least a portion of the above-mentioned recognition system 10 using the stacked image sensor 20, the recognition system 10 can be made more compact and power consumption can be suppressed. Furthermore, as a result, the recognition system 10 involved in the embodiment of the present invention can be mounted on various electronic devices described later.

[0221] <<6. Fourth Implementation Method>>

[0222] As described above, the recognition system 10 (stacked image sensor 20) according to the first to third embodiments can be mounted on electronic devices such as desktop PCs (Personal Computers), notebook PCs, portable PCs, smartphones, mobile phones, cameras, and wearable devices.

[0223] For example, in order to authenticate users of electronic devices or perform electronic settlement, the recognition system 10 of this embodiment can be installed on various PCs, smart phones, mobile phones, etc. as a facial recognition device. In addition, as a device for detecting suspicious persons, the recognition system 10 can be installed on monitoring cameras or monitoring systems. In addition, as an inspection device for inspecting whether a product is properly manufactured by image recognition based on its shape, etc. in various factories, the recognition system 10 of this embodiment can be installed on inspection systems, etc.

[0224] Furthermore, in order to correctly identify the space around the user, the recognition system 10 may be mounted in an HMD (Head Mounted Display) or a glasses-type wearable device as a device for identifying the surrounding space. In addition, the recognition system 10 according to the present embodiment may be mounted on a robot capable of autonomous walking, etc., in order to identify surrounding obstacles. Furthermore, by mounting the recognition system 10 in a camera having an autofocus function, the detection accuracy of the subject of the camera can be improved, thereby enabling the focus of the camera to be controlled with higher accuracy.

[0225] Therefore, as a fourth embodiment of the present invention, refer to Fig.18 A configuration example of an electronic device 900 equipped with the above-described recognition system 10 will be described. Fig.18 This is a block diagram showing an example of the hardware configuration of the electronic device 900 according to the present embodiment.

[0226] The electronic device 900 includes, for example, a CPU 950, a ROM 952, a RAM 954, a recording medium 956, an input / output interface 958, and an operation input device 960. In addition, the electronic device 900 includes a display device 962, a communication interface 968, and a TOF sensor 980. In addition, in the electronic device 900, for example, each component is connected using a bus 970 as a data transmission path.

[0227] (CPU950)

[0228] The CPU 950 is composed of one or more processors or various processing circuits formed by a calculation circuit such as a CPU, and functions as a control unit that controls the entire electronic device 900. Specifically, the CPU 950 realizes functions such as the distance information calculation unit 202, the direct reflected light calculation unit 204, the subject detection unit 206, the three-dimensional transformation unit 208, the subject normalization unit 210, and the subject recognition unit 212.

[0229] (ROM952 and RAM954)

[0230] The ROM 952 stores control data such as programs and calculation parameters used by the CPU 950 . The RAM 954 temporarily stores programs executed by the CPU 950 , for example.

[0231] (Recording medium 956)

[0232] The recording medium 956 stores various data such as images used in the recognition method according to the present embodiment. Here, as the recording medium 956 , for example, a nonvolatile memory such as a flash memory can be cited. Furthermore, the recording medium 956 is attachable to and detachable from the electronic device 900 .

[0233] (Input / output interface 958, operation input device 960, and display device 962)

[0234] The input / output interface 958 is connected to, for example, an operation input device 960 or a display device 962. Examples of the input / output interface 958 include a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal, and various processing circuits.

[0235] The operation input device 960 functions as an input unit that receives, for example, an operation of the electronic device 900 by a user, and is connected to the input / output interface 958 inside the electronic device 900 .

[0236] The display device 962 functions as the display unit 400 that outputs the recognition result to the user, is provided on the electronic device 900, and is connected to the input / output interface 958 inside the electronic device 900. As the display device 962, for example, a liquid crystal display or an organic EL display (Organic Electro-Luminescence Display) can be cited.

[0237] In addition, the input / output interface 958 can also be connected to an external device such as an operation input device (such as a keyboard or a mouse, etc.) or an external display device outside the electronic device 900. In addition, the input / output interface 958 can also be connected to a drive (not shown). The drive is a reader / writer for a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, and is built-in or external to the electronic device 900. The drive reads the information recorded in the equipped removable recording medium and outputs it to the RAM 954. In addition, the drive can also write records to the equipped removable recording medium.

[0238] (Communication interface 968)

[0239] The communication interface 968 functions as a communication unit for communicating wirelessly or wired with an external device of the electronic device 900. Here, as the communication interface 968, for example, a communication antenna and an RF (radio frequency) circuit (wireless communication), an IEEE802.15.1 port and a transceiver circuit (wireless communication), an IEEE802.11 port and a transceiver circuit (wireless communication), or a LAN (local area network) terminal and a transceiver circuit (wired communication), etc. can be cited.

[0240] (TOF sensor 980)

[0241] The TOF sensor 980 functions as the TOF sensor 100 described above.

[0242] The above is an example of the hardware configuration of the electronic device 900. In addition, the hardware configuration of the electronic device 900 is not limited to Fig.18 Specifically, each of the above-mentioned components may be formed using general-purpose components or may be formed using specialized hardware to realize the functions of each component. The above-mentioned configuration may be appropriately changed according to the technical level at the time of implementation.

[0243] For example, the electronic device 900 may not have the communication interface 968 when it is configured to perform processing as a single machine. In addition, the communication interface 968 may also have a configuration that can communicate with one or more external devices through multiple communication methods. In addition, the electronic device 900 may also be configured to not have the recording medium 956, the operation input device 960, the display device 962, etc.

[0244] Furthermore, the electronic device 900 according to the present embodiment may also be a system composed of multiple devices, such as cloud computing, which is based on connection to a network (or communication between devices). In such a case, the processing for recognition or the image used for recognition may be performed by a computer (not shown) on the cloud. That is, the electronic device 900 according to the present embodiment described above can be implemented as a processing system that uses multiple devices to perform the processing involved in the recognition method according to the present embodiment.

[0245] <<7. Conclusion>>

[0246] As described above, in each embodiment of the present invention, recognition is performed using the distance image 600, 3D information (not shown), and the direct reflected light image 700, which are not easily affected by changes in ambient light, rather than using a 2D image that is easily affected by changes in ambient light. Therefore, according to these embodiments, even when the ambient light changes, recognition can be performed stably with high accuracy.

[0247] Specifically, according to the present embodiment, it is not necessary to estimate the lighting conditions when the registration facial image 502 is taken and control the lighting so that the lighting conditions when the comparison facial image 504 is taken are the same as in the above-mentioned Patent Document 1. Therefore, according to the present embodiment, compared with the above-mentioned Patent Document 1, it is possible to avoid a longer processing time or an increase in power consumption, and it is also possible to avoid a more complicated configuration of the recognition system or an increase in manufacturing cost.

[0248] Furthermore, according to this embodiment, unlike the method using a stereo camera described above, it is not necessary to install two cameras. Therefore, according to this embodiment, it is possible to avoid an increase in the size of the device or an increase in manufacturing cost.

[0249] Furthermore, according to this embodiment, unlike the structured light method described above, it is not necessary to project light having a predetermined pattern onto the surface of the object. Therefore, according to this embodiment, the image of the object projected with the predetermined pattern is not used for recognition, so the recognition accuracy can be improved.

[0250] Furthermore, according to this embodiment, the subject can be continuously photographed while the camera is moved around the subject, and the distance information of the subject can be calculated even without acquiring a plurality of photographed frames of the subject. Therefore, according to this embodiment, the subject can be identified even when the subject moves or the shape of the subject changes.

[0251] In addition, in each embodiment of the present invention, as described above, at least one of the distance information (distance image 600), the direct reflected light information (direct reflected light image 700), and the three-dimensional coordinate information (three-dimensional image) may be used for recognition. However, in this embodiment, in order to further improve the accuracy of recognition, it is preferred to use two of the distance information (distance image 600) or the three-dimensional coordinate information (three-dimensional image) and the direct reflected light information (direct reflected light image 700) for recognition.

[0252] Furthermore, in the above-mentioned embodiment of the present invention, it is assumed that human face recognition is performed, but in the above-mentioned embodiment, it is not limited to human face recognition, and the shape of a product or the like, or recognition of other objects can also be performed.

[0253] <<8.Supplement>>

[0254] The steps in the identification method involved in the above-mentioned embodiment do not have to be processed in the order described. For example, the steps can be processed in a suitably changed order. Moreover, the steps can also be processed locally in parallel or individually instead of being processed in a time series. In addition, the processing of each step does not have to be processed in the described method, and can be processed in other methods using other functional modules, for example.

[0255] In addition, at least a part of the identification method involved in the above-mentioned embodiment can also be constituted by software as an information processing program that enables a computer to function. In the case of being constituted by software, the program that implements at least a part of these methods can be stored in a recording medium, and read into the processing unit 200 or the electronic device 900, or other devices connected to the processing unit 200 or the electronic device 900 and executed. In addition, the program that implements at least a part of the identification method can also be released via a communication line such as the Internet (including wireless communication). In addition, the program can also be encrypted or modulated, and released in a compressed state via a wired line or wireless line such as the Internet, or stored in a recording medium.

[0256] The preferred embodiments of the present invention are described in detail with reference to the accompanying drawings, but the technical scope of the present invention is not limited to the above examples. It is known that a person with ordinary knowledge in the technical field of the present invention can think of various variations or modifications within the scope of the technical ideas recorded in the technical solution, and it should be understood that these variations or modifications also belong to the technical scope of the present invention.

[0257] Furthermore, the effects described in this specification are only illustrative or exemplary, and not restrictive. That is, the technology involved in the present invention can achieve other effects that can be clearly understood by those skilled in the art based on the description of this specification together with or instead of the above effects.

[0258] In addition, the following configurations also belong to the technical scope of the present invention.

[0259] (1) An identification device, comprising:

[0260] a directly reflected light information calculation unit that calculates directly reflected light information related to directly reflected light from the object based on sensor data obtained by a TOF sensor that irradiates light to the object and detects the light;

[0261] an object detection unit that detects the object based on the direct reflected light information; and

[0262] The object recognition unit recognizes the object based on the directly reflected light information of the detected object.

[0263] (2) The identification device according to (1) above, characterized in that:

[0264] The TOF sensor includes a first light receiving unit and a second light receiving unit which are differential with each other.

[0265] The direct reflected light information calculation unit calculates the direct reflected light information based on the intensity of the light detected by the first light receiving unit and the second light receiving unit.

[0266] (3) The identification device according to (1) above, characterized in that:

[0267] The TOF sensor includes a light receiving unit, and a first reading unit and a second reading unit that read light received by the light receiving unit at different timings.

[0268] The direct reflected light information calculation unit calculates the direct reflected light information based on the intensity of the light read by the first reading unit and the second reading unit.

[0269] (4) The identification device according to (2) or (3) above, characterized in that:

[0270] The directly reflected light information calculation unit,

[0271] based on a difference between an integrated value of the intensity of light detected by the first light receiving unit and an integrated value of the intensity of light detected by the second light receiving unit, or

[0272] Based on the difference between the integrated value of the intensity of the light read by the first reading unit and the integrated value of the intensity of the light read by the second reading unit,

[0273] The direct reflected light information is calculated.

[0274] (5) The recognition device according to any one of (1) to (4) above, further comprising a normalization processing unit that normalizes the detected direct reflected light information of the object.

[0275] (6) The identification device according to (5) above, characterized in that:

[0276] The normalization processing unit,

[0277] so that the distance from the TOF sensor to the object coincides with a predetermined distance, or

[0278] so that the size of the object is consistent with a prescribed size, or

[0279] so that the orientation of the object is consistent with a prescribed orientation, or

[0280] in such a manner that the brightness of the directly reflected light information of the object detected is consistent with a predetermined brightness,

[0281] The detected direct reflected light information of the object is normalized.

[0282] (7) The identification device according to any one of (1) to (6) above, characterized in that:

[0283] further comprising a storage unit for storing direct reflected light information of the object,

[0284] The object recognition unit recognizes the object by comparing pre-stored direct reflected light information of the object with newly calculated direct reflected light information of the object.

[0285] (8) The recognition device according to (7) above, wherein the storage unit stores normalized direct reflected light information of the object.

[0286] (9) The identification device according to any one of (1) to (8) above, characterized in that:

[0287] further comprising a distance information calculation unit for calculating the distance information of the object based on the sensing data,

[0288] The object recognition unit recognizes the object based on the distance information.

[0289] (10) An identification device, comprising:

[0290] a distance information calculation unit that calculates the distance information of the object based on sensing data obtained by a TOF sensor that irradiates light to the object and detects the light;

[0291] an object detection unit that detects the object based on the distance information; and

[0292] The object recognition unit recognizes the object based on the distance information of the detected object.

[0293] (11) The recognition device according to (10) above, wherein the distance information calculation unit calculates the distance information based on a phase difference between the irradiated light and the detected light.

[0294] (12) The identification device according to (10) or (11) above, characterized in that:

[0295] The TOF sensor includes a first light receiving unit and a second light receiving unit which are differential with each other.

[0296] The distance information calculation unit calculates the distance information based on the intensity of the light detected by the first light receiving unit and the second light receiving unit.

[0297] (13) The identification device according to (10) or (11) above, characterized in that:

[0298] The TOF sensor includes a light receiving unit, and a first reading unit and a second reading unit that read light received by the light receiving unit at different timings.

[0299] The distance information calculation unit calculates the distance information based on the intensity of the light read by the first reading unit and the second reading unit.

[0300] (14) The identification device according to (12) or (13) above, characterized in that:

[0301] The distance information calculation unit,

[0302] based on a difference between an integrated value of the intensity of light detected by the first light receiving unit and an integrated value of the intensity of light detected by the second light receiving unit, or

[0303] Based on the difference between the integrated value of the intensity of the light read by the first reading unit and the integrated value of the intensity of the light read by the second reading unit,

[0304] The distance information is calculated.

[0305] (15) The identification device according to any one of (10) to (14) above, characterized in that:

[0306] further comprising a normalization processing unit for normalizing the distance information of the detected object,

[0307] The normalization processing unit,

[0308] so that the distance from the TOF sensor to the object coincides with a predetermined distance, or

[0309] so that the size of the object is consistent with a prescribed size, or

[0310] so that the orientation of the object is consistent with a prescribed orientation, or

[0311] in such a manner that the brightness of the directly reflected light information of the object detected is consistent with a predetermined brightness,

[0312] The detected distance information of the object is normalized.

[0313] (16) The identification device according to any one of (10) to (15) above, characterized in that:

[0314] further comprising a storage unit for storing distance information of the object,

[0315] The object recognition unit recognizes the object by comparing pre-stored distance information of the object with newly calculated distance information of the object.

[0316] (17) The identification device according to (10) above, characterized in that:

[0317] further comprising a 3D coordinate calculation unit, the 3D coordinate calculation unit calculating the 3D coordinate information of the object based on the distance information,

[0318] The object recognition unit recognizes the object based on the three-dimensional coordinate information.

[0319] (18) The identification device according to (17) above, characterized in that:

[0320] further comprising a normalization processing unit, the normalization processing unit normalizing the three-dimensional coordinate information,

[0321] The normalization processing unit,

[0322] so that the distance from the TOF sensor to the object coincides with a predetermined distance, or

[0323] so that the size of the object is consistent with a prescribed size, or

[0324] so that the orientation of the object is consistent with a prescribed orientation, or

[0325] in such a manner that the brightness of the directly reflected light information of the object detected is consistent with a predetermined brightness,

[0326] The 3D coordinate information is normalized.

[0327] (19) The identification device according to (17) or (18) above, characterized in that:

[0328] further comprising a storage unit for storing the three-dimensional coordinate information of the object,

[0329] The object recognition unit recognizes the object by comparing pre-stored three-dimensional coordinate information of the object with newly calculated three-dimensional coordinate information of the object.

[0330] (20) The recognition device according to any one of (1) to (19) above, further comprising the TOF sensor.

[0331] (21) The recognition device according to (20) above is characterized in that a pixel region that functions as the TOF sensor and a signal processing circuit region that functions as the object detection unit and the object recognition unit are stacked on top of each other.

[0332] (22) An electronic device, characterized in that:

[0333] The electronic device is equipped with an identification device, and the identification device has:

[0334] a directly reflected light information calculation unit that calculates directly reflected light information related to directly reflected light from the object based on sensor data obtained by a TOF sensor that irradiates light to the object and detects the light;

[0335] an object detection unit that detects the object based on the direct reflected light information; and

[0336] The object recognition unit recognizes the object based on the directly reflected light information of the detected object.

Claims

1. A recognition device, characterized in that: have: a distance information calculation unit that calculates a phase difference between the irradiated light and the reflected light based on sensing data obtained by a TOF sensor that irradiates light to the object and detects the light, and calculates the distance information of the object based on the phase difference; a direct reflected light information calculation unit that performs processing to eliminate indirect reflected light on the sensing data and calculates direct reflected light information related to the direct reflected light from the object; an object detection unit that detects the object based on the direct reflected light information; an object recognition unit that recognizes the object based on the directly reflected light information of the detected object, and A normalization processing unit normalizes a subject region of the detected direct reflected light information of the object and a subject region of the distance information of the object so that the orientation of the object matches a predetermined orientation.

2. The identification device according to claim 1, characterized in that The TOF sensor includes a first light receiving unit and a second light receiving unit which are differential with each other. The direct reflected light information calculation unit calculates the direct reflected light information based on the intensity of the light detected by the first light receiving unit and the second light receiving unit.

3. The identification device according to claim 1, characterized in that: The TOF sensor includes a light receiving unit, and a first reading unit and a second reading unit that read light received by the light receiving unit at different timings. The direct reflected light information calculation unit calculates the direct reflected light information based on the intensity of the light read by the first reading unit and the second reading unit.

4. The identification device according to claim 2, characterized in that: The direct reflected light information calculation unit calculates the direct reflected light information based on a difference between an integrated value of the intensity of light detected by the first light receiving unit and an integrated value of the intensity of light detected by the second light receiving unit.

5. The identification device according to claim 1, characterized in that: The normalization processing unit, so that the distance from the TOF sensor to the object coincides with a predetermined distance, or so that the size of the object is consistent with a prescribed size, or in such a manner that the brightness of the directly reflected light information of the detected object is consistent with a predetermined brightness, The directly reflected light information of the detected object is normalized.

6. The identification device according to claim 1, characterized in that: further comprising a storage unit for storing direct reflected light information of the object, The object recognition unit recognizes the object by comparing pre-stored direct reflected light information of the object with newly calculated direct reflected light information of the object.

7. The identification device according to claim 6, characterized in that The storage unit stores normalized direct reflected light information of the object.

8. The identification device according to claim 1, characterized in that: The object recognition unit recognizes the object based on the distance information.

9. The identification device according to claim 1, characterized in that: The device further includes the TOF sensor.

10. The identification device according to claim 9, characterized in that: A pixel region functioning as the TOF sensor and a signal processing circuit region functioning as the object detection unit and the object recognition unit are stacked on each other.

11. An electronic device, characterized in that: The electronic device is equipped with an identification device, and the identification device has: a distance information calculation unit that calculates a phase difference between the irradiated light and the reflected light based on sensing data obtained by a TOF sensor that irradiates light to the object and detects the light, and calculates the distance information of the object based on the phase difference; a direct reflected light information calculation unit that performs processing to eliminate indirect reflected light on the sensing data and calculates direct reflected light information related to the direct reflected light from the object; an object detection unit that detects the object based on the direct reflected light information; an object recognition unit that recognizes the object based on the detected direct reflected light information of the object; as well as A normalization processing unit normalizes a subject region of the detected direct reflected light information of the object and a subject region of the distance information of the object so that the orientation of the object matches a predetermined orientation.

Images