Ranging device and ranging method

Abstract

A ranging device (100) comprises: a light source (10) that emits light in a prescribed range; a light reception unit (20) that has a plurality of pixels for receiving reflected light resulting from the emitted light being reflected within the prescribed range; an image recognition unit (30) that acquires an image in which at least a portion of the prescribed range appears, and that detects a preset recognition target object (OBJ) by subjecting the acquired image to image recognition; a drive control unit (40) that, if the recognition target object (OBJ) is detected by the image recognition unit (30), adjusts an emission condition of the emitted light to be emitted by the light source (10), on the basis of first outputs that are respectively output by one or more pixels corresponding to a region of the recognition target object (OBJ) on the basis of the reflected light of the emitted light emitted under a prescribed emission condition; and a distance calculation unit (50) that calculates the distance to the recognition target object (OBJ) on the basis of second outputs that are respectively output by the plurality of pixels on the basis of the reflected light of the emitted light emitted under the adjusted emission condition.

Description

Distance measuring device and distance measuring method

[0001] The present disclosure relates to a distance measuring device and a distance measuring method.

[0002] Distance measuring devices employing an indirect time-of-flight (TOF) method have been known. A distance measuring device employing the indirect TOF method includes, for example, a light source and a light receiving unit. The light receiving unit receives light irradiated from the light source and reflected by an object, and generates a distance image based on a signal output from the light receiving unit based on the reflected light.

[0003] In a distance measuring device employing an indirect TOF method, if the amount of light irradiated onto the target is small, the signal level output by the light receiving unit is low, resulting in a lower S / N ratio and reduced distance measurement accuracy. Furthermore, if the amount of light irradiated onto the target is too large, the signal level output by the light receiving unit is saturated, making accurate distance measurement impossible. Therefore, in order to improve distance measurement accuracy, a technique has been proposed for adjusting the amount of light irradiated by the light source of the distance measuring device (see, for example, Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2008-241435

[0005] In the conventional technology, further improvement in distance measurement accuracy is required.

[0006] The present disclosure provides a distance measuring device and a distance measuring method that can improve distance measurement accuracy.

[0007] A distance measuring device according to one aspect of the present disclosure includes a light source that irradiates irradiation light onto a predetermined range; a light receiving unit having a plurality of pixels that receive light reflected from the irradiation light within the predetermined range; an image recognition unit that acquires an image that includes at least a portion of the predetermined range and detects a predetermined object to be recognized by image recognition of the acquired image; a drive control unit that, when the image recognition unit detects the object to be recognized, adjusts the irradiation conditions of the irradiation light to be irradiated by the light source based on a first output output from each of one or more pixels among the plurality of pixels that corresponds to an area of ​​the object to be recognized based on the reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causes the light source to irradiate the irradiation light under the adjusted irradiation conditions; and a distance calculation unit that calculates a distance to the object to be recognized based on a second output output from each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

[0008] A distance measuring method according to one aspect of the present disclosure is a distance measuring method using a distance measuring device, the distance measuring device including a light source that irradiates irradiation light onto a predetermined range and a light receiving unit having a plurality of pixels that receive light reflected from the irradiation light within the predetermined range, the distance measuring method including an image recognition step of acquiring an image that includes at least a portion of the predetermined range and detecting a predetermined object to be recognized by image recognition of the acquired image, a drive control step of, when the object to be recognized is detected in the image recognition step, adjusting the irradiation conditions of the irradiation light to be irradiated by the light source based on a first output output from each of one or more pixels among the plurality of pixels that corresponds to a region of the object to be recognized based on the reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causing the light source to irradiate the irradiation light under the adjusted irradiation conditions, and a distance calculation step of calculating a distance to the object to be recognized based on a second output output from each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

[0009] According to the present disclosure, distance measurement accuracy can be improved.

[0010] FIG. 1 is a functional block diagram showing an example of the configuration of a distance measuring device according to an embodiment. FIG. 2 is a schematic diagram of a light receiving unit included in the distance measuring device according to an embodiment. FIG. 3A is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 3B is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 3C is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 3D is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 3E is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 3F is a schematic diagram for explaining an example of pixel arrangement in an image sensor included in the light receiving unit according to an embodiment. FIG. 4 is a diagram showing an example of a drive sequence of the distance measuring device according to an embodiment. FIG. 5A is a diagram showing an example of timing of an emission control pulse and an exposure control pulse in a period A0 / A1. FIG. 5B is a diagram showing an example of timing of an emission control pulse and an exposure control pulse in a period A2 / A3. FIG. 6A is a diagram showing an example of a signal based on charges generated by exposure in the A0 / A1 period. FIG. 6B is a diagram showing an example of a signal based on charges generated by exposure in the A2 / A3 period. FIG. 7 is a flowchart showing an example of operation of a distance measuring device according to an embodiment. FIG. 8 is a diagram showing a specific example of detection of a recognition target by an image recognition unit according to an embodiment. FIG. 9 is a diagram for explaining a condition table for adjusting the irradiation conditions of irradiation light. FIG. 10 is a diagram for explaining a method of correcting a second irradiation condition based on the temperature of the light receiving unit.

[0011] (Background of the Invention) As described above, in a distance measuring device employing an indirect TOF method, if the amount of light irradiated onto an object is inappropriate, the distance measurement accuracy decreases. For example, the amount of light irradiated onto the object attenuates in proportion to the square of the distance between the light source and the object, so the appropriate amount of light irradiated onto the object changes depending on the distance between the light source and the object.

[0012] Patent Literature 1 discloses a technology for adjusting the amount of light emitted by a light source of a distance measuring device to improve distance measurement accuracy. Specifically, in the technology disclosed in Patent Literature 1, a light receiving unit receives reflected light from an object, converts it into an electric charge, and adjusts the amount of light from the light source based on the electric charge accumulated in a charge storage unit. However, with the technology disclosed in Patent Literature 1, it is not possible to determine whether the reflected light from the object is the light reflected from the object to be measured, which may result in inappropriate adjustment of the amount of light from the light source. For example, if there are multiple objects within the range illuminated by the light from the light source, the amount of light emitted may be adjusted using reflected light from objects other than the object to be measured. Furthermore, for example, the amount of light reflected from the object received by the light receiving unit varies depending on the reflectivity of the object. Therefore, adjusting the amount of light emitted using reflected light from an object with a reflectivity different from that of the object to be measured may result in inappropriate adjustment of the amount of light for the object to be measured.

[0013] The present disclosure has been made based on the above-mentioned points of view of the inventors of the present application, and provides a distance measuring device and a distance measuring method that can improve distance measurement accuracy by more appropriately adjusting the irradiation conditions of the light emitted by the light source.

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0015] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step sequences shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. In each figure, substantially identical components are designated by the same reference numerals, and redundant descriptions may be omitted or simplified.

[0016] (Embodiment) [Configuration] First, the configuration of a distance measuring device according to this embodiment will be described. Fig. 1 is a functional block diagram showing an example of the configuration of a distance measuring device 100 according to this embodiment. Fig. 2 is a schematic diagram of a light receiving unit 20 provided in distance measuring device 100 according to this embodiment.

[0017] The distance measuring device 100 is a distance measuring device that measures distance using an indirect TOF method. The distance measuring device 100 generates a distance image that indicates the distance to an object included in an imaging range. The distance measuring device 100 according to this embodiment can improve distance measurement accuracy, which can lead to higher performance of authentication processing, for example, by reducing the number of authentication retries in systems that perform image authentication, such as face authentication systems and driver monitoring systems. Therefore, the distance measuring device 100 may be used in systems that perform image authentication, such as face authentication systems and driver monitoring systems.

[0018] As shown in FIG. 1, the distance measuring device 100 includes a light source 10, a light receiving unit 20, an image recognition unit 30, a drive control unit 40, a distance calculation unit 50, a background light measurement unit 60, a temperature sensor 70, and a memory unit 80.

[0019] The light source 10 irradiates a predetermined range (a target space to be measured) including at least a portion of the imaging range of the light receiving unit 20 with irradiation light in accordance with an input light emission control signal. The irradiation conditions of the light emitted by the light source 10 are controlled by the input light emission control signal. For example, the light source 10 irradiates the predetermined range with pulsed light at a predetermined pulse width multiple times in accordance with the timing indicated by a light emission control pulse included in the input light emission control signal. FIG. 1 schematically shows a case where a predetermined recognition object OBJ is present in the predetermined range. The light source 10 includes, for example, a light emitting element such as a light emitting diode or laser element that emits infrared light (IR), and an optical system that receives the light from the light emitting element and controls the light distribution from the light emitting element.

[0020] The light source 10 may be capable of changing the light emission intensity of the light source 10 as an irradiation condition of the emitted light. A known dimming method may be used to change the light emission intensity of the light source 10. The light source 10 may include, for example, a plurality of light-emitting elements, and the light emission intensity is changed by changing the number of light-emitting elements that are turned on. The light source 10 may also change the light emission intensity by changing the voltage input to the light-emitting elements.

[0021] Furthermore, the light source 10 may be capable of changing the light distribution angle of the irradiation light as an irradiation condition of the irradiation light. For example, the light source 10 may include multiple sub-light sources with different light distribution angles of the irradiation light, and the light distribution angle of the irradiation light may be changed by switching the sub-light sources to be emitted. The light source 10 may also change the light distribution angle of the irradiation light by switching the optical system onto which light from the light-emitting elements is incident. The light source 10 may also continuously change the light distribution angle by combining multiple optical systems onto which light from the light-emitting elements is incident and adjusting the spacing between the multiple optical systems, for example. Even when the light source 10 emits light with the same emission intensity, the narrower the light distribution angle of the irradiation light, the higher the irradiation intensity of the irradiation light and the higher the illuminance of the irradiation light irradiated on the recognition target OBJ. In other words, by changing the light distribution angle, the amount of irradiation light reaching the recognition target OBJ can be increased without increasing the emission intensity and amount of light emitted by the light-emitting elements of the light source 10.

[0022] In the light source 10, the control targets of the irradiation conditions of the irradiated light controlled by the input light emission control signal include, for example, at least one of the number of times pulsed light is emitted, the light emission intensity of the light source 10, and the light distribution angle of the irradiated light.

[0023] As shown in FIG. 2 , the light receiving unit 20 includes one or more image sensors 20A. The image sensor 20A receives light from a predetermined range and captures an image that captures at least a portion of the predetermined range. The image sensor 20A is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The image sensor 20A has a plurality of pixels 21 arranged two-dimensionally. For the sake of explanation, FIG. 2 shows a configuration of 4 pixels horizontally and 4 pixels vertically, for a total of 16 pixels, but the number of pixels 21 included in the image sensor 20A is not particularly limited. The number of pixels 21 included in the image sensor 20A may be, for example, between 20,000 pixels and 5,000,000 pixels.

[0024] Each of the plurality of pixels 21 includes at least one photoelectric conversion element that receives light emitted from the light source 10 and reflected by a recognition target object OBJ or the like within a predetermined range and converts the light into an electric charge. For example, a photodiode or the like is used as the photoelectric conversion element.

[0025] Each of the multiple pixels 21 outputs a signal based on the charge obtained by exposure in accordance with the exposure control signal. The image sensor 20A, for example, transfers the charge obtained by exposure in accordance with the exposure control signal to a vertical transfer path (VCCD) or a charge storage unit (FD) for each pixel 21, and reads out the signal based on the charge obtained by exposure. The image sensor 20A, for example, performs analog-to-digital (AD) conversion when reading out the signal, and outputs the AD-converted digital signal. The exposure period of the image sensor 20A corresponds, for example, to the emission period of light emitted by the light source 10. Furthermore, when the light source 10 emits pulsed light multiple times, the number of exposure periods is set according to the number of times the pulsed light is emitted.

[0026] An example of the arrangement of pixels 21 in the image sensor 20A will now be described. Figures 3A to 3F are schematic diagrams for explaining an example of the arrangement of pixels in the image sensor included in the light receiving unit 20 according to this embodiment.

[0027] The light receiving unit 20 includes, for example, a plurality of image capturing elements 20A. In this case, the light receiving unit 20 includes, for example, the image capturing element 20A1 shown in Fig. 3A and at least one of the image capturing elements 20A2 and 20A3 shown in Fig. 3B and 3C as the plurality of image capturing elements 20A.

[0028] 3A, the image sensor 20A1 includes a plurality of pixels 21, each of which is an IR pixel 21a that is sensitive to infrared light. The plurality of IR pixels 21a are arranged in a matrix in the image sensor 20A1. The IR pixels 21a receive reflected light of light emitted by the light source 10 and output signals based on the received reflected light. The signals output by the IR pixels 21a are used to generate an IR image and a distance image.

[0029] 3B, the image sensor 20A2 has a plurality of pixels 21, including R pixels 21b sensitive to red light, G pixels 21c sensitive to green light, and B pixels 21d sensitive to blue light. In the image sensor 20A2, the R pixels 21b, G pixels 21c, and B pixels 21d are arranged in a Bayer array. Signals output from the R pixels 21b, G pixels 21c, and B pixels 21d are used to generate a color visible light image (RGB image).

[0030] 3C, the image sensor 20A3 includes a plurality of pixels 21 each composed of a BW pixel 21e that is sensitive to all visible light. The image sensor 20A3 includes a plurality of BW pixels 21e arranged in a matrix. The signals output by the BW pixels 21e are used to generate a black-and-white visible light image (BW image).

[0031] Furthermore, the light receiving unit 20 may include a single imaging element 20A. In this case, the light receiving unit 20 includes, as the imaging element 20A, the imaging element 20A4 shown in Fig. 3D, the imaging element 20A5 shown in Fig. 3E, or the imaging element 20A6 shown in Fig. 3F.

[0032] 3D, the image sensor 20A4 has IR pixels 21a, R pixels 21b, G pixels 21c, and B pixels 21d as the plurality of pixels 21. The image sensor 20A4 has a pixel arrangement in which half of the G pixels 21c of the image sensor 20A2 are replaced with IR pixels 21a.

[0033] 3E and 3F, the image sensors 20A5 and 20A6 each have IR pixels 21a and BW pixels 21e as the plurality of pixels 21. In the image sensor 20A5, adjacent pixels in both the row and column directions are arranged as an IR pixel 21a and a BW pixel 21e. In the image sensor 20A6, adjacent pixels in the column direction are an IR pixel 21a and a BW pixel 21e, and in the row direction, IR pixels 21a and BW pixels 21e are arranged side by side.

[0034] The light receiving unit 20 having the above-described configuration makes it possible to generate an IR image, a distance image, and a visible light image using the signal output by the light receiving unit 20. Note that it is not essential to generate a visible light image using the signal output by the light receiving unit 20, and if a visible light image is not to be generated, the light receiving unit 20 may be configured with a single image sensor 20A1.

[0035] Referring back to FIG. 1 , the image recognition unit 30 acquires an image that captures at least a portion of a predetermined range irradiated with light from the light source 10 and performs image recognition on the acquired image. The image recognition unit 30 detects a predetermined recognition target object OBJ through image recognition of the image that captures at least a portion of the predetermined range. A known method can be used for image recognition by the image recognition unit 30. For example, the image recognition unit 30 outputs the position of an area in which the recognition target object OBJ appears using a trained image recognition model 81 that has been trained in advance through machine learning to detect the recognition target object OBJ from the image. For example, the image recognition unit 30 extracts feature amounts from the image and inputs the extracted feature amounts to the image recognition model 81. The image recognition model 81 is, for example, stored in the storage unit 80 as an image recognition model 81 that has been trained in advance using a training image, and outputs the probability that the predetermined recognition target object OBJ exists in the image in response to the input feature amounts. The image recognition unit 30 determines that a recognition target object OBJ exists when the probability output by the image recognition model 81 is equal to or greater than a predetermined threshold. The image recognition model 81 may be, for example, a neural network model, but is not particularly limited to this.

[0036] The type of recognition object OBJ is not particularly limited and is set, for example, depending on the application in which the distance measuring device 100 is used. Examples of recognition object OBJ include a human face, a human hand, an entire person, a moving object such as a vehicle, a building, and an animal. The method of image recognition by the image recognition unit 30 is not particularly limited. For example, if the recognition object OBJ is a human face, the human face may be detected using a known face detection algorithm. Furthermore, the image recognition unit 30 may detect multiple types of recognition object OBJ in the image recognition. For example, the image recognition unit 30 may detect a person and a vehicle as the recognition object OBJ.

[0037] The drive control unit 40 outputs various control signals for controlling the drive of the light source 10 and the light receiving unit 20. As a control signal for controlling the drive of the light source 10, the drive control unit 40 outputs, for example, a light emission control signal that instructs the light source 10 to irradiate irradiation light at a predetermined pulse width. The drive control unit 40 also outputs, as a control signal for controlling the drive of the light receiving unit 20, an exposure control signal that instructs each pixel 21 of the light receiving unit 20 to expose to light. The drive control unit 40 adjusts the irradiation conditions of the irradiation light emitted by the light source 10 based on the output of the light receiving unit 20 and the detection result of the recognition object OBJ by the image recognition unit 30.

[0038] The distance calculation unit 50 generates a distance image by performing a predetermined calculation based on a signal output from each pixel 21 (specifically, the IR pixel 21 a) that receives reflected light of the irradiated light within a predetermined range. The distance calculation unit 50 calculates the distance to the recognition target object OBJ for each pixel 21, for example, based on the signal output from each pixel 21. The distance calculation unit 50 outputs the calculated distance for each pixel 21 as a pixel value.

[0039] The background light measuring unit 60 acquires signals output by the pixels 21 of the light receiving unit 20 based on background light that does not include reflected light of the light emitted from the light source 10. Note that the distance measuring device 100 does not necessarily have to include the background light measuring unit 60.

[0040] The drive control unit 40, the distance calculation unit 50, and the background light measurement unit 60 are processing circuits realized by, for example, a memory that stores a program and a processor that executes the program. The program may be stored in the storage unit 80. Although shown as separate blocks in the block diagram, all or part of the drive control unit 40, the distance calculation unit 50, and the background light measurement unit 60 may be configured using the same memory and processor. Details of the processing by the drive control unit 40, the distance calculation unit 50, and the background light measurement unit 60 will be described later.

[0041] The temperature sensor 70 measures the temperature of the light receiving unit 20 (specifically, the image sensor 20A). The temperature sensor 70 outputs the measured temperature of the light receiving unit 20 to the drive control unit 40. The temperature sensor 70 may directly measure the temperature of the light receiving unit 20, or may indirectly measure the temperature of the light receiving unit 20 by measuring the temperature around the light receiving unit 20, for example. Note that the distance measuring device 100 does not necessarily have to include the temperature sensor 70.

[0042] The storage unit 80 is a storage device that stores information and data necessary for processing performed by the distance measuring device 100. The storage unit 80 stores, for example, an image recognition model 81 used for image recognition and a condition table 82 for determining the irradiation conditions of the light source 10. The storage unit 80 is realized by, for example, a semiconductor memory or an HDD (Hard Disk Drive). Note that at least a part of the storage unit 80 may be provided in a device separate from the distance measuring device 100, and the distance measuring device 100 may acquire the data stored in the storage unit 80 via a network such as the Internet.

[0043] [Driving Sequence of Distance Measuring Device] Next, the driving of the light source 10 and the light receiving unit 20 when the distance measuring device 100 performs distance measurement will be described. Fig. 4 is a diagram showing an example of the driving sequence of the distance measuring device 100 according to this embodiment. Fig. 4 shows the driving sequence when the light receiving unit 20 outputs a signal for the distance calculation unit 50 to calculate the distance to an object such as a recognition object OBJ.

[0044] 4, in the drive sequence when the distance measuring device 100 performs distance measurement, one frame period is composed of a light emission exposure period and a readout period. First, in the light emission exposure period, the light source 10 emits light and the IR pixels 21a of the light receiving unit 20 are exposed to light. Then, in the readout period, signals based on the charges generated in the IR pixels 21a during the light emission exposure period are read out.

[0045] During the light emission exposure period, the same light emission and exposure sequence is repeated β times. In the example shown in FIG. 4 , β times is 8 times, and the same light emission and exposure sequence as during the β1 period is repeated from the β2 period to the β8 period. In the example shown in FIG. 4 , the β1 period is divided into an A0 / A1 period and an A2 / A3 period. During the A0 / A1 period, a light emission control signal including α light emission control pulses is output from the drive control unit 40 to the light source 10, and an exposure control signal including α×2 exposure control pulses is output from the drive control unit 40 to the light receiving unit 20. During the A2 / A3 period, no light emission control pulses are output from the drive control unit 40 to the light source 10, and an exposure control signal including α×2 exposure control pulses is output from the drive control unit 40 to the light receiving unit 20. As a result, during the light emission exposure period, the light source 10 emits α×β pulses of light as irradiation light.

[0046] The light emission control pulse is a control pulse that instructs the light source 10 to emit pulsed light, and when it is at a high level, causes the light source 10 to emit irradiation light. The exposure control pulse is a control pulse that instructs the IR pixel 21a to expose, and when it is at a low level, causes the IR pixel 21a to expose. During the A0 / A1 period, the light emission period of the light source 10 and the exposure period of the IR pixel 21a are related to each other so as to have a predetermined phase difference.

[0047] Next, the details of the light-emission control pulse and the exposure control pulse will be described using Figures 5A and 5B. Figure 5A is a diagram showing an example of the timing of the light-emission control pulse and the exposure control pulse in the A0 / A1 period. Figure 5B is a diagram showing an example of the timing of the light-emission control pulse and the exposure control pulse in the A2 / A3 period. Figures 5A and 5B show a repeat unit of the light-emission control pulse and the exposure control pulse, and in the A0 / A1 period, the sequence shown in Figure 5A is performed α times, and in the A2 / A3 period, the sequence shown in Figure 5B is performed α times.

[0048] As shown in FIG. 5A , during the A0 / A1 period, two exposure control pulses A0 and A1 are output in response to one light emission control pulse. The drive control unit 40 outputs a light emission control pulse with a pulse width Tp. This causes the light source 10 to emit pulsed light with the pulse width Tp. The drive control unit 40 also outputs exposure control pulses A0 and A1 with different start timings relative to the light emission control pulse. In the example shown in FIG. 5A , the exposure widths (exposure periods) of the exposure control pulses A0 and A1 are the same as the pulse width Tp of the irradiated light and do not overlap with each other. The exposure control pulse A0, for example, starts at the same timing as the start timing of the light emission control pulse. The exposure control pulse A1, for example, starts at the end timing of the light emission control pulse and the exposure control pulse A0. In other words, the phase difference between the exposure control pulse A0 and the exposure control pulse A1 is the pulse width Tp of the irradiated light. Note that, as long as the exposure control pulse A0 and the exposure control pulse A1 are output with a phase difference, the exposure control pulse A0 does not have to start at the same timing as the start timing of the light-emission control pulse. For example, the exposure control pulse A0 may start later than the start timing of the light-emission control pulse by a predetermined offset.

[0049] As shown in FIG. 5B , during the A2 / A3 period, no light emission control pulse is output, and two exposure control pulses A2 and A3 are output. The drive control unit 40 outputs exposure control pulses A2 and A3 that start at different times. In the example shown in FIG. 5B , the exposure widths (exposure periods) of the exposure control pulses A2 and A3 are the same as the pulse width Tp of the irradiated light, and do not overlap with each other. The exposure control pulse A3 starts, for example, at the end timing of the exposure control pulse A2. Note that during the A2 / A3 period, only the exposure control pulse A2 may be output, without the exposure control pulse A3 being output.

[0050] The IR pixel 21a has, for example, multiple charge accumulation sections, and the charges generated by each exposure of the multiple exposure control pulses A0 to A3 are allocated and accumulated in different charge accumulation sections. For example, the number of the multiple exposure control pulses A0 to A3 is the same as the number of the multiple charge accumulation sections of the IR pixel 21a. During the light emission exposure period, the light source 10 irradiates α×β pulsed light as irradiation light, and charges generated by the α×β exposures are accumulated in each charge accumulation section. During the readout period, signals based on the charges accumulated in each charge accumulation section by the α×β exposures are read out. In other words, the IR pixel 21a outputs signals corresponding to the amount of charge generated by each exposure of the α×β exposure control pulses A0 to A3.

[0051] The drive control unit 40 can change the number of times that the pulsed light is emitted by changing the number of times that the light emission control pulse is output. For example, the drive control unit 40 changes α when changing the number of times that the pulsed light is emitted as an irradiation condition of the light source 10. Because the drive control unit 40 outputs the exposure control pulses A0 to A3 the same number of times as the light emission control pulse, when changing the number of times that the light emission control pulse is output, the drive control unit 40 also changes the number of times that the exposure control pulses A0 to A3 are output by the same number.

[0052] The distance calculation unit 50 calculates the distance to an object, such as a recognition object OBJ, that is irradiated with pulsed light based on the signal output from each IR pixel 21a by the above-described drive sequence. The distance calculation unit 50 calculates the distance for each IR pixel 21a, thereby generating a distance image. The calculation of the distance (e.g., the distance from the ranging device 100 to the recognition object OBJ) by the distance calculation unit 50 is performed, for example, as follows.

[0053] 6A is a diagram showing an example of a signal based on charges generated by exposure during the A0 / A1 period. FIG. 6B is a diagram showing an example of a signal based on charges generated by exposure during the A2 / A3 period. In FIGS. 6A and 6B, signals based on charges generated by exposure according to each exposure control pulse are schematically shown as patterned rectangles. The area of ​​the rectangle corresponds to the signal size. Note that while FIGS. 6A and 6B show signals S0 to S3 corresponding to charges accumulated during a single exposure, in reality, each of the multiple IR pixels 21a outputs signals S0 to S3 based on charges accumulated during α×β exposures according to the drive sequence described above.

[0054] As shown in FIG. 6A , the reflected light emitted from the light source 10 and reflected by the object OBJ returns to the distance measuring device 100 with a delay of Δt from the irradiation of the emitted light and enters the light receiving unit 20. A portion of the reflected light is received by the IR pixel 21a during exposure in accordance with the exposure control pulse A0 and converted into an electric charge. Furthermore, since the reflected light returns with a delay of Δt from the irradiation of the emitted light, the remaining portion of the reflected light corresponding to the delayed time Δt is received by the IR pixel 21a during exposure in accordance with the exposure control pulse A1 and converted into an electric charge. Furthermore, background light, which does not include the reflected light, is received by the IR pixel 21a during each exposure in accordance with the exposure control pulses A0 and A1 and converted into an electric charge. Therefore, the signal S0 based on the electric charge generated by the exposure in accordance with the exposure control pulse A0 is a signal including a component of the signal S0a corresponding to the reflected light for a time width obtained by subtracting the time Δt from the pulse width Tp, and a component of the signal S0b corresponding to the background light. Furthermore, the signal S1 based on the charge generated by exposure in accordance with the exposure control pulse A1 is a signal that includes a component of the signal S1a corresponding to the reflected light of the time width Δt and a component of the signal S1b corresponding to the background light.

[0055] Also, as shown in FIG. 6B , during the A2 / A3 period, the light source 10 does not emit any irradiating light. Therefore, only background light, not including reflected light, is received by the IR pixel 21a during the exposures according to the exposure control pulses A2 and A3 and converted into electric charges. Therefore, signals S2 and S3 based on the electric charges generated during the exposures according to the exposure control pulses A2 and A3 correspond to the background light. Therefore, the magnitudes of signals S0b, S1b, S2, and S3 are substantially the same. Therefore, the influence of background light can be eliminated by subtracting signal S2 or S3 from signals S0 and S1, respectively. Furthermore, the sum of signals S0 and S1 after eliminating the influence of background light is a signal based on reflected light with a time width of pulse width Tp. Therefore, the time Δt is calculated using the following formula:

[0056] Δt=Tp×(S1-S3) / [(S0-S2)+(S1-S3)]

[0057] As a result, if the distance from the distance measuring device 100 to the recognition target object OBJ is D and the speed of light is c, the light emitted by the light source 10 travels the distance D round trip in time Δt, and the distance D can be calculated by the following formula: The distance D can also be said to be the flight distance of the emitted light and the reflected light.

[0058] D=c×Δt / 2 =(c×Tp / 2)×(S1-S3) / [(S0-S2)+(S1-S3)]

[0059] Because distance D is calculated as described above, if the amount of reflected light is small and the signal levels of signals S0 and S1 based on the reflected light are small, the signal-to-noise ratio decreases, resulting in poor distance measurement accuracy. Furthermore, if the amount of reflected light increases and the amount of charge that can be stored in the charge storage unit becomes saturated, the signal levels of signals S1 and S2 also become saturated, making accurate distance measurement impossible. In distance measuring device 100, the amount of reflected light received by light receiving unit 20 can be adjusted by adjusting the irradiation conditions of light emitted by light source 10 using a method described below, thereby improving distance measurement accuracy.

[0060] The distance calculated by the distance calculation unit 50 does not have to be an absolute distance, but may be a relative distance value. For example, the distance calculated by the distance calculation unit 50 may be a normalized distance ranging from 0 to 1, or a time Δt proportional to the distance may be used as a relative distance value. Furthermore, as long as the distance can be calculated based on the signal output by the light receiving unit 20, the timing of the light-emission control pulse and the exposure control pulse is not limited to the example described above and is not particularly limited. For example, if the influence of background light is minor, the distance calculation unit 50 may calculate the distance without subtracting the signal S2 or S3 from the signal S1 and the signal S0. In this case, the A2 / A3 period may be omitted. Furthermore, for example, during the A0 / A1 period in which the light source 10 emits irradiation light, three or more exposure control pulses may be output for one light-emission control pulse, each having a different start timing relative to the light-emission control pulse. In this case, in the exposure according to at least one of the three or more exposure control pulses, only background light is received by the IR pixel 21 a, so the A2 / A3 period does not need to be included in the drive sequence. In addition, the distance measuring device 100 may perform drive sequences and distance calculations other than those described above by using various known indirect TOF methods such as a continuous wave (CW) TOF method.

[0061] An IR image may also be generated based on the signals obtained by the above drive sequence. For example, an IR image can be generated by calculating the sum of the signals S0 and S1 of each IR pixel 21a as a luminance value. The IR image may be generated by the distance calculation unit 50 or the image recognition unit 30. Furthermore, background light components may also be subtracted when generating the IR image. Furthermore, the IR image may be generated by the light receiving unit 20 performing an imaging sequence similar to that of an imaging device that generates a normal two-dimensional image while the light source 10 is emitting irradiation light.

[0062] [Operation] Next, an example of operation of the distance measuring device 100 according to this embodiment will be described. FIG. 7 is a flowchart showing an example of operation of the distance measuring device 100 according to this embodiment. FIG. 7 shows an example of a distance measuring method using the distance measuring device 100. In FIG. 7, steps S13 and S14 combined are an example of an image recognition step, step S15 is an example of a drive control step, and step S18 is an example of a distance calculation step. Note that the following description will be given of a case in which the light receiving unit 20 includes an image sensor 20A1 having a plurality of IR pixels 21a shown in FIG. 3A as an image sensor. The distance measuring method described below is a method in which distance is measured using a plurality of IR pixels 21a of the image sensor 20A1.

[0063] First, the drive control unit 40 causes the light source 10 to emit light to a predetermined area under a first illumination condition, which is a predetermined illumination condition (step S11). The light receiving unit 20 receives light reflected from the predetermined area of ​​the light emitted under the first illumination condition in step S11 and outputs a signal based on the reflected light (step S12). In steps S11 and S12, for example, the drive sequence described above in FIGS. 4, 5A, and 5B is performed, and each of the IR pixels 21a of the light receiving unit 20 outputs the signals S0 to S3. At this time, the distance calculation unit 50 may acquire the output signals S0 to S3 and calculate the distance based on the signals S0 to S3 to generate a distance image. The first illumination condition for causing the light source 10 to emit light may be a predetermined illumination condition or may be the illumination condition of the light emitted by the light source 10 during the previous operation of the distance measuring device 100.

[0064] Next, the image recognition unit 30 acquires an image showing at least a portion of the predetermined range illuminated by the light emitted by the light source 10 (step S13). For example, the image recognition unit 30 acquires at least one of an IR image captured using infrared light, a visible light image (an RGB image or a BW image) captured using visible light, and a distance image as the image showing at least a portion of the predetermined range. The IR image, visible light image, and distance image acquired by the image recognition unit 30 are, for example, images generated based on signals output by the light receiving unit 20. The image recognition unit 30 may acquire the signals output by the light receiving unit 20 and use the acquired signals as they are as images, or may acquire images by performing a predetermined calculation on the signals output by the light receiving unit 20 to generate images. For example, the drive control unit 40 controls the light receiving unit 20 and, if necessary, the light source 10 at the timing up to step S13 to cause the light receiving unit 20 to output signals for generating the image used in step S13.

[0065] When image recognition unit 30 acquires an IR image or a distance image, image recognition unit 30 may generate the image based on the signal output by light receiving unit 20 as a result of performing steps S11 and S12. Image recognition unit 30 may also acquire the distance image from distance calculation unit 50. In this case, distance calculation unit 50 may generate the distance image based on the signal output by light receiving unit 20 as a result of performing steps S11 and S12.

[0066] Next, the image recognition unit 30 detects a predetermined recognition target object OBJ by image recognition of the acquired image (step S14). FIG. 8 is a diagram showing a specific example of detection of the recognition target object OBJ by the image recognition unit 30. FIG. 8 shows an example in which a human face is detected as the recognition target object OBJ. The image recognition unit 30 detects the recognition target object OBJ from the acquired image 35 using, for example, an image recognition model 81 stored in the storage unit 80. Specifically, the image recognition unit 30 extracts feature amounts from the image 35 and inputs the extracted feature amounts to the image recognition model 81. The image recognition model 81 outputs a probability that the recognition target object OBJ exists in the image based on the input feature amounts. If the probability output by the image recognition model 81 is equal to or greater than a predetermined threshold, the image recognition unit 30 determines that the recognition target object OBJ exists in the image 35 and outputs the position of the area 36 in which the recognition target object OBJ appears. For example, the area 36 is rectangular, and the image recognition unit 30 outputs coordinates that define the rectangular area as the position of the area 36 .

[0067] Furthermore, when a plurality of recognition objects OBJ are detected in the image 35, the image recognition unit 30 may select one recognition object OBJ from the plurality of recognition objects OBJ to be used for adjusting the irradiation conditions, which will be described later.

[0068] For example, when selecting one recognition target object OBJ from a plurality of recognition target objects OBJ, the image recognition unit 30 may select the recognition target object OBJ based on registration information. The registration information is, for example, information indicating the face of a person using the application, and is stored in the storage unit 80. The image recognition unit 30 preferentially selects the recognition target object OBJ corresponding to the registration information.

[0069] Furthermore, for example, when selecting one recognition object OBJ from a plurality of recognition objects OBJ, the image recognition unit 30 may select the recognition object OBJ based on the distance to the recognition object OBJ. For example, the image recognition unit 30 selects the recognition object OBJ whose distance to the recognition object OBJ is closest as calculated by the distance calculation unit 50. Furthermore, the image recognition unit 30 may select the recognition object OBJ whose distance to the recognition object OBJ is closest to a predetermined distance.

[0070] Furthermore, for example, when selecting one recognition object OBJ from a plurality of recognition objects OBJ, the image recognition unit 30 may select the recognition object OBJ based on the size of the area 36 in which the recognition object OBJ appears. For example, the image recognition unit 30 selects the recognition object OBJ appearing in the area 36 having the largest number of pixels within the rectangle of the area 36. Furthermore, the image recognition unit 30 may select the recognition object OBJ appearing in the area 36 having the number of pixels within the rectangle of the area 36 closest to a predetermined number of pixels.

[0071] Furthermore, for example, when selecting one recognition object OBJ from multiple recognition objects OBJ, the image recognition unit 30 may select the recognition object OBJ based on the reflectance of the recognition object OBJ. Even if the distance to the recognition object OBJ is the same, the signal level of the signal used to calculate the distance changes depending on the reflectance of the recognition object OBJ. Therefore, the reflectance of the recognition object OBJ can be calculated from the distance and the signal level. For example, the image recognition unit 30 selects the recognition object OBJ whose reflectance is closest to a predetermined reflectance. For example, if the recognition object OBJ is a human face, the image recognition unit 30 selects the recognition object OBJ whose reflectance is closest to 50%, which is the approximate reflectance of a human face.

[0072] Note that steps S11 and S12 may be performed after step S13 or step S14, provided that they are performed before step S15 below.

[0073] Next, when the image recognition unit 30 detects the object OBJ, the drive control unit 40 adjusts the irradiation conditions of the light emitted by the light source 10 based on the first output output from one or more IR pixels 21a among the multiple IR pixels 21a that correspond to the region 36 of the object OBJ (step S15). As a result, the drive control unit 40 determines second irradiation conditions, which are the adjusted irradiation conditions. The first output is output from each of the one or more IR pixels 21a based on reflected light within a predetermined range of the light emitted under the first irradiation conditions. The first output is an output including one or more signals output from the IR pixels 21a, and includes, for example, signals S0 and S1, which are signals based on the reflected light described in FIG. 6A . The first output may further include signals S2 and S3. Based on the first output, the drive control unit 40 adjusts the irradiation conditions of the light so that the amount of reflected light received by the IR pixels 21a is a desired amount. The drive control unit 40 adjusts the amount of reflected light received by the IR pixel 21 a to a desired amount by adjusting the irradiation conditions of the irradiation light, for example. Furthermore, the drive control unit 40 adjusts the exposure conditions of the IR pixel 21 a along with the irradiation conditions of the irradiation light, as necessary.

[0074] The one or more IR pixels 21a may be, for example, IR pixels 21a that receive light from region 36 of the target object OBJ. Alternatively, the one or more IR pixels 21a may be IR pixels 21a that receive light from a central region among regions obtained by dividing region 36 of the target object OBJ or from a closest region of the target object OBJ. This allows region 36 to be excluded from regions used for adjusting the illumination conditions, even if it includes a background or the like. The central region is the central region among regions obtained by dividing region 36. The closest region is a region that includes a point, among regions obtained by dividing region 36, at which the distance from distance measuring device 100 to the target object OBJ is shortest. The closest region is determined using the distance calculated by distance calculation unit 50.

[0075] The drive control unit 40 adjusts the irradiation conditions of the irradiated light by, for example, adjusting at least one of the number of times that the light source 10 emits pulsed light as irradiated light, the light emission intensity of the light source 10, and the light distribution angle of the irradiated light. In other words, the means for adjusting the irradiation conditions by the drive control unit 40 (the object of adjustment of the irradiation conditions by the drive control unit 40) includes at least one of the number of times that the light source 10 emits pulsed light as irradiated light, the light emission intensity of the light source 10, and the light distribution angle of the irradiated light. The drive control unit 40 may adjust two or more of the number of times that the light source 10 emits pulsed light as irradiated light, the light emission intensity of the light source 10, and the light distribution angle of the irradiated light.

[0076] Here, a method (algorithm) for adjusting the irradiation conditions of the irradiation light by the drive control unit 40 in step S15 will be described. Examples of the method for adjusting the irradiation conditions of the irradiation light by the drive control unit 40 include the following first and second methods.

[0077] In the first method, the drive control unit 40 calculates a representative value of the signal level of the first output output from each of the one or more IR pixels 21a and adjusts the irradiation conditions of the irradiation light based on the calculated representative value. The representative value is, for example, an average value or a median value. The signal level of the first output is, for example, the signal level obtained by adding up the signal S0 and the signal S1. The signal level of the first output can also be considered as the luminance value of the reflected light at the IR pixel 21a.

[0078] In calculating the representative value of the signal level of the first output, the drive control unit 40 may calculate the representative value by calculating the average or median of the signal levels of the first outputs output by each of the one or more IR pixels 21 a that are equal to or greater than a predetermined threshold. As a result, the first output output by the IR pixel 21 a that receives reflected light from a background area is excluded from the calculation of the representative value because the IR pixel 21 a receives a small amount of reflected light and has a low signal level. Furthermore, if the recognition target object OBJ is a human face, the first output output by the IR pixel 21 a that receives reflected light from hair, which has low reflectivity, is excluded from the calculation of the representative value because the IR pixel 21 a receives a small amount of reflected light and has a low signal level.

[0079] The drive control unit 40 adjusts the irradiation conditions so that the calculated representative value becomes the target signal level, for example. For example, the drive control unit 40 calculates the ratio between the calculated representative value and the target signal level (ratio = target signal level / calculated representative value), and adjusts the irradiation conditions of the irradiation light so that the amount of light reaching the recognition target object OBJ changes by that ratio from the first irradiation conditions. The drive control unit 40 sets the irradiation conditions after adjustment to, for example, the number of times the pulsed light is emitted or the light emission intensity of the light source 10 under the first irradiation conditions multiplied by that ratio.

[0080] In the second method, the drive control unit 40 calculates a representative value of the distance to the region of the object to be recognized OBJ, which is calculated based on the first outputs output by each of the one or more IR pixels 21 a, and adjusts the irradiation conditions of the irradiation light based on the calculated representative value. The representative value is, for example, an average value or a median value. The distance is calculated, for example, by the distance calculation unit 50 using the signals S0 to S3 included in the first outputs using the method described above.

[0081] In calculating the representative value of the distance, the drive control unit 40 may calculate the representative value by calculating the average or median of distances to the region of the recognition target object OBJ that are equal to or less than a predetermined threshold value, calculated based on the first outputs output from each of the one or more IR pixels 21 a. As a result, the first outputs output from the IR pixels 21 a that receive reflected light from a background region are excluded from the calculation of the representative value.

[0082] The drive control unit 40 adjusts the irradiation conditions of the irradiation light, for example, using the calculated representative value of the distance and a condition table 82 stored in the storage unit 80. The condition table 82 is an example of a table in which the representative value of the distance corresponds to the setting values ​​of the irradiation conditions. The drive control unit 40 adjusts the irradiation conditions of the irradiation light by referring to the condition table 82. FIG. 9 is a diagram illustrating the condition table 82 for adjusting the irradiation conditions of the irradiation light. (a) of FIG. 9 shows the relationship between the distance and the signal level of the signal output by the IR pixel 21a when the distance to the recognition target object OBJ is calculated by irradiating the recognition target object OBJ with irradiation light under certain irradiation conditions. (b) of FIG. 9 shows the relationship between the representative value of the distance in the condition table 82 and the setting values ​​of the irradiation conditions (e.g., the number of times pulsed light is emitted or the emission intensity of the light source 10).

[0083] As the distance from the distance measuring device 100 to the target object OBJ increases, the amount of light reaching the target object OBJ decreases due to attenuation of the irradiated light. Therefore, for a predetermined target object OBJ, the reflectivity of the target object OBJ is also determined, and the amount of reflected light received by the IR pixel 21a is also determined by the distance to the target object OBJ. For example, if the target object OBJ is a human face, the reflectivity is approximately 50%. As a result, as shown in FIG. 9A, as the distance to the target object OBJ increases, the signal level of the signal output by the IR pixel 21a based on the reflected light also decreases. Therefore, as the distance to the target object OBJ increases, the signal-to-noise ratio decreases, resulting in a decrease in distance measurement accuracy. Therefore, as shown in FIG. 9B, a condition table 82 is set based on the relationship shown in FIG. 9A, such that the number of light emissions or the light emission intensity increases as the representative value of the distance calculated above increases, so that the signal level of the signal output by the IR pixel 21a reaches the target signal level. This allows the illumination conditions to be appropriately adjusted, thereby improving distance measurement accuracy. If the reflectance of the object irradiated with the illumination light differs from that of the object OBJ, the relationship between distance and signal level will differ from that shown in FIG. 9A. However, since the illumination conditions are adjusted when the image recognition unit 30 detects the object OBJ as described above, it is possible to avoid a situation in which the illumination conditions are not appropriately adjusted due to the reflectance of the object irradiated with the illumination light differing from the expected value. In the second method, the first illumination conditions are set to the illumination conditions used to set the condition table 82, for example. For example, when the condition table 82 having the relationship shown in FIG. 9B is used, the first illumination conditions are set to obtain the relationship shown in FIG. 9A. In the first method, a table in which representative signal level values ​​correspond to set values ​​of the illumination conditions may also be used.

[0084] In the second method, the drive control unit 40 may correct the second irradiation conditions, which are determined by adjusting the irradiation conditions of the irradiation light based on the representative distance value, based on the temperature of the light-receiving unit 20 measured by the temperature sensor 70. Specifically, the drive control unit 40 corrects the second irradiation conditions so that the amount of irradiation light irradiated on the recognition target object OBJ decreases as the temperature of the light-receiving unit 20 measured by the temperature sensor 70 increases. FIG. 10 is a diagram illustrating a method for correcting the second irradiation conditions based on the temperature of the light-receiving unit 20. (a) of FIG. 10 illustrates the relationship between the temperature of the light-receiving unit 20 and the signal level of the signal output by the IR pixel 21a when the light-receiving unit 20 receives a constant amount of light. (b) of FIG. 10 illustrates the relationship between the correction gain for correcting the second irradiation conditions and the temperature of the light-receiving unit 20.

[0085] Because the sensitivity of the IR pixel 21a increases with increasing temperature, as shown in FIG. 10A, for a given amount of received light, the higher the temperature of the light-receiving unit 20, the higher the signal level of the signal output by the IR pixel 21a. Therefore, for example, the drive control unit 40 applies a correction gain, as shown in FIG. 10B, that decreases as the temperature of the light-receiving unit 20 increases to the number of pulsed light emissions or the light emission intensity of the light source 10, adjusted based on the above-described condition table 82. In this case, the condition table 82 used is one that is set based on the relationship between the distance to the recognition target object OBJ and the signal level of the signal output by the IR pixel 21a at a reference temperature Ts. At the reference temperature Ts, the correction gain is 1. For correction gains at temperatures other than the reference temperature Ts, the signal level at the reference temperature Ts is defined as a reference level Ls in the relationship shown in FIG. 10A, and a value obtained by dividing the reference level Ls by the signal level at the temperature in question is used. In this way, by the drive control unit 40 correcting the second irradiation condition based on the temperature of the light receiving unit 20, the signal level of the signal output by the IR pixel 21a can be brought closer to the target signal level even if the temperature of the light receiving unit 20 is different from the reference temperature Ts when the condition table 82 was set.

[0086] If the image recognition unit 30 does not detect the recognition target object OBJ, the drive control unit 40 may adjust the irradiation conditions of the irradiation light based on the output of the IR pixel 21 a based on the reflected light reflected by an object other than the recognition target object OBJ, or may not adjust the irradiation conditions of the irradiation light. Furthermore, if the image recognition unit 30 does not detect the recognition target object OBJ, the drive control unit 40 may stop driving the light source 10 and the light receiving unit 20.

[0087] The drive control unit 40 may also change the adjustment means for adjusting the irradiation conditions of the irradiated light in accordance with the signal level of the third output output from each of the IR pixels 21a based on the amount of received background light excluding reflected light. In this case, the adjustment means for the irradiation conditions may include, for example, the number of times the light source 10 emits pulsed light as the irradiated light, and at least one of the light emission intensity of the light source 10 and the light distribution angle of the irradiated light. The third output includes one or more signals output from the IR pixels 21a and is acquired by the background light measurement unit 60. For example, the background light measurement unit 60 acquires at least one of the signals S2 and S3 output from the IR pixels 21a by performing steps S11 and S12. The signal level of the third output can also be considered a luminance value of the background light at the IR pixels 21a. The background light measurement unit 60 may also acquire the third output output from the IR pixels 21a when the light receiving unit 20 receives background light at a time other than step S12.

[0088] The drive control unit 40 calculates, for example, a representative value of the signal levels of the third outputs output by each of the multiple IR pixels 21 a. The representative value is, for example, an average value or a median value. For example, when the calculated representative value of the signal levels of the third outputs is equal to or greater than a predetermined threshold value, that is, when the amount of background light is large, the drive control unit 40 uses an adjustment means for adjusting the illumination conditions such that the illuminance of the illumination light irradiated on the recognition target object OBJ is higher than that of the other adjustment means.

[0089] For example, when the calculated representative value of the signal level of the third output is equal to or greater than a predetermined threshold and the illumination conditions are adjusted so that the amount of illumination light reaching the target object OBJ is increased compared to the first illumination conditions, the drive control unit 40 adjusts at least one of the light emission intensity of the light source 10 and the light distribution angle of the illumination light as the illumination conditions for the illumination light. The drive control unit 40 may determine whether to adjust the illumination conditions so that the amount of illumination light reaching the target object OBJ is increased compared to the first illumination conditions based on the signal level of the first output or the distance calculated based on the first output. As described with reference to FIG. 6A , the signals S0 and S1 based on the light reflected by the target object OBJ each include signals S0b and S1b, which are components corresponding to background light. Therefore, if the background light component in the signals S0 and S1 increases, the signals S0a and S1a, which are components of the signals S0 and S1 corresponding to the reflected light used to calculate the distance, decrease, resulting in a lower signal-to-noise ratio and reduced ranging accuracy. Therefore, when the number of pulsed light emissions is increased as an irradiation condition of the irradiation light under conditions where the background light component is high, the number of exposures also increases, resulting in an increase in the background light component, making it difficult to effectively improve distance measurement accuracy. In contrast, when the amount of light reaching the recognition target OBJ is increased by adjusting the light emission intensity of the light source 10 or the light distribution angle of the irradiation light, the illuminance of the irradiation light irradiated onto the recognition target OBJ is higher than when the number of pulsed light emissions is increased. In other words, the luminous flux of the irradiation light incident on the surface of the recognition target OBJ is increased. Therefore, when the light emission intensity or the light distribution angle of the irradiation light is adjusted as an irradiation condition of the irradiation light, the amount of reflected light relative to the amount of background light per exposure can be increased. As a result, the background light component in the signals S0 and S1 is reduced, thereby increasing the signal-to-noise ratio and effectively improving distance measurement accuracy.

[0090] On the other hand, when the calculated representative value of the signal level of the third output is equal to or greater than a predetermined threshold value and the irradiation conditions are adjusted so that the amount of irradiation light reaching the recognition target object OBJ is reduced compared to the first irradiation condition, the drive control unit 40 adjusts the number of times that pulsed light is emitted as the irradiation condition for the irradiation light. This prevents a decrease in the amount of reflected light relative to the amount of background light per exposure, thereby preventing a further decrease in the S / N ratio.

[0091] Furthermore, for example, when the calculated representative value of the signal level of the third output is less than a predetermined threshold value, the drive control unit 40 may adjust the irradiation conditions using a predetermined adjustment means regardless of how the irradiation conditions are adjusted, or may adjust the irradiation conditions using the same adjustment means as when the representative value is equal to or greater than the predetermined threshold value.

[0092] 7 , the drive control unit 40 then causes the light source 10 to emit light to a predetermined area under second illumination conditions, which are the illumination conditions adjusted in step S15 (step S16). The light receiving unit 20 receives reflected light from the predetermined area of ​​the light emitted under the second illumination conditions in step S16 and outputs a signal based on the reflected light (step S17). In steps S16 and S17, the drive sequence described in FIGS. 4, 5A, and 5B is performed, and the IR pixel 21 a of the light receiving unit 20 outputs the signals S0 to S3.

[0093] Next, the distance calculation unit 50 calculates the distance to the target object OBJ based on the second output from each of the IR pixels 21a (step S18). The second output is output from each of the IR pixels 21a based on reflected light within a predetermined range of the light emitted under the second irradiation conditions. The second output includes one or more signals output from the IR pixels 21a, such as the signals S0 to S3. The distance calculation unit 50 acquires the second output including the signals S0 to S3 from each of the IR pixels 21a and calculates the distance for each of the IR pixels 21a based on the signals S0 to S3 to generate a distance image. The distance image includes the distances calculated based on the reflected light received by the IR pixels 21a as pixel values ​​of the IR pixels 21a. In this embodiment, the irradiation conditions of the light emitted by the light source 10 are adjusted based on the first output from one or more IR pixels 21a corresponding to the target object OBJ detected by the image recognition unit 30. Therefore, it is possible to appropriately adjust the amount of irradiated light that reaches the recognition target object OBJ, and the accuracy of measuring the distance to the recognition target object OBJ can be improved.

[0094] The distance calculation unit 50 outputs the generated distance image to the outside, for example. The distance image output by the distance calculation unit 50 is input to an external information processing device, for example, and used in applications such as a face authentication system and a driver monitoring system.

[0095] (Others) While the distance measuring device according to one or more aspects of the present disclosure has been described above based on the embodiments, the present disclosure is not limited to the embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to each embodiment and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0096] Furthermore, the distance measuring device according to the present disclosure does not need to include all of the components described in the above embodiments, and may be configured with only the components required to perform the intended operation.

[0097] Furthermore, in the above-described embodiment, the distance measuring device is a distance measuring device that measures distance using an indirect TOF method, but this is not limited thereto. The distance measuring device according to the present disclosure may be a distance measuring device that measures distance using a direct TOF method. Even in a distance measuring device that measures distance using a direct TOF method, the drive control unit can adjust the irradiation conditions for irradiating the irradiation light in the same manner as in the above-described embodiment, thereby adjusting the irradiation conditions to be suitable for the recognition target object.

[0098] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0099] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0100] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0101] For example, the present disclosure may be realized as a ranging device according to the above-described embodiments, as a control device that controls the ranging device, as a ranging method including steps (processing) performed by components that make up the ranging device, as a program for causing a computer to execute such a ranging method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0102] Examples of the distance measuring device and distance measuring method according to the present disclosure that have been described based on the above embodiment are shown below. The distance measuring device and distance measuring method according to the present disclosure are not limited to the following examples.

[0103] For example, a distance measuring device according to a first aspect of the present disclosure includes: a light source that irradiates irradiation light onto a predetermined range; a light receiving unit having a plurality of pixels that receive light reflected from the irradiation light within the predetermined range; an image recognition unit that acquires an image that includes at least a portion of the predetermined range and detects a predetermined object to be recognized by image recognition of the acquired image; a drive control unit that, when the image recognition unit detects the object to be recognized, adjusts the irradiation conditions of the irradiation light to be irradiated by the light source based on a first output output from each of one or more pixels among the plurality of pixels that corresponds to a region of the object to be recognized based on the reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causes the light source to irradiate the irradiation light under the adjusted irradiation conditions; and a distance calculation unit that calculates a distance to the object to be recognized based on a second output output from each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

[0104] This allows the drive control unit to adjust the irradiation conditions for irradiating the irradiation light for distance measurement based on the first output output from one or more pixels corresponding to a predetermined area of ​​the recognition object, thereby making it possible to adjust the irradiation conditions to suit the recognition object and improving the distance measurement accuracy.

[0105] Also, for example, a distance measuring device according to a second aspect of the present disclosure is a distance measuring device according to the first aspect, in which the drive control unit calculates a representative value of the signal level of the first output output by each of the one or more pixels, and adjusts the irradiation conditions of the irradiation light based on the calculated representative value.

[0106] Since the signal level of the first output corresponds to the amount of reflected light received by one or more pixels, it becomes possible to adjust the irradiation conditions with high precision using a representative value of the signal level of the first output.

[0107] Also, for example, a distance measuring device according to a third aspect of the present disclosure is the distance measuring device according to the second aspect, in which the drive control unit adjusts the irradiation conditions of the irradiation light so that the representative value becomes a target signal level.

[0108] This allows the irradiation conditions to be adjusted so that the amount of reflected light received by one or more pixels is a desired amount.

[0109] Also, for example, a distance measuring device according to a fourth aspect of the present disclosure is a distance measuring device according to the second or third aspect, in which the drive control unit calculates the representative value from a signal level of the first output output by each of the one or more pixels that is equal to or greater than a predetermined threshold.

[0110] As a result, even if the area of ​​the detected recognition target object includes a background, the representative value can be calculated by excluding the background portion where the signal level is low from the calculation of the representative value.

[0111] Also, for example, a distance measuring device according to a fifth aspect of the present disclosure is a distance measuring device according to the first aspect, wherein the drive control unit calculates a representative value of the distance to the area of ​​the object to be recognized, which is calculated based on the first output output by each of the one or more pixels, and adjusts the irradiation conditions of the irradiation light based on the calculated representative value.

[0112] Since the amount of light reaching the object to be recognized changes depending on the distance to the object, it is possible to adjust the illumination conditions with high precision using a representative value of the distance.

[0113] Also, for example, a distance measuring device according to a sixth aspect of the present disclosure is a distance measuring device according to the fifth aspect, in which the drive control unit adjusts the irradiation conditions of the irradiation light by referring to a table in which the representative value is associated with the setting values ​​of the irradiation conditions of the irradiation light.

[0114] This simplifies the process because the illumination conditions can be adjusted simply by referring to the table. In addition, since the distance is calculated from the first output output from one or more pixels corresponding to the area of ​​the object to be recognized, the reflectance of the object to be distance-calculated is predetermined, and a table of appropriate illumination condition setting values ​​can be used.

[0115] Also, for example, a distance measuring device according to a seventh aspect of the present disclosure is a distance measuring device according to the fifth or sixth aspect, in which the drive control unit calculates the representative value from a distance to the area of ​​the object to be recognized that is calculated based on the first output output by each of the one or more pixels and that is less than a predetermined threshold.

[0116] As a result, even if the area of ​​the detected recognition target object includes a background, the representative value can be calculated by excluding the background portion from the calculation of the representative value.

[0117] Also, for example, a distance measuring device according to an eighth aspect of the present disclosure is a distance measuring device according to any one of the fifth to seventh aspects, further comprising a temperature sensor that measures the temperature of the light receiving unit, and the drive control unit corrects the adjusted irradiation conditions based on the temperature of the light receiving unit measured by the temperature sensor.

[0118] As a result, even if the sensitivity of the light receiving section changes depending on the temperature, the influence of the sensitivity can be suppressed by correcting the irradiation conditions after adjustment.

[0119] Furthermore, for example, a distance measuring device according to a ninth aspect of the present disclosure is a distance measuring device according to any one of the first to eighth aspects, wherein the one or more pixels corresponding to the area of ​​the object to be recognized are pixels that receive light from a central area of ​​an area obtained by dividing the area of ​​the object to be recognized or from an area closest to the object to be recognized.

[0120] This makes it possible to use the first output from the pixel that is more suitable for adjusting the illumination conditions within the region of the recognition object.

[0121] Also, for example, a distance measuring device according to a tenth aspect of the present disclosure is a distance measuring device according to any one of the first to ninth aspects, further comprising a background light measuring unit that acquires a third output output by each of the plurality of pixels based on the amount of background light received that does not include the reflected light, and the drive control unit changes the means for adjusting the irradiation conditions of the irradiation light according to the signal level of the third output.

[0122] This allows for the use of an illumination condition adjustment means that reduces the influence of background light depending on the amount of background light. For example, when the amount of background light is high, an adjustment means can be used that increases the illuminance of the illumination light irradiated onto the recognition target compared to other adjustment means.

[0123] Also, for example, a distance measuring device according to an eleventh aspect of the present disclosure is a distance measuring device according to any one of the first to tenth aspects, wherein the light source irradiates pulsed light multiple times as the irradiation light, and the drive control unit adjusts the irradiation conditions of the irradiation light by adjusting at least the number of times the pulsed light is emitted.

[0124] This allows the irradiation conditions to be adjusted by the number of times the light source emits pulsed light, making it easier to adjust the irradiation conditions.

[0125] Also, for example, a distance measuring device according to a twelfth aspect of the present disclosure is a distance measuring device according to any one of the first to eleventh aspects, wherein the light source is capable of changing the light emission intensity of the light source, and the drive control unit adjusts the irradiation conditions of the irradiated light by adjusting at least the light emission intensity.

[0126] This makes it possible to adjust the irradiation conditions without changing the irradiation time of the irradiation light.

[0127] Also, for example, a distance measuring device according to a thirteenth aspect of the present disclosure is a distance measuring device according to any one of the first to twelfth aspects, wherein the light source is capable of changing the light distribution angle of the irradiation light, and the drive control unit adjusts the irradiation conditions of the irradiation light by adjusting at least the light distribution angle.

[0128] This allows the amount of light reaching the object to be recognized to be adjusted even if the amount of light emitted from the light source remains the same. For example, even if the amount of light reaching the object to be recognized is increased, there is no need to increase the amount of light emitted from the light source, which results in energy savings.

[0129] Furthermore, for example, a distance measuring method according to a fourteenth aspect of the present disclosure is a distance measuring method using a distance measuring device, the distance measuring device including a light source that irradiates irradiation light onto a predetermined range, and a light receiving unit having a plurality of pixels that receive reflected light of the irradiation light reflected within the predetermined range, the distance measuring method including an image recognition step of acquiring an image that includes at least a part of the predetermined range and detecting a predetermined object to be recognized by image recognition of the acquired image, a drive control step of, when the object to be recognized is detected in the image recognition step, adjusting the irradiation conditions of the irradiation light to be irradiated by the light source based on first outputs output by each of one or more pixels among the plurality of pixels that correspond to a region of the object to be recognized based on reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causing the light source to irradiate the irradiation light under the adjusted irradiation conditions, and a distance calculation step of calculating a distance to the object to be recognized based on second outputs output by each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

[0130] According to this distance measuring method, like the distance measuring device according to the first aspect, it is possible to adjust the illumination conditions to suit the object to be recognized, thereby improving the distance measuring accuracy.

[0131] The distance measuring device and the like according to the present disclosure can be applied to a variety of applications, such as distance measurement systems, sensing systems using distance images, and authentication systems.

[0132] 10 Light source 20 Light receiving unit 20A, 20A1, 20A2, 20A3, 20A4, 20A5, 20A6 Imaging element 21 Pixel 21a IR pixel 21b R pixel 21c G pixel 21d B pixel 21e BW pixel 30 Image recognition unit 35 Image 36 Area 40 Drive control unit 50 Distance calculation unit 60 Background light measurement unit 70 Temperature sensor 80 Storage unit 81 Image recognition model 82 Condition table 100 Distance measurement device OBJ Recognition target object

Claims

1. A distance measuring device comprising: a light source that irradiates irradiation light onto a predetermined range; a light receiving unit having a plurality of pixels that receive light reflected from the irradiation light within the predetermined range; an image recognition unit that acquires an image that includes at least a portion of the predetermined range and detects a predetermined object to be recognized by image recognition of the acquired image; a drive control unit that, when the image recognition unit detects the object to be recognized, adjusts the irradiation conditions of the irradiation light to be irradiated by the light source based on a first output output from each of the plurality of pixels that corresponds to an area of ​​the object to be recognized based on the reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causes the light source to irradiate the irradiation light under the adjusted irradiation conditions; and a distance calculation unit that calculates a distance to the object to be recognized based on a second output output from each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

2. The distance measuring device according to claim 1, wherein the drive control unit calculates a representative value of the signal level of the first output output by each of the one or more pixels, and adjusts the irradiation conditions of the irradiation light based on the calculated representative value.

3. The distance measuring device according to claim 2, wherein the drive control unit adjusts the irradiation conditions of the irradiation light so that the representative value becomes a target signal level.

4. The distance measuring device according to claim 2 or 3, wherein the drive control unit calculates the representative value from a signal level equal to or higher than a predetermined threshold, among the signal levels of the first output outputted by each of the one or more pixels.

5. The distance measuring device according to claim 1, wherein the drive control unit calculates a representative value of the distance to the region of the object to be recognized based on the first output output by each of the one or more pixels, and adjusts the irradiation conditions of the irradiation light based on the calculated representative value.

6. The distance measuring device according to claim 5, wherein the drive control unit adjusts the irradiation conditions of the irradiation light by referring to a table in which the representative value corresponds to a setting value of the irradiation conditions of the irradiation light.

7. The distance measuring device according to claim 5 or 6, wherein the drive control unit calculates the representative value from a distance to the area of ​​the object to be recognized that is calculated based on the first output output by each of the one or more pixels and that is equal to or less than a predetermined threshold value.

8. A distance measuring device as described in any one of claims 5 to 7, further comprising a temperature sensor that measures the temperature of the light receiving unit, and the drive control unit corrects the adjusted irradiation conditions based on the temperature of the light receiving unit measured by the temperature sensor.

9. A distance measuring device as described in any one of claims 1 to 8, wherein the one or more pixels corresponding to the area of ​​the object to be recognized are pixels that receive light from a central area of ​​an area into which the area of ​​the object to be recognized is divided or from an area closest to the object to be recognized.

10. A distance measuring device as described in any one of claims 1 to 9, further comprising a background light measuring unit that acquires a third output output by each of the plurality of pixels based on the amount of background light received that does not include the reflected light, and the drive control unit changes the adjustment means for the irradiation conditions of the irradiated light depending on the signal level of the third output.

11. A distance measuring device as claimed in any one of claims 1 to 10, wherein the light source irradiates pulsed light as the irradiation light multiple times, and the drive control unit adjusts the irradiation conditions of the irradiation light by adjusting at least the number of times the pulsed light is emitted.

12. A distance measuring device according to any one of claims 1 to 11, wherein the light source is capable of changing the light emission intensity of the light source, and the drive control unit adjusts the irradiation conditions of the irradiated light by adjusting at least the light emission intensity.

13. A distance measuring device as described in any one of claims 1 to 12, wherein the light source is capable of changing a light distribution angle of the irradiated light, and the drive control unit adjusts the irradiation conditions of the irradiated light by adjusting at least the light distribution angle.

14. A distance measuring method using a distance measuring device, the distance measuring device comprising: a light source that irradiates irradiation light onto a predetermined range; and a light receiving unit having a plurality of pixels that receive light reflected from the irradiation light within the predetermined range, the distance measuring method comprising: an image recognition step of acquiring an image including at least a portion of the predetermined range, and detecting a predetermined object to be recognized by image recognition of the acquired image; a drive control step of adjusting, when the object to be recognized is detected in the image recognition step, the irradiation conditions of the irradiation light to be irradiated by the light source based on a first output output from each of the plurality of pixels corresponding to an area of ​​the object to be recognized based on the reflected light of the irradiation light irradiated under predetermined irradiation conditions, and causing the light source to irradiate the irradiation light under the adjusted irradiation conditions; and a distance calculation step of calculating a distance to the object to be recognized based on a second output output from each of the plurality of pixels based on the reflected light of the irradiation light irradiated under the adjusted irradiation conditions.

Images