Distance measuring device and distance measuring method

Abstract

A distance measuring device (100) is provided with: a light source (10) that irradiates irradiation light to a predetermined range; a light receiving unit (20) having a plurality of pixels that receive reflected light reflected by the irradiation light within a predetermined range; an image recognition unit (30) that acquires an image in which at least a portion of a predetermined range is projected, and detects a predetermined object to be recognized (OBJ) by performing image recognition on the acquired image; a drive control unit (40) that, when the image recognition unit (30) detects the object to be recognized (OBJ), adjusts an irradiation condition in which the light source (10) irradiates the irradiation light on the basis of a first output that is a signal output from each of one or more pixels corresponding to the region of the object to be recognized (OBJ) on the basis of reflected light of the irradiation light irradiated under a predetermined irradiation condition; and a distance calculation unit (50) that calculates the distance to the object to be recognized (OBJ) on the basis of a second output that is a signal output by each of the plurality of pixels on the basis of reflected light of the irradiation light irradiated under the adjusted irradiation condition.

Description

Technical Field

[0001] This disclosure relates to a ranging device and a ranging method. Background Technology

[0002] Previously, ranging devices employing indirect Time of Flight (ToF) methods were known. These indirect ToF ranging devices typically include a light source and a light-receiving unit. Such devices receive reflected light from an object illuminated by the light source via the light-receiving unit and generate a distance image based on the signal output from the light-receiving unit based on the reflected light.

[0003] In ranging devices employing the indirect Time-of-Flight (TOF) method, when the amount of illumination light reaching the object is low, the signal level output by the light-receiving unit decreases, leading to a reduction in ranging accuracy due to the decreased signal-to-noise ratio (SN ratio). Furthermore, when the amount of illumination light reaching the object is excessive, the signal level output by the light-receiving unit saturates, making accurate ranging impossible. Therefore, to improve ranging accuracy, techniques for adjusting the amount of illumination light emitted by the light source of the ranging device have been proposed (for example, see Patent Document 1).

[0004] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2008-241435 Summary of the Invention

[0005] The problem that the invention aims to solve There is still a need to further improve ranging accuracy in the existing technology.

[0006] This disclosure provides a ranging device and a ranging method that can improve ranging accuracy.

[0007] Methods for solving problems One aspect of the ranging device disclosed herein includes: a light source that illuminates a predetermined range with illumination light; a light-receiving unit having a plurality of pixels that receive reflected light from the illumination light within the predetermined range; an image recognition unit that acquires an image covering at least a portion of the predetermined range and detects a predetermined object to be identified by performing image recognition on the acquired image; a drive control unit that, when the image recognition unit detects the object to be identified, adjusts the illumination conditions of the illumination light illuminating the illumination light based on a first output, and causes the light source to illuminate the illumination light with the adjusted illumination conditions, wherein the first output is a signal output by one or more pixels among the plurality of pixels corresponding to the area of ​​the object to be identified, based on the reflected light of the illumination light illuminated under the predetermined illumination conditions; and a distance calculation unit that calculates the distance to the object to be identified based on a second output, wherein the second output is a signal output by each of the plurality of pixels based on the reflected light of the illumination light illuminated under the adjusted illumination conditions.

[0008] One aspect of this disclosure relates to a ranging method performed by a ranging device comprising: a light source that illuminates a predetermined range with illumination light; and a light-receiving unit having a plurality of pixels that receive reflected light from the illumination light within the predetermined range. The ranging method includes: an image recognition step of acquiring an image that reflects at least a portion of the predetermined range and detecting a predetermined object to be identified by performing image recognition on the acquired image; a drive control step of adjusting the illumination conditions of the illumination light from the light source based on a first output when the object to be identified is detected in the image recognition step, and causing the light source to illuminate the illumination light with the adjusted illumination conditions, wherein the first output is a signal output by one or more pixels from the plurality of pixels corresponding to the region of the object to be identified, based on the reflected light from the illumination light illuminated under the predetermined illumination conditions; and a distance calculation step of calculating the distance to the object to be identified based on a second output, wherein the second output is a signal output by each of the plurality of pixels based on the reflected light from the illumination light illuminated under the adjusted illumination conditions.

[0009] Invention Effects This disclosure can improve ranging accuracy. Attached Figure Description

[0010] Figure 1 This is a functional block diagram illustrating an example of the structure of the ranging device involved in the implementation.

[0011] Figure 2 This is a schematic diagram of the light-receiving part of the ranging device according to the embodiment.

[0012] Figure 3A This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0013] Figure 3B This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0014] Figure 3C This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0015] Figure 3D This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0016] Figure 3E This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0017] Figure 3F This is a schematic diagram illustrating an example of the arrangement of pixels in the imaging element included in the light-receiving section according to the embodiment.

[0018] Figure 4 This is a diagram illustrating an example of the driving sequence of the ranging device involved in the embodiment.

[0019] Figure 5A This is a diagram illustrating an example of the timing of the emission control pulse and exposure control pulse during the A0 / A1 period.

[0020] Figure 5B This is a diagram illustrating an example of the timing of the emission control pulse and exposure control pulse during A2 / A3.

[0021] Figure 6A This is a diagram illustrating an example of a signal based on the charge generated during exposure in the A0 / A1 period.

[0022] Figure 6B This is a diagram illustrating an example of a signal based on the charge generated during exposure in A2 / A3.

[0023] Figure 7 This is a flowchart illustrating an example of the operation of the ranging device involved in the implementation method.

[0024] Figure 8 This diagram illustrates a specific example of the detection of an object to be identified performed by the image recognition unit according to the embodiment.

[0025] Figure 9It is a diagram used to illustrate the condition table for adjusting the illumination conditions of the illumination light.

[0026] Figure 10 This is a diagram illustrating a method for correcting the second irradiation conditions based on the temperature of the light-receiving part. Detailed Implementation

[0027] (The process of obtaining this disclosure) As described above, in distance measuring devices employing the indirect TOF method, the ranging accuracy decreases when the amount of illumination light reaching the object is inappropriate. For example, since the amount of illumination light reaching the object decreases proportionally to the square of the distance between the light source and the object, the appropriate amount of illumination light varies depending on the distance between the light source and the object.

[0028] Patent Document 1 discloses a technique for adjusting the amount of light emitted by a light source in a ranging device to improve ranging accuracy. Specifically, in the technique disclosed in Patent Document 1, the light-receiving section receives reflected light from an object and converts it into charge, and the amount of light emitted by the light source is adjusted based on the charge stored in the charge storage section. However, in the technique disclosed in Patent Document 1, since it is unclear whether the reflected light from the object is reflected light from the object to be used for ranging, there is a possibility that the amount of light emitted by the light source cannot be adjusted appropriately. For example, if there are multiple objects within the range of the light emitted by the light source, there is a possibility that reflected light from objects other than the objects to be used for ranging may be used to adjust the amount of light emitted by the light source. Furthermore, since the amount of reflected light received by the light-receiving section from the object varies depending on the reflectivity of the object, if reflected light from an object with a different reflectivity than the object to be used for ranging is used to adjust the amount of light emitted by the light source, there is a possibility that the amount of light emitted by the object to be used for ranging may not be properly adjusted for the object to be used for ranging.

[0029] This disclosure is made from the perspective of the inventors of this application as described above, and provides a ranging device and a ranging method that can improve ranging accuracy by more appropriately adjusting the illumination conditions of the illumination light illuminating the light source.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0031] Furthermore, the embodiments described below are all general or specific examples. The numerical values, shapes, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Additionally, in the constituent elements of this embodiment described below, constituent elements not described in the independent technical solutions are explained as arbitrary constituent elements. Furthermore, the figures are schematic diagrams and are not necessarily rigorous illustrations. Also, in the figures, substantially identical structures are labeled with the same symbols, and repeated descriptions are omitted or simplified.

[0032] (Implementation Method) [constitute] First, the structure of the ranging device involved in this embodiment will be explained. Figure 1 This is a functional block diagram illustrating an example of the structure of the ranging device 100 according to this embodiment. Figure 2 This is a schematic diagram of the light-receiving unit 20 of the ranging device 100 according to this embodiment.

[0033] The ranging device 100 is a ranging device that measures distance using an indirect Time-of-Flight (TOF) method. The ranging device 100 generates a distance image representing the distance to an object contained within the camera's field of view. In the ranging device 100 of this embodiment, since ranging accuracy is improved, it can reduce the number of retries during authentication in image authentication systems such as face authentication systems and driver monitoring systems, thereby achieving high-performance authentication processing. Therefore, the ranging device 100 can be used in image authentication systems such as face authentication systems and driver monitoring systems.

[0034] like Figure 1 As shown, the ranging device 100 includes a light source 10, a light receiving unit 20, an image recognition unit 30, a drive control unit 40, a distance calculation unit 50, a background light measuring unit 60, a temperature sensor 70, and a storage unit 80.

[0035] The light source 10 illuminates a predetermined area (the object space of the ranging object) including at least a portion of the imaging range of the light-receiving unit 20, according to an input light emission control signal. The illumination conditions of the light from the light source 10 are controlled by the input light emission control signal. For example, the light source 10 illuminates the predetermined area multiple times with a predetermined pulse width according to the timing indicated by the light emission control pulses included in the input light emission control signal. Figure 1 The diagram schematically illustrates a case where a predetermined object OBJ for identification exists within a defined range. The light source 10 includes, for example, a light-emitting element such as a light-emitting diode or a laser element that emits infrared light (IR), and an optical system that receives light incident from the light-emitting element and controls the light distribution from the light-emitting element.

[0036] The light source 10 can be used as an illumination condition to change the luminous intensity of the light source 10. Known dimming methods can be used to change the luminous intensity of the light source 10. The light source 10 may include multiple light-emitting elements, and the luminous intensity can be changed by varying the number of light-emitting elements. The light source 10 can also change its luminous intensity by changing the voltage input to the light-emitting elements.

[0037] Furthermore, the light source 10 can also change the light distribution angle of the illumination light as an illumination condition. The light source 10 may include multiple sub-light sources with different light distribution angles, and the light distribution angle of the illumination light can be changed by switching between the sub-light sources that emit light. Additionally, the light source 10 can also change the light distribution angle of the illumination light by switching the optical system through which light from the light-emitting element is incident. Furthermore, the light source 10 can combine multiple optical systems through which light from the light-emitting element is incident, and adjust the spacing between the multiple optical systems to continuously change the light distribution angle. Even when the light source 10 emits light with the same luminous intensity, the narrower the light distribution angle of the illumination light, the higher the luminous intensity of the illumination light, and the higher the illuminance of the illumination light illuminating the object OBJ to be identified. In other words, by changing the light distribution angle, the amount of illumination light reaching the object OBJ can be increased without increasing the luminous intensity and amount of light emitted by the light-emitting element of the light source 10.

[0038] In the light source 10, the objects controlled by the illumination conditions of the illumination light controlled by the input light emission control signal include, for example, at least one of the number of pulse light emission, the light emission intensity of the light source 10, and the light distribution angle of the illumination light.

[0039] like Figure 2 As shown, the light-receiving unit 20 includes one or more imaging elements 20A. The imaging element 20A is an imaging element that receives light from a defined range and captures an image of at least a portion of the defined range. The imaging element 20A is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging element 20A has a plurality of pixels 21 arranged in a two-dimensional pattern. Figure 2 For illustrative purposes, a structure of 16 pixels (4 pixels horizontally and 4 pixels vertically) is used, but there is no particular limitation on the number of pixels 21 in the image sensor 20A. The number of pixels 21 in the image sensor 20A can be, for example, more than 20,000 pixels and less than 5 million pixels.

[0040] Each of the plurality of pixels 21 includes at least one photoelectric conversion element, which receives reflected light from illumination light emanating from light source 10 within a specified range at the object to be identified, such as OBJ, and converts it into electrical charge. A photoelectric conversion element, such as a photodiode, is used as an example.

[0041] Each of the multiple pixels 21 outputs a signal based on the charge obtained from exposure according to the exposure control signal. For example, the imaging element 20A, for each pixel 21, transfers the charge obtained from exposure according to the exposure control signal to the vertical transfer path (VCCD) or charge storage section (FD), and reads out the signal based on the charge obtained from exposure. The imaging element 20A, for example, performs AD (Analog-to-Digital) conversion during signal readout and outputs a digital signal that has undergone AD conversion. The exposure period in the imaging element 20A corresponds, for example, to the emission period of the illumination light from the light source 10. Furthermore, in the case where the light source 10 illuminates pulsed light multiple times, an exposure period corresponding to the number of times the pulsed light is emitted is set.

[0042] Here, an example of the configuration of pixels 21 in the imaging element 20A is explained. Figures 3A to 3F This is a schematic diagram illustrating an example of the pixel arrangement in the imaging element included in the light-receiving section 20 according to this embodiment.

[0043] The light-receiving portion 20 includes, for example, a plurality of imaging elements 20A. In this case, the light-receiving portion 20 includes, for example, a plurality of imaging elements 20A. Figure 3A The image sensor 20A1 shown, and Figure 3B and Figure 3C At least one of the image sensors 20A2 and 20A3 shown.

[0044] like Figure 3A As shown, in the imaging element 20A1, multiple pixels 21 are each composed of IR pixels 21a, each sensitive to infrared light. The multiple IR pixels 21a are arranged in a matrix in the imaging element 20A1. The multiple IR pixels 21a receive reflected light from the illumination light from the light source 10 and output signals based on the received reflected light. The signals output by the IR pixels 21a are used to generate IR images and distance images.

[0045] like Figure 3B As shown, the imaging element 20A2 has an R pixel 21b sensitive to red light, a G pixel 21c sensitive to green light, and a B pixel 21d sensitive to blue light, serving as a plurality of pixels 21. In the imaging element 20A2, the R pixel 21b, G pixel 21c, and B pixel 21d are arranged in a Bayer configuration. The signals output from the R pixel 21b, G pixel 21c, and B pixel 21d are used to generate a color visible light image (RGB image).

[0046] like Figure 3C As shown, in the imaging element 20A3, multiple pixels 21 are each composed of BW pixels 21e, each sensitive to visible light as a whole. The multiple BW pixels 21e are arranged in a matrix in the imaging element 20A3. The signals output by the BW pixels 21e are used to generate a black-and-white visible light image (BW image).

[0047] Furthermore, the light-receiving unit 20 may include only one imaging element 20A. In this case, the light-receiving unit 20 includes... Figure 3D The image sensor shown is 20A4, or Figure 3E The image sensor shown is 20A5 or Figure 3F The image sensor 20A6 shown is used as the image sensor 20A.

[0048] like Figure 3D As shown, the imaging element 20A4 has IR pixels 21a, R pixels 21b, G pixels 21c, and B pixels 21d as a plurality of pixels 21. The imaging element 20A4 has a pixel arrangement after replacing half of the G pixels 21c of the imaging element 20A2 with IR pixels 21a.

[0049] like Figure 3E and Figure 3F As shown, imaging elements 20A5 and 20A6 each have an IR pixel 21a and a BW pixel 21e, serving as a plurality of pixels 21. In imaging element 20A5, adjacent pixels in either the row or column direction are IR pixels 21a and BW pixels 21e. In imaging element 20A6, adjacent pixels in the column direction are IR pixels 21a and BW pixels 21e, and in the row direction, IR pixels 21a are arranged next to each other or BW pixels 21e are arranged next to each other.

[0050] With the above-described structure, the light-receiving unit 20 can generate IR images, distance images, and visible light images using the signals output from it. Furthermore, generating visible light images using the signals output from the light-receiving unit 20 is not mandatory; the light-receiving unit 20 can also be composed of a single imaging element 20A1 if a visible light image is not generated.

[0051] Refer again Figure 1The image recognition unit 30 acquires an image of at least a portion of a predetermined range of illumination light from the light source 10, and performs image recognition on the acquired image. The image recognition unit 30 detects a predetermined object OBJ by performing image recognition on the image of at least a portion of the predetermined range. Known methods can be used for image recognition performed by the image recognition unit 30. For example, the image recognition unit 30 uses a learned image recognition model 81 that has undergone machine learning to detect the object OBJ from the image, and outputs the location of the region reflecting the object OBJ. The image recognition unit 30 extracts features from the image and inputs the extracted features into the image recognition model 81. The image recognition model 81 is stored in the storage unit 80, for example, as an image recognition model that has undergone machine learning using learned images, and outputs the probability that the predetermined object OBJ exists in the image after receiving the feature input. The image recognition unit 30 determines that the object OBJ exists if the probability output by the image recognition model 81 is above a predetermined threshold. For example, a neural network model can be used as the image recognition model 81, but there are no particular limitations.

[0052] There are no particular restrictions on the types of object OBJs to be identified, and the types can be set according to the application using the ranging device 100. Examples of object OBJs to be identified include human faces, human hands, the entire person, moving objects such as vehicles, buildings, and animals. Furthermore, there are no particular restrictions on the image recognition method performed by the image recognition unit 30. For example, if the object OBJ to be identified is a human face, a known face detection algorithm can be used to detect the human face. Moreover, the image recognition unit 30 can detect multiple types of object OBJs in image recognition. For example, the image recognition unit 30 can also detect people and vehicles as object OBJs.

[0053] The drive control unit 40 outputs various control signals for controlling the driving of the light source 10 and the light-receiving unit 20. For example, the drive control unit 40 outputs a light emission control signal instructing the light source 10 to irradiate the light with a predetermined pulse width, serving as a control signal for controlling the driving of the light source 10. Furthermore, the drive control unit 40 outputs exposure control signals instructing each pixel 21 of the light-receiving unit 20 to be exposed, serving as a control signal for controlling the driving of the light-receiving unit 20. Based on the output of the light-receiving unit 20 and the detection results of the identified object OBJ performed by the image recognition unit 30, the drive control unit 40 adjusts the irradiation conditions of the light source 10 irradiating the light.

[0054] The distance calculation unit 50 performs a predetermined calculation based on the signals output by each of the pixels 21 (specifically, IR pixels 21a) that have received reflected light within a predetermined range of the illumination light, thereby generating a distance image. For example, the distance calculation unit 50 calculates the distance from each pixel 21 to the identified object OBJ based on the signals output by each pixel 21. The distance calculation unit 50 outputs the distance calculated for each pixel 21 as a pixel value.

[0055] The background light measuring unit 60 acquires the signal output by the pixel 21 of the light receiving unit 20 based on the background light, which does not include reflected light from the illumination light irradiated by the light source 10. Alternatively, the ranging device 100 may not include the background light measuring unit 60.

[0056] Furthermore, the drive control unit 40, distance calculation unit 50, and background light measurement unit 60 are processing circuits implemented, for example, by a memory storing a program and a processor executing that program. This program can be stored in the memory unit 80. Moreover, although they are separate in the block diagram, all or part of the drive control unit 40, distance calculation unit 50, and background light measurement unit 60 can be configured using the same memory and processor. Details of the processing performed by the drive control unit 40, distance calculation unit 50, and background light measurement unit 60 will be described later.

[0057] Temperature sensor 70 measures the temperature of light-receiving part 20 (specifically, imaging element 20A). Temperature sensor 70 outputs the measured temperature of light-receiving part 20 to drive control unit 40. Temperature sensor 70 can directly measure the temperature of light-receiving part 20, or indirectly measure the temperature of light-receiving part 20 by measuring the temperature of its surrounding environment. Alternatively, the ranging device 100 may not include temperature sensor 70.

[0058] The storage unit 80 is a storage device that stores information and data required for processing by the ranging device 100. For example, the storage unit 80 stores an image recognition model 81 for image recognition and a condition table 82 for determining the illumination conditions of the light source 10. The storage unit 80 is implemented, for example, using a semiconductor memory or an HDD (Hard Disk Drive). Alternatively, at least a portion of the storage unit 80 may be located in a device different from the ranging device 100, and the ranging device 100 may access the data stored in the storage unit 80 via a network such as the Internet.

[0059] [Driving sequence of the ranging device] Next, the driving of the light source 10 and the light receiving unit 20 when the ranging device 100 is measuring distance will be explained. Figure 4 This is a diagram illustrating an example of the drive sequence of the ranging device 100 according to this embodiment. Figure 4The diagram shows the driving sequence when the light-receiving unit 20 outputs a signal for the distance calculation unit 50 to the object to be identified, such as the object OBJ.

[0060] like Figure 4 As shown, in the driving sequence when the ranging device 100 performs ranging, one frame consists of a light emission exposure period and a readout period. First, during the light emission exposure period, the illumination light from the light source 10 is applied, and the IR pixel 21a of the light-receiving section 20 is exposed. Then, during the readout period, a signal based on the charge generated in the IR pixel 21a during the light emission exposure period is read out.

[0061] During the emission exposure, the same emission and exposure sequence is repeated β times. Figure 4 In the example shown, β is 8 times, and the same emission and exposure sequence as during β1 is repeated from β2 to β8. Figure 4 In the example shown, the β1 period is divided into the A0 / A1 period and the A2 / A3 period. During the A0 / A1 period, a light emission control signal containing an α-th pulse is output from the drive control unit 40 to the light source 10, and an exposure control signal containing an α×2 pulse is output from the drive control unit 40 to the light receiving unit 20. Furthermore, during the A2 / A3 period, no light emission control pulse is output from the drive control unit 40 to the light source 10, and an exposure control signal containing an α×2 pulse is output from the drive control unit 40 to the light receiving unit 20. As a result, during the light emission exposure period, the light source 10 illuminates the α×β pulse light as the illumination light.

[0062] The emission control pulse is a control pulse that instructs the light source 10 to emit pulsed light; when it is high, the light source 10 emits illumination light. The exposure control pulse is a control pulse that instructs the exposure of the IR pixel 21a; when it is low, the IR pixel 21a is exposed. During A0 / A1, the emission period of the light source 10 and the exposure period of the IR pixel 21a are correlated in a manner that establishes a predetermined phase difference.

[0063] Next, use Figure 5A and Figure 5B This section provides a detailed explanation of the luminescence control pulse and the exposure control pulse. Figure 5A This is a diagram illustrating an example of the timing of the emission control pulse and exposure control pulse during the A0 / A1 period. Figure 5B This diagram illustrates an example of the timing of the emission control pulse and exposure control pulse during A2 / A3. Figure 5A and Figure 5B The diagram shows the repetition units of the emission control pulse and the exposure control pulse during A0 / A1. Figure 5A The sequence shown is performed α times, during A2 / A3. Figure 5BThe sequence shown is performed α times.

[0064] like Figure 5A As shown, during A0 / A1, corresponding to the output of one emission control pulse, two exposure control pulses A0 and A1 are output. The drive control unit 40 outputs an emission control pulse with a pulse width Tp. Thus, the light source 10 illuminates pulsed light with a pulse width Tp. Furthermore, the drive control unit 40 outputs exposure control pulses A0 and A1 with different start timings based on the emission control pulse. Figure 5A In the example shown, the exposure widths (exposure periods) of exposure control pulses A0 and A1 are the same as the pulse width Tp of the illumination light, and they do not overlap. Exposure control pulse A0 starts, for example, at the same timing as the start timing of the emission control pulse. Exposure control pulse A1 starts, for example, at the end timing of the emission control pulse and exposure control pulse A0. That is, the phase difference between exposure control pulse A0 and exposure control pulse A1 is the pulse width Tp of the illumination light. Alternatively, as long as exposure control pulses A0 and A1 are output with a phase difference, exposure control pulse A0 may not start at the same timing as the start timing of the emission control pulse. Exposure control pulse A0 may also start, for example, with a predetermined offset from the start timing of the emission control pulse.

[0065] like Figure 5B As shown, during A2 / A3, no light emission control pulse is output, but two exposure control pulses A2 and A3 are output. The drive control unit 40 outputs exposure control pulses A2 and A3 with different start timings. Figure 5B In the example shown, the exposure widths (exposure periods) of exposure control pulses A2 and A3 are the same as the pulse width Tp of the illumination light, and they do not overlap. Exposure control pulse A3 starts, for example, at the end timing of exposure control pulse A2. Alternatively, during A2 / A3, exposure control pulse A3 may not be output, and only exposure control pulse A2 may be output.

[0066] The IR pixel 21a, for example, has multiple charge storage sections, and the charge generated during the exposure of each of the multiple exposure control pulses A0 to A3 is distributed and stored in different charge storage sections. For example, the number of multiple exposure control pulses A0 to A3 is the same as the number of multiple charge storage sections of the IR pixel 21a. During the light emission exposure, the light source 10 illuminates α×β pulse light as illumination light, and the charge generated by the α×β exposures is stored in each charge storage section. During the readout period, a signal based on the charge stored in each charge storage section by the α×β exposures is read out. That is, the IR pixel 21a outputs a signal corresponding to the amount of charge generated by the exposure of each of the α×β exposure control pulses A0 to A3.

[0067] The drive control unit 40 can change the number of times the pulsed light is emitted by changing the number of output emission control pulses. For example, when the illumination conditions of the light source 10 are changed, the drive control unit 40 changes α. Since the drive control unit 40 outputs exposure control pulses A0 to A3 with the same number of output emission control pulses, when the number of output emission control pulses is changed, the number of output exposure control pulses A0 to A3 is also changed to the same number.

[0068] The distance calculation unit 50 calculates the distance to the object being identified, such as the object OBJ, which is illuminated by the pulsed light, based on the signals output from each IR pixel 21a via the aforementioned driving sequence. The distance calculation unit 50 generates a distance image by calculating the distance to each IR pixel 21a. The distance calculation performed by the distance calculation unit 50 (e.g., the distance from the ranging device 100 to the object being identified, OBJ) is performed, for example, in the following manner.

[0069] Figure 6A This is a diagram illustrating an example of a signal based on the charge generated during exposure in A0 / A1. Figure 6B This is a diagram illustrating an example of a signal based on the charge generated during exposure through A2 / A3. Figure 6A and Figure 6B In the diagram, the signal of charge generated based on exposure according to each exposure control pulse is schematically represented by patterned rectangles. The area of ​​the rectangle corresponds to the magnitude of the signal. Furthermore, in... Figure 6A and Figure 6B The diagram shows signals S0 to S3 corresponding to the charge accumulated in one exposure. However, in reality, each of the multiple IR pixels 21a outputs signals S0 to S3 based on the charge accumulated in α×β exposures according to the aforementioned driving sequence.

[0070] like Figure 6AAs shown, the reflected light from the illumination light from the light source 10, reflected at the object OBJ, returns to the ranging device 100 after a delay of time Δt from the illumination of the light source, and is incident on the light receiving unit 20. A portion of the reflected light is received by the IR pixel 21a and converted into charge during exposure according to the exposure control pulse A0. Furthermore, since the reflected light returns after a delay of time Δt from the illumination of the light source, the remaining portion of the reflected light corresponding to the delay time Δt is received by the IR pixel 21a and converted into charge during exposure according to the exposure control pulse A1. In addition, the background light, which does not contain reflected light, is received by the IR pixel 21a and converted into charge during the respective exposures performed according to the exposure control pulses A0 and A1. Therefore, the signal S0 based on the charge generated by the exposure performed according to the exposure control pulse A0 is a signal containing the following components: a component of signal S0a corresponding to the reflected light after removing time Δt from the pulse width Tp, and a component of signal S0b corresponding to the background light. In addition, the signal S1 based on the charge generated by exposure according to the exposure control pulse A1 is a signal containing the following components: the component of signal S1a corresponding to the time width of the reflected light with time Δt, and the component of signal S1b corresponding to the background light.

[0071] In addition, such as Figure 6B As shown, during A2 / A3, since the light source 10 does not illuminate the illumination light, only the background light, which does not contain reflected light, is received by the IR pixel 21a and converted into charge during the respective exposures performed according to the exposure control pulses A2 and A3. Therefore, signals S2 and S3, based on the charges generated during the exposures performed according to the exposure control pulses A2 and A3, are signals corresponding to the background light. Therefore, the magnitudes of signals S0b, S1b, S2, and S3 are substantially the same. Thus, by subtracting signal S2 or S3 from signals S0 and S1 respectively, the influence of the background light can be removed. Furthermore, the sum of signals S0 and S1 after removing the influence of the background light becomes the signal of reflected light based on the time width Tp of the pulse width. Therefore, the time Δt is calculated by the following formula.

[0072] Δt=Tp×(S1-S3) / [(S0-S2)+(S1-S3)] As a result, if the distance from the ranging device 100 to the identified object OBJ is set as D, and the speed of light is set as c, then the distance D is calculated by the following formula, since the illumination light from the light source 10 travels a round trip distance D in time Δt. Alternatively, the distance D can be described as the individual flight distances of the illumination light and the reflected light.

[0073] D=c×Δt / 2 =(c×Tp / 2)×(S1-S3) / [(S0-S2)+(S1-S3)] Since the distance D is calculated as described above, if the amount of reflected light is low, the signal levels of signals S0 and S1 based on the reflected light are low, resulting in a lower SN ratio and reduced ranging accuracy. Furthermore, if the amount of reflected light is high, the amount of charge that can be stored in the charge storage section becomes saturated, leading to saturation of the signal levels of signals S1 and S2, making accurate ranging impossible. In the ranging device 100, it is possible to adjust the amount of reflected light received by the light-receiving section 20 by adjusting the illumination conditions of the illumination light from the light source 10 using the method described later, thereby improving ranging accuracy.

[0074] Furthermore, the distance calculated by the distance calculation unit 50 may not be an absolute distance, but a relative distance value. For example, the distance calculated by the distance calculation unit 50 may be a normalized distance taking values ​​from 0 to 1, or the time Δt, which is proportional to the distance, may be used as a relative distance value. Moreover, as long as the distance can be calculated based on the signal output by the light-receiving unit 20, the timing of the emission control pulse and the exposure control pulse is not limited to the examples described above, and there are no special limitations. For example, if the influence of background light is slight, the distance calculation unit 50 may calculate the distance without subtracting signals S2 or S3 from signals S1 and S0. In this case, the A2 / A3 period may be omitted. Furthermore, for example, during the A0 / A1 period when the light source 10 illuminates the illumination light, three or more exposure control pulses with different start timings relative to each other, corresponding to one emission control pulse, may be output. In this case, during exposure according to at least one of the three or more exposure control pulses, only background light is received by the IR pixel 21a, so the A2 / A3 period does not need to be included in the drive sequence. In addition, in the ranging device 100, the driving sequence and distance other than those mentioned above can also be calculated by using various known indirect TOF methods such as continuous wave (CW) TOF.

[0075] Furthermore, an IR image can also be generated based on the signal obtained through the aforementioned driving sequence. For example, an IR image can be generated by using the sum of signals S0 and S1 of each IR pixel 21a as a brightness value. The generation of the IR image can be performed by the distance calculation unit 50 or by the image recognition unit 30. Additionally, the background light component can be subtracted during the generation of the IR image. Furthermore, the IR image can also be generated by the light receiving unit 20 performing the same imaging sequence as a typical two-dimensional image generation imaging device while the light source 10 is illuminating the image with illumination light.

[0076] [action] Next, an example of the operation of the ranging device 100 according to this embodiment will be described. Figure 7 This is a flowchart illustrating an example of the operation of the ranging device 100 according to this embodiment. Figure 7The image shows an example of a distance measurement method performed by the distance measuring device 100. Figure 7 In this process, the step that combines steps S13 and S14 is an example of an image recognition step, step S15 is an example of a drive control step, and step S18 is an example of a distance calculation step. Furthermore, the following description explains that the light-receiving unit 20, as an imaging element, includes... Figure 3A The image sensor 20A1 shown here has a plurality of IR pixels 21a. The ranging method described below is a method for ranging using the plurality of IR pixels 21a of the image sensor 20A1.

[0077] First, the drive control unit 40 causes the light source 10 to irradiate a predetermined range with irradiation light under a first irradiation condition (step S11). The light receiving unit 20 receives the reflected light reflected within the predetermined range by the irradiation light irradiated under the first irradiation condition in step S11, and outputs a signal based on the reflected light (step S12). In steps S11 and S12, for example, the above-described procedure is performed. Figure 4 , Figure 5A and Figure 5B In the driving sequence described herein, each of the plurality of IR pixels 21a of the light-receiving unit 20 outputs the aforementioned signals S0 to S3. At this time, the distance calculation unit 50 can also acquire the output signals S0 to S3, calculate the distance based on the signals S0 to S3, and generate a distance image. The first illumination condition for the light source 10 to illuminate the illumination light can be a predetermined illumination condition, or it can be the illumination condition when the light source 10 illuminated the illumination light during the last operation of the ranging device 100.

[0078] Next, the image recognition unit 30 acquires an image of at least a portion of a predetermined area illuminated by the illumination light from the light source 10 (step S13). The image recognition unit 30 acquires, for example, at least one of the following as the image of at least a portion of the predetermined area: an IR image captured by infrared light, a visible light image captured by visible light (RGB image or BW image), and a distance image. The IR image, visible light image, and distance image acquired by the image recognition unit 30 are, for example, images generated based on the signal output by the light receiving unit 20. The image recognition unit 30 can acquire the signal output by the light receiving unit 20 and use the acquired signal directly as an image, or it can generate an image by performing a predetermined operation on the signal output by the light receiving unit 20. The drive control unit 40, for example, controls the light receiving unit 20 and, as needed, controls the light source 10 during the timing up to step S13, causing the light receiving unit 20 to output a signal for generating the image used in step S13.

[0079] When the image recognition unit 30 acquires an IR image or a distance image, the image recognition unit 30 may also generate an image based on the signal output by the light-receiving unit 20 through steps S11 and S12. Furthermore, the image recognition unit 30 may also acquire a distance image from the distance calculation unit 50. In this case, the distance calculation unit 50 may also generate the distance image based on the signal output by the light-receiving unit 20 through steps S11 and S12.

[0080] Next, the image recognition unit 30 detects the predetermined object OBJ by performing image recognition on the acquired image (step S14). Figure 8 This diagram illustrates a specific example of the image recognition unit 30 detecting the object OBJ to be recognized. Figure 8 The example shown illustrates a case where the face of a person is detected as the object of identification (OBJ). The image recognition unit 30, for example, uses an image recognition model 81 stored in the storage unit 80 to detect the object OBJ from the acquired image 35. Specifically, the image recognition unit 30 extracts features from the image 35 and inputs the extracted features into the image recognition model 81. Based on the input features, the image recognition model 81 outputs the probability that the object OBJ is present in the image. If the probability output by the image recognition model 81 is above a predetermined threshold, the image recognition unit 30 determines that the object OBJ is present in the image 35 and outputs the position of the region 36 that represents the object OBJ. For example, if region 36 is rectangular, the image recognition unit 30 outputs the coordinates of the rectangular region as the position of region 36.

[0081] Furthermore, if the image recognition unit 30 detects multiple object OBJs in the image 35, it can also select one object OBJ from the multiple object OBJs for adjusting the illumination conditions as described later.

[0082] For example, when the image recognition unit 30 selects one object OBJ from multiple objects to be recognized, it can also select the object OBJ based on registration information. Registration information, for example, is information representing the face of the person using the application and is stored in the storage unit 80. The image recognition unit 30 preferentially selects the object OBJ corresponding to the registration information.

[0083] Furthermore, for example, when the image recognition unit 30 selects one object OBJ from a plurality of objects to be recognized, it can also select the object OBJ based on its distance to the object OBJ. For example, the image recognition unit 30 selects the object OBJ whose distance to the object OBJ is closest as calculated by the distance calculation unit 50. Alternatively, the image recognition unit 30 can also select the object OBJ whose distance to the object OBJ is closest to a predetermined distance.

[0084] Furthermore, when the image recognition unit 30 selects one object OBJ from a plurality of objects to be recognized (OBJ), it may also select the object OBJ based on the size of the region 36 that projects the object OBJ. For example, the image recognition unit 30 may select the object OBJ projected in the region 36 with the highest number of pixels within the rectangle of region 36. Alternatively, the image recognition unit 30 may select the object OBJ projected in the region 36 within the rectangle of region 36 whose pixel count is closest to a predetermined pixel count.

[0085] Furthermore, when the image recognition unit 30 selects one object OBJ from a plurality of object OBJs, it can also select the object OBJ based on its reflectivity. Since the signal level of the signal used to calculate the distance changes even at the same distance, the reflectivity of the object OBJ can be calculated based on both the distance and the signal level. For example, the image recognition unit 30 selects the object OBJ whose reflectivity is closest to a predetermined reflectivity. For instance, if the object OBJ is a human face, the image recognition unit 30 will select the object OBJ whose reflectivity is closest to 50% of the typical reflectivity of a human face.

[0086] In addition, steps S11 and S12 can be performed before step S15 below, or after step S13 or step S14.

[0087] Next, when the image recognition unit 30 detects the object OBJ, the drive control unit 40 adjusts the illumination conditions for the light source 10 to illuminate the object OBJ based on the first outputs of one or more IR pixels 21a corresponding to the region 36 of the object OBJ (step S15). The drive control unit 40 then determines a second illumination condition as the adjusted illumination condition. The first output is a signal output from each of the one or more IR pixels 21a based on the reflected light from the illumination under the first illumination condition within a specified range. The first output is an output containing one or more signals output from the IR pixels 21a, for example, including signals as... Figure 6AThe first output may further include signals S0 and S1, which are based on the reflected light signal. The drive control unit 40 adjusts the illumination conditions of the illumination light based on the first output to ensure that the amount of reflected light received by the IR pixel 21a is the desired amount. For example, the drive control unit 40 adjusts the amount of illumination light reaching the object OBJ to be identified according to the illumination conditions of the illumination light, thereby making the amount of reflected light received by the IR pixel 21a close to the desired amount. Furthermore, the drive control unit 40 adjusts the exposure conditions of the IR pixel 21a as needed, together with the illumination conditions of the illumination light.

[0088] The aforementioned one or more IR pixels 21a are, for example, IR pixels 21a that receive light from the region 36 of the object to be identified (OBJ). Furthermore, the aforementioned one or more IR pixels 21a may also be IR pixels 21a that receive light from a region that is either the central region of the region 36 divided by the object to be identified (OBJ) or the nearest region of the object to be identified (OBJ). Thus, even if the region 36 contains a background, it is possible to exclude it from the region used to adjust the illumination conditions. The central region is the region located in the center of the region 36 divided by the object to be identified. Furthermore, the nearest region is the region within the region 36 divided by the object to be identified that contains the point with the shortest distance from the ranging device 100 to the object to be identified (OBJ). The nearest region is determined by the distance calculated using the distance calculation unit 50.

[0089] The drive control unit 40 adjusts, for example, at least one of the number of pulses emitted by the light source 10 as illumination light, the luminous intensity of the light source 10, and the beam angle of the illumination light, thereby adjusting the illumination conditions of the illumination light. In other words, the means by which the drive control unit 40 adjusts the illumination conditions (the objects of the illumination condition adjustment performed by the drive control unit 40) include at least one of the number of pulses emitted by the light source 10 as illumination light, the luminous intensity of the light source 10, and the beam angle of the illumination light. The drive control unit 40 may also adjust two or more of the following: the number of pulses emitted by the light source 10 as illumination light, the luminous intensity of the light source 10, and the beam angle of the illumination light.

[0090] Here, a method (algorithm) for the drive control unit 40 to adjust the illumination conditions of the illumination light in step S15 will be described. Examples of methods for the drive control unit 40 to adjust the illumination conditions of the illumination light include the following first method and second method.

[0091] In the first method, the drive control unit 40 calculates a representative value of the signal level of the first output of each of the one or more IR pixels 21a, and adjusts the illumination conditions of the illumination light based on the calculated representative values. The representative value is, for example, an average value or a median value. The signal level of the first output is, for example, the signal level obtained by summing signals S0 and S1. Alternatively, the signal level of the first output can be described as the brightness value of the reflected light in the IR pixel 21a.

[0092] In calculating the representative value of the signal level of the first output, the drive control unit 40 can also calculate the representative value by calculating the average or median of the signal levels above a predetermined threshold among the signal levels of the first outputs of each of the more than one IR pixel 21a. Therefore, regarding the first output of an IR pixel 21a that receives reflected light from a region that becomes the background, since the amount of reflected light received by that IR pixel 21a is low and the signal level is low, it is excluded from the calculation of the representative value. Furthermore, in the case where the identified object OBJ is a human face, regarding the first output of an IR pixel 21a that receives reflected light from hair with low reflectivity, since the amount of reflected light received by that IR pixel 21a is low and the signal level is low, it is excluded from the calculation of the representative value.

[0093] The drive control unit 40 adjusts the illumination conditions, for example, so that the calculated representative value becomes the signal level of the target. For instance, the drive control unit 40 calculates the ratio of the calculated representative value to the signal level of the target (ratio = target signal level / calculated representative value), and adjusts the illumination conditions by changing the amount of light reaching the object OBJ from the first illumination condition according to this ratio. The drive control unit 40, for example, uses the adjusted illumination conditions as the illumination conditions obtained by multiplying the number of pulses emitted or the luminous intensity of the light source 10 under the first illumination condition by this ratio.

[0094] In the second method, the drive control unit 40 calculates a representative value of the distance to the area of ​​the identified object OBJ, and adjusts the illumination conditions of the illumination light based on the calculated representative value. This distance to the area of ​​the identified object OBJ is calculated based on the first output of each of the one or more IR pixels 21a. The representative value is, for example, the average or median value. This distance is calculated, for example, by the distance calculation unit 50 using the signals S0 to S3 included in the first output, through the method described above.

[0095] In calculating the representative value of the distance, the drive control unit 40 may also calculate the representative value by calculating the average or median of the distances to the area of ​​the identified object OBJ calculated based on the first output of each of the more than one IR pixels 21a, which are below a predetermined threshold. Therefore, the first output of the IR pixels 21a that receives reflected light from the area that becomes the background is excluded from the calculation of the representative value.

[0096] The drive control unit 40 adjusts the illumination conditions of the illumination light, for example, using a calculated representative value of the distance and a condition table 82 stored in the storage unit 80. The condition table 82 is an example of a table that establishes a correspondence between the representative value of the distance and the set value of the illumination conditions. The drive control unit 40 adjusts the illumination conditions of the illumination light by referring to the condition table 82. Figure 9 This is a diagram from Table 82, which illustrates the conditions used to adjust the illumination light. Figure 9 (a) shows the relationship between the distance to the object OBJ and the signal level of the signal output by the IR pixel 21a when the distance to the object OBJ is calculated by illuminating the object OBJ with illumination light under certain illumination conditions. Furthermore, in Figure 9 (b) shows the relationship between the representative value of the distance in condition table 82 and the set value of the illumination conditions (e.g., the number of pulses or the luminous intensity of the light source 10).

[0097] If the distance from the ranging device 100 to the object OBJ increases, the amount of light reaching the object OBJ decreases due to the attenuation of the incident light. Therefore, when the object OBJ is predetermined, since its reflectivity is already determined, the amount of reflected light received by the IR pixel 21a is also determined by the distance to the object OBJ. For example, if the object OBJ is a human face, the reflectivity is approximately 50%. As a result, such as Figure 9 As shown in (a), if the distance to the object OBJ increases, the signal level of the signal output by the IR pixel 21a based on the reflected light also decreases. Therefore, if the distance to the object OBJ increases, the ranging accuracy decreases due to the reduced SN ratio. Thus, as... Figure 9 As shown in (b), according to Figure 9 The relationship shown in (a) is defined in the following condition table 82. In condition table 82, the signal level of the signal output by the IR pixel 21a is made to match the signal level of the target. The larger the representative value of the distance calculated above, the higher the number of light emission times or the light emission intensity. Therefore, since the illumination conditions can be appropriately adjusted, the ranging accuracy can be improved. If the reflectivity of the object being illuminated is different from that of the object being identified (OBJ), the relationship between distance and signal level changes from... Figure 9The relationship shown in (a) changes. However, since the illumination conditions are adjusted when the image recognition unit 30 detects the object OBJ as described above, it is possible to avoid a situation where the illumination conditions cannot be properly adjusted due to the reflectivity of the object being illuminated being different from what is expected. Furthermore, in the second method, the first illumination condition is, for example, set to the illumination condition used in the settings of condition table 82. For example, when using Figure 9 Given the condition table 82 for the relationship shown in (b), it is set that it is possible to obtain Figure 9 The first irradiation condition is shown in (a). Furthermore, in the first method, a table corresponding to representative values ​​of signal levels and set values ​​of irradiation conditions can also be used.

[0098] Furthermore, in the second method, the drive control unit 40 can also correct the second illumination condition, which is determined by adjusting the illumination condition of the illumination light based on a representative value of distance, according to the temperature of the light-receiving part 20 measured by the temperature sensor 70. Specifically, the drive control unit 40 corrects the second illumination condition in such a way that the higher the temperature of the light-receiving part 20 measured by the temperature sensor 70, the smaller the amount of illumination light illuminating the object OBJ to be identified. Figure 10 This diagram illustrates a method for correcting the second irradiation conditions based on the temperature of the light-receiving part 20. Figure 10 Figure (a) shows the relationship between the temperature of the light-receiving part 20 and the signal level of the signal output by the IR pixel 21a when the light-receiving part 20 receives a certain amount of light. Furthermore, in Figure 10 (b) shows the relationship between the correction gain used to correct the second irradiation condition and the temperature of the light-receiving part 20.

[0099] Since the higher the sensitivity of the IR pixel 21a, the higher the temperature, therefore... Figure 10 As shown in (a), under the same amount of received light, the higher the temperature of the light-receiving part 20, the higher the signal level of the signal output by the IR pixel 21a. Therefore, for example, the drive control unit 40 multiplies the number of pulse light emission times or the luminous intensity of the light source 10, which have been adjusted based on the above condition table 82, by... Figure 10 As shown in (b), the correction gain decreases as the temperature of the light-receiving part 20 increases. In this case, condition table 82 is used, which is set based on the relationship between the distance to the identified object OBJ and the signal level of the signal output by the IR pixel 21a at the reference temperature Ts. At the reference temperature Ts, the correction gain is 1. For the correction gain at temperatures other than the reference temperature Ts, the following values ​​are used, which are... Figure 10In the relationship shown in (a), the signal level at the reference temperature Ts is used as the reference level Ls, for example, by dividing the reference level Ls by the signal level at that temperature. In this way, the drive control unit 40 corrects the second irradiation conditions according to the temperature of the light-receiving unit 20, so that even if the temperature of the light-receiving unit 20 is different from the reference temperature Ts set in the condition table 82, the signal level of the signal output by the IR pixel 21a can be close to the signal level of the target.

[0100] Furthermore, if the image recognition unit 30 does not detect the object OBJ, the drive control unit 40 may adjust the illumination conditions of the illumination light based on the output of the IR pixel 21a, which reflects light from an object other than the object OBJ, or it may not adjust the illumination conditions. Additionally, if the image recognition unit 30 does not detect the object OBJ, the drive control unit 40 may also stop driving the light source 10 and the light-receiving unit 20.

[0101] Furthermore, the drive control unit 40 can also adjust the illumination conditions of the illumination light according to the signal level of the third output, which is a signal output by each of the multiple IR pixels 21a based on the amount of background light received without reflected light. The illumination condition adjustment means in this case may include, for example, the number of pulses emitted by the light source 10 as the illumination light, and at least one of the light intensity of the light source 10 and the light distribution angle of the illumination light. The third output is an output containing one or more signals output from the IR pixels 21a, and is acquired by the background light measurement unit 60. The background light measurement unit 60 acquires, for example, at least one of the signals S2 and S3 output by the IR pixels 21a during steps S11 and S12 as the third output. Alternatively, the signal level of the third output can be described as the brightness value of the background light in the IR pixels 21a. Additionally, the background light measurement unit 60 can also acquire the third output output by the IR pixels 21a when the light-receiving unit 20 receives background light outside of step S12.

[0102] The drive control unit 40 calculates, for example, a representative value of the signal level of the third output from each of the plurality of IR pixels 21a. The representative value is, for example, the average value or the median value. For example, when the calculated representative value of the signal level of the third output is above a predetermined threshold, that is, when the amount of background light is high, the drive control unit 40 uses an adjustment means that increases the illuminance of the illumination light illuminating the object OBJ compared with other adjustment means in the illumination condition adjustment means.

[0103] For example, if the representative value of the calculated signal level of the third output is above a predetermined threshold, and the irradiation conditions are adjusted in a way that increases the amount of light reaching the object to be identified (OBJ) compared to the amount of light in the first irradiation condition, the drive control unit 40 adjusts at least one of the luminous intensity of the light source 10 and the beam angle of the irradiation light as the irradiation conditions. The drive control unit 40 may also determine whether to adjust the irradiation conditions in a way that increases the amount of light reaching the object to be identified (OBJ) compared to the amount of light in the first irradiation condition based on the signal level of the first output or the distance calculated from the first output. For example, using... Figure 6A As explained, signals S0 and S1, based on the reflected light from the object OBJ, each contain signals S0b and S1b, which are components corresponding to the background light. Therefore, if the background light component in signals S0 and S1 increases, the signals S0a and S1a, which are components corresponding to the reflected light used for distance calculation, decrease, resulting in a lower range accuracy due to the reduced SN ratio. Thus, under conditions with a high background light component, increasing the number of pulsed light emission times as the illumination condition also increases the number of exposures, resulting in an increased background light component, making it difficult to efficiently improve range accuracy. Conversely, by adjusting the luminous intensity of the light source 10 or the beam angle of the illumination light to increase the amount of light reaching the object OBJ, the illuminance of the illumination light reaching the object OBJ increases compared to increasing the number of pulsed light emission times; in other words, the number of beams of illumination light incident on the surface of the object OBJ increases. Therefore, by adjusting the luminous intensity of the light source 10 or the beam angle of the illumination light as the illumination condition, the amount of reflected light relative to the amount of background light can be increased in each exposure. As a result, the background light component in signals S0 and S1 is reduced, thus improving the SN ratio and thereby efficiently improving ranging accuracy.

[0104] On the other hand, when the representative value of the calculated third output signal level is above a predetermined threshold and the irradiation conditions are adjusted by the drive control unit 40 in a manner that reduces the amount of light reaching the identified object OBJ compared to the first irradiation condition, the number of pulse light emission times is adjusted as the irradiation condition of the irradiation light. Therefore, since the amount of reflected light relative to the background light does not decrease in each exposure, further reduction in the SN ratio can be suppressed.

[0105] Furthermore, if the representative value of the calculated signal level of the third output is less than a predetermined threshold, the drive control unit 40 can adjust the irradiation conditions by a predetermined adjustment method regardless of how the irradiation conditions are adjusted, or it can adjust the irradiation conditions by the same adjustment method as when the representative value is above the predetermined threshold.

[0106] Refer again Figure 7 Next, the drive control unit 40 causes the light source 10 to irradiate a predetermined range with illumination light under a second illumination condition, which is the adjusted illumination condition from step S15 (step S16). The light receiving unit 20 receives the reflected light from the illumination light irradiated under the second illumination condition in step S16 within the predetermined range, and outputs a signal based on the reflected light (step S17). In steps S16 and S17, for example, the above-described procedure is performed. Figure 4 , Figure 5A and Figure 5B The driving sequence described herein is such that the IR pixel 21a of the light-receiving unit 20 outputs the aforementioned signals S0 to S3.

[0107] Next, the distance calculation unit 50 calculates the distance to the identified object OBJ based on the second outputs output by each of the plurality of IR pixels 21a (step S18). The second output is a signal output by each of the plurality of IR pixels 21a based on the reflected light from the illumination light irradiated under the second illumination condition within a specified range. The second output is an output containing one or more signals output from the IR pixels 21a, such as the signals S0 to S3 described above. The distance calculation unit 50 acquires the second output containing signals S0 to S3 from each of the plurality of IR pixels 21a, and calculates the distance for each of the plurality of IR pixels 21a based on signals S0 to S3, thereby generating a distance image. The distance image is an image containing the distance calculated based on the reflected light received by the plurality of IR pixels 21a, which is the pixel value of the plurality of IR pixels 21a. In this embodiment, the illumination conditions of the illumination light irradiated by the light source 10 are adjusted based on the first output, which is a signal output by one or more IR pixels 21a corresponding to the identified object OBJ detected by the image recognition unit 30. Therefore, by appropriately adjusting the amount of light illuminating the object OBJ to be identified, the ranging accuracy of the distance to the object OBJ can be improved.

[0108] The distance calculation unit 50 outputs the generated distance image to an external device, for example. The distance image output by the distance calculation unit 50 is then input to an external information processing device for applications such as facial recognition systems and driver monitoring systems.

[0109] (other) The foregoing has described a ranging device according to one or more embodiments of the present disclosure, but the present disclosure is not limited to the embodiments. Various modifications that would be conceived by those skilled in the art to the embodiments, as well as forms constructed by combining the constituent elements of different embodiments, may also be included within the scope of one or more embodiments of the present disclosure without departing from the spirit of the present disclosure.

[0110] Furthermore, the ranging device disclosed herein may not possess all of the constituent elements described in the above embodiments, or may consist only of constituent elements for performing actions on a target.

[0111] Furthermore, in the above embodiments, the ranging device is a ranging device that measures distance using an indirect TOF method, but it is not limited to this. The ranging device involved in this disclosure can also be a ranging device that measures distance using a direct TOF method. In a ranging device that measures distance using a direct TOF method, the drive control unit can also adjust the illumination conditions of the illumination light using the same method as in the above embodiments, thereby adjusting the illumination conditions to be suitable for identifying the object.

[0112] Furthermore, in the above embodiments, each component can also be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0113] Furthermore, each component can be implemented in hardware. Each component can also be a circuit (or integrated circuit). These circuits can form a single circuit as a whole, or they can be separate circuits. Additionally, these circuits can be general-purpose circuits or dedicated circuits.

[0114] Furthermore, the present disclosure, in its entirety or in specific forms, can also be implemented by a system, apparatus, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. Additionally, it can be implemented by any combination of the system, apparatus, method, integrated circuit, computer program, and recording medium.

[0115] For example, this disclosure can also be implemented as a ranging device of the above embodiments, as a control device for controlling a ranging device, as a ranging method including steps (processes) performed on the constituent elements constituting the ranging device, as a program for causing a computer to execute such a ranging method, or as a computer-readable non-transitory recording medium recording such a program.

[0116] The following examples illustrate the ranging apparatus and ranging method described in this disclosure based on the embodiments described above. The ranging apparatus and ranging method disclosed herein are not limited to the examples below.

[0117] For example, the ranging device according to the first aspect of this disclosure includes: a light source that illuminates a predetermined range with illumination light; a light-receiving unit having a plurality of pixels that receive reflected light from the illumination light within the predetermined range; an image recognition unit that acquires an image covering at least a portion of the predetermined range and detects a predetermined object to be identified by performing image recognition on the acquired image; a drive control unit that, when the image recognition unit detects the object to be identified, adjusts the illumination conditions of the illumination light illuminating the illumination light based on a first output, and causes the light source to illuminate the illumination light with the adjusted illumination conditions, wherein the first output is a signal output by one or more pixels among the plurality of pixels corresponding to the area of ​​the object to be identified, based on the reflected light of the illumination light illuminated under the predetermined illumination conditions; and a distance calculation unit that calculates the distance to the object to be identified based on a second output, wherein the second output is a signal output by each of the plurality of pixels based on the reflected light of the illumination light illuminated under the adjusted illumination conditions.

[0118] Accordingly, the drive control unit can adjust the illumination conditions of the illumination light, which is used for ranging, based on a first output corresponding to one or more pixels outputting a predetermined area of ​​the object to be identified. Therefore, by adjusting the illumination conditions to suit the object to be identified, ranging accuracy can be improved.

[0119] Furthermore, for example, in the ranging device according to the second aspect of this disclosure, according to the ranging device according to the first aspect, the drive control unit calculates a representative value of the signal level of the first output output by each of the one or more pixels, and adjusts the illumination conditions of the illumination light based on the calculated representative value.

[0120] Since the signal level of the first output corresponds to the amount of reflected light received by one or more pixels, the illumination conditions can be adjusted with good precision using a representative value of the signal level of the first output.

[0121] Furthermore, for example, in the ranging device according to the third aspect of this disclosure, according to the ranging device according to the second aspect, the drive control unit adjusts the illumination conditions of the illumination light in such a way that the representative value becomes the signal level of the target.

[0122] Accordingly, the illumination conditions can be adjusted in a way that makes the amount of reflected light received by more than one pixel the desired amount.

[0123] Furthermore, for example, in the ranging device according to the fourth aspect of this disclosure, according to the ranging device according to the second or third aspect, the drive control unit calculates the representative value based on a signal level above a predetermined threshold among the signal levels of the first output signal levels output by each of the one or more pixels.

[0124] Therefore, even when the area of ​​the detected object contains background, the representative value can be calculated by excluding the background part with a lower signal level from the calculation of the representative value.

[0125] Furthermore, for example, in the ranging device according to the fifth aspect of this disclosure, according to the ranging device according to the first aspect, the drive control unit calculates a representative value of the distance to the area of ​​the identified object, adjusts the illumination conditions of the illumination light based on the calculated representative value, and the distance to the area of ​​the identified object is calculated based on the first output output by each of the one or more pixels.

[0126] Since the amount of light reaching the object being identified varies with the distance to the object, representative values ​​of the distance can be used to precisely adjust the illumination conditions.

[0127] Furthermore, for example, in the ranging device according to the sixth aspect of this disclosure, according to the ranging device according to the fifth aspect, the drive control unit adjusts the illumination conditions of the illumination light by referring to a table that establishes a correspondence between the representative value and the set value of the illumination conditions of the illumination light.

[0128] Therefore, since the illumination conditions can be adjusted simply by referring to the table, the process can be simplified. Furthermore, since the distance is calculated based on the first output of one or more pixels corresponding to the region of the identified object, and the reflectivity of the object being used for distance calculation is predetermined, a table of appropriate illumination condition settings can be used.

[0129] Furthermore, for example, in the ranging device according to the seventh aspect of this disclosure, according to the ranging device according to the fifth or sixth aspect, the drive control unit calculates the representative value based on a distance below a predetermined threshold in the distance to the area of ​​the identified object, and the distance to the area of ​​the identified object is calculated based on the first output output by each of the one or more pixels.

[0130] Therefore, even when the area of ​​the detected object contains background, the representative value can be calculated by excluding the background portion from the calculation of the representative value.

[0131] Furthermore, for example, the ranging device according to the eighth aspect of this disclosure, according to any of the fifth to seventh aspects, further includes a temperature sensor that measures the temperature of the light-receiving part, and the drive control unit corrects the adjusted illumination conditions based on the temperature of the light-receiving part measured by the temperature sensor.

[0132] Therefore, even when the sensitivity of the light-receiving part changes with temperature, the influence on sensitivity can be suppressed by correcting the adjusted irradiation conditions.

[0133] Furthermore, for example, in the ranging device according to the ninth aspect of this disclosure, or in the ranging device according to any one of the first to eighth aspects, the one or more pixels corresponding to the region of the identified object are pixels that receive light from a region that is the central region of a region that is a segmented region of the region of the identified object or the nearest region of the identified object.

[0134] Accordingly, in the area where the object is identified, the first output can be used from the adjusted pixels that are more suitable for the illumination conditions.

[0135] Furthermore, for example, the ranging device according to the 10th aspect of this disclosure, according to any of the 1st to 9th aspects, further includes a background light measuring unit that acquires a third output, which is a signal output by each of the plurality of pixels based on the amount of light received by the background light that does not include the reflected light, and the drive control unit adjusts the illumination conditions of the illumination light according to the signal level of the third output.

[0136] Accordingly, illumination condition adjustment methods that reduce the influence of background light can be used according to the amount of background light. For example, when the amount of background light is high, an adjustment method that increases the illuminance of the illumination light illuminating the object to be identified can be used in the illumination condition adjustment methods compared to other adjustment methods.

[0137] Furthermore, for example, in the ranging device according to the 11th aspect of this disclosure, according to the ranging device according to any one of the 1st to 10th aspects, the light source irradiates the light multiple times with pulsed light as the irradiation light, and the drive control unit adjusts the irradiation conditions of the irradiation light at least by adjusting the number of times the pulsed light is emitted.

[0138] Therefore, since the irradiation conditions can be adjusted by the number of pulses emitted by the light source, the adjustment of irradiation conditions becomes easy.

[0139] Furthermore, for example, in the ranging device according to the 12th aspect of this disclosure, according to the ranging device according to any one of the 1st to 11th aspects, the light source is capable of changing the luminous intensity of the light source, and the drive control unit adjusts the illumination conditions of the illumination light at least by adjusting the luminous intensity.

[0140] Therefore, the irradiation conditions can be adjusted without changing the irradiation time of the irradiation light.

[0141] Furthermore, for example, in the ranging device according to the 13th aspect of this disclosure, according to the ranging device according to any one of the 1st to 12th aspects, the light source is capable of changing the light distribution angle of the illumination light, and the drive control unit adjusts the illumination conditions of the illumination light at least by adjusting the light distribution angle.

[0142] Therefore, even if the amount of light emitted by the light source is the same, the amount of light reaching the object being identified can be adjusted. For example, even if the amount of light reaching the object being identified is increased, energy savings can be achieved because there is no need to increase the amount of light emitted by the light source.

[0143] Furthermore, for example, the ranging method according to the 14th aspect of this disclosure is a ranging method performed by a ranging device, the ranging device comprising: a light source that illuminates a predetermined range with illumination light; and a light-receiving unit having a plurality of pixels that receive reflected light from the illumination light within the predetermined range, the ranging method comprising: an image recognition step of acquiring an image that reflects at least a portion of the predetermined range, and detecting a predetermined object to be identified by performing image recognition on the acquired image; a drive control step of adjusting the illumination conditions of the illumination light by the light source based on a first output when the object to be identified is detected in the image recognition step, and causing the light source to illuminate the illumination light with the adjusted illumination conditions, the first output being a signal output by one or more pixels among the plurality of pixels corresponding to the area of ​​the object to be identified based on the reflected light of the illumination light illuminated under the predetermined illumination conditions; and a distance calculation step of calculating the distance to the object to be identified based on a second output, the second output being a signal output by the plurality of pixels based on the reflected light of the illumination light illuminated under the adjusted illumination conditions.

[0144] Based on this ranging method, similar to the ranging device involved in the first method described above, it is possible to adjust the illumination conditions to suit the object being identified, thereby improving ranging accuracy.

[0145] Industrial availability The ranging devices and the like disclosed herein can be applied to various uses such as distance measurement systems, sensing systems using distance images, and authentication systems.

[0146] Explanation of reference numerals in the attached figures 10 Light Source 20 Light-receiving section 20A, 20A1, 20A2, 20A3, 20A4, 20A5, 20A6 camera elements 21 pixels 21a IR pixels 21b R pixels 21c G pixels 21dB pixels 21e BW pixels 30 Image Recognition Department 35 images 36 regions 40 Drive Control Unit 50 Distance Calculation Unit 60 Background Light Measurement Section 70 Temperature Sensor 80 Storage Department 81 Image Recognition Model 82 Condition table 100 Distance Measuring Device OBJ identifies objects.

Claims

1. A ranging device, comprising: A light source directs illumination light onto a specified area; The light-receiving part has multiple pixels, which receive reflected light reflected by the irradiated light within a predetermined range; The image recognition unit acquires an image that reflects at least a portion of the defined range, and detects a predetermined object by performing image recognition on the acquired image. The drive control unit, when the image recognition unit detects the object to be identified, adjusts the illumination conditions of the light source illuminating the illumination light based on the first output, and causes the light source to illuminate the illumination light under the adjusted illumination conditions. The first output is a signal output by one or more pixels among the plurality of pixels corresponding to the region of the object to be identified, based on the reflected light of the illumination light illuminating under the prescribed illumination conditions. as well as The distance calculation unit calculates the distance to the identified object based on the second output, which is a signal output by each of the plurality of pixels based on the reflected light of the illumination light illuminating under the adjusted illumination conditions.

2. The ranging device as described in claim 1, The drive control unit calculates a representative value of the signal level of the first output by each of the one or more pixels, and adjusts the illumination conditions of the illumination light based on the calculated representative value.

3. The ranging device as described in claim 2, The drive control unit adjusts the irradiation conditions of the irradiation light in a manner that makes the representative value the signal level of the target.

4. The ranging device as described in claim 2 or 3, The drive control unit calculates the representative value based on a signal level above a predetermined threshold among the signal levels of the first output signal level output by each of the one or more pixels.

5. The ranging device as described in claim 1, The drive control unit calculates a representative value of the distance to the area of ​​the identified object, and adjusts the illumination conditions of the illumination light based on the calculated representative value. The distance to the area of ​​the identified object is calculated based on the first output output by each of the one or more pixels.

6. The ranging device as described in claim 5, The drive control unit adjusts the irradiation conditions of the irradiation light by referring to a table that corresponds to the representative value and the set value of the irradiation conditions of the irradiation light.

7. The ranging device as described in claim 5 or 6, The drive control unit calculates the representative value based on a distance below a predetermined threshold in the distance to the region of the identified object. The distance to the region of the identified object is calculated based on the first output output by each of the one or more pixels.

8. The ranging device as described in any one of claims 5 to 7, The ranging device also includes a temperature sensor that measures the temperature of the light-receiving part. The drive control unit corrects the adjusted irradiation conditions based on the temperature of the light-receiving part measured by the temperature sensor.

9. The ranging device as described in any one of claims 1 to 8, The one or more pixels corresponding to the region of the identified object are pixels that receive light from a region that is the central region of a region that is a segment of the region of the identified object or the nearest region of the identified object.

10. The ranging device as described in any one of claims 1 to 9, The ranging device further includes a background light measurement unit, which acquires a third output. This third output is a signal output by each of the plurality of pixels based on the amount of background light received, excluding the reflected light. The drive control unit adjusts the irradiation conditions of the irradiation light according to the signal level output by the third unit.

11. The ranging device as described in any one of claims 1 to 10, The light source uses pulsed light as the illumination light to irradiate multiple times. The drive control unit adjusts the irradiation conditions of the irradiation light by adjusting at least the number of times the pulse light is emitted.

12. The ranging device as described in any one of claims 1 to 11, The light source can change its luminous intensity. The drive control unit adjusts the irradiation conditions of the irradiation light by adjusting the luminous intensity at least once.

13. The ranging device as described in any one of claims 1 to 12, The light source can change the light distribution angle of the illumination light. The drive control unit adjusts the illumination conditions of the illumination light by adjusting the light distribution angle at least once.

14. A distance measurement method, which is a distance measurement method performed by a distance measuring device. The ranging device includes: A light source directs light onto a specified area; and The light-receiving part has multiple pixels, which receive reflected light from the irradiated light within a predetermined range. The ranging method includes: The image recognition step involves acquiring an image that reflects at least a portion of the defined range, and detecting a predetermined object by performing image recognition on the acquired image. In the drive control step, when the object to be identified is detected in the image recognition step, the illumination conditions of the light source illuminating the illumination light are adjusted based on the first output, and the light source illuminates the illumination light under the adjusted illumination conditions. The first output is a signal output by one or more pixels among the plurality of pixels corresponding to the region of the object to be identified, based on the reflected light of the illumination light illuminated under the specified illumination conditions. as well as The distance calculation step calculates the distance to the identified object based on the second output, which is a signal output by each of the plurality of pixels based on the reflected light of the illumination light under the adjusted illumination conditions.

Images