Distance measurement sensor, signal processing method and distance measurement module

Abstract

The present technology relates to a distance measuring sensor, a signal processing method, and a distance measuring module, which can perform calculation of distance information and outdoor judgment by the distance measuring sensor alone. The distance measuring sensor includes: a distance measuring unit that calculates distance information to an object based on a signal obtained by a light receiving unit, the light receiving unit receiving reflected light obtained by reflecting irradiation light emitted from a predetermined light source from the object; an ambient light calculation unit that calculates an ambient light component included in the signal obtained by the light receiving unit; and an outdoor information calculation unit that calculates outdoor information based on the ambient light component. The present technology can be applied to, for example, a distance measuring module that measures the distance to a subject, etc.

Description

Technical Field

[0001] The present technology relates to a distance measurement sensor, a signal processing method, and a distance measurement module, and more particularly, to a distance measurement sensor, a signal processing method, and a distance measurement module capable of performing calculation of distance information and outdoor judgment by the distance measurement sensor alone. Background Art

[0002] In recent years, with the advancement of semiconductor technology, the miniaturization of distance measurement modules that measure the distance to an object has progressed. As a result, for example, installation of distance measurement modules on mobile terminals such as so-called smartphones, which are small information processing devices with communication functions, has been achieved.

[0003] Examples of the distance measurement method in the distance measurement module include an indirect time-of-flight (ToF) method and a structured light method. In the indirect ToF method, light is emitted toward an object and light reflected on the surface of the object is detected, and the distance to the object is calculated based on a measurement value obtained by measuring the flight time of the light. In the structured light method, patterned light is emitted toward an object, and the distance to the object is calculated based on an image obtained by imaging distortion of the pattern on the surface of the object.

[0004] Distance measurement methods, in which an object is illuminated with active light and reflected light reflected by the object is received, such as indirect ToF methods, and structured light methods are easily affected by ambient light such as sunlight. Therefore, it is useful to be able to judge whether the measurement location is outdoors or indoors.

[0005] For example, there is an imaging device in which an outdoor detection sensor is provided separately from an imaging element to detect that the imaging device is outdoors (for example, Patent Document 1).

[0006] Reference List

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2005-175888 Summary of the invention

[0009] Problems to be solved by the present invention

[0010] However, it is desirable to perform outdoor determination while acquiring distance information through a distance measurement sensor alone, rather than separately providing an outdoor detection sensor.

[0011] The present technology has been proposed in view of such circumstances, and is capable of performing calculation of distance information and outdoor determination by the distance measurement sensor alone.

[0012] Solution to the problem

[0013] A distance measuring sensor according to a first aspect of the present technology includes: a distance measuring unit that calculates distance information to an object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting irradiation light emitted from a predetermined light source from the object; an ambient light calculation unit that calculates an ambient light component included in the signal obtained by the light receiving unit; and an outdoor information calculation unit that calculates outdoor information based on the ambient light component.

[0014] A signal processing method according to a second aspect of the present technology includes: calculating distance information to an object based on a signal obtained by a light receiving unit by using a distance measuring sensor, the light receiving unit receiving reflected light obtained by reflecting irradiation light emitted from a predetermined light source by the object; calculating an ambient light component included in the signal obtained by the light receiving unit by using the distance measuring sensor; and calculating outdoor information based on the ambient light component by using the distance measuring sensor.

[0015] A distance measuring module according to a third aspect of the present technology includes a predetermined light source and a distance measuring sensor, the distance measuring sensor including: a distance measuring unit that calculates distance information to an object based on a signal obtained by a light receiving unit, the light receiving unit receiving reflected light obtained by reflecting the irradiation light emitted from the predetermined light source from the object; an ambient light calculating unit that calculates an ambient light component included in the signal obtained by the light receiving unit; and an outdoor information calculating unit that calculates outdoor information based on the ambient light component.

[0016] In the first to third aspects of the present technology, distance information to an object is calculated based on a signal obtained by a light receiving unit, the light receiving unit receives reflected light obtained by reflecting illumination light emitted from a predetermined light source by the object, an ambient light component included in the signal obtained by the light receiving unit is calculated, and outdoor information is calculated based on the ambient light component.

[0017] The distance measurement sensor and the distance measurement module may be independent devices or modules integrated into another device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram showing a schematic configuration example in the first embodiment of the distance measurement module to which the present technology is applied.

[0019] Figure 2 is a diagram for explaining the operation of the distance measuring sensor.

[0020] Figure 3 is a diagram illustrating the operation of a pixel according to the 4-phase method.

[0021] Figure 4 is a diagram illustrating the 4-phase method.

[0022] Figure 5 is a diagram illustrating the 4-phase method.

[0023] Figure 6 is a diagram illustrating a method of calculating a depth value by using a 2-phase method and a 4-phase method.

[0024] Figure 7 is a block diagram of a distance measurement module including a detailed configuration of a distance measurement sensor.

[0025] Figure 8 is a diagram showing the configuration of RAW data obtained by one pixel.

[0026] Fig. 9 is a diagram showing an example of RAW data according to the 4-phase method.

[0027] Fig.10 is a flow chart illustrating a depth value calculation process performed by a distance measurement module.

[0028] Fig.11 is a diagram illustrating a second operation mode of the distance measurement module.

[0029] Fig.12 is a diagram illustrating a third operation mode of the distance measurement module.

[0030] Fig.13 is a block diagram of a second embodiment of a distance measurement module.

[0031] Fig.14 2 is a diagram for explaining the processing of the ambient light calculation unit according to the second embodiment.

[0032] Fig.15 is a block diagram of a third embodiment of a distance measurement module.

[0033] Fig.16 is a block diagram showing a configuration example of an electronic device to which the present technology is applied.

[0034] Fig.17 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied.

[0035] Fig.18 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0036] Fig.19 2 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments for implementing the present technology (hereinafter, referred to as embodiments) will be described. Note that the description will be given in the following order.

[0038] 1. Schematic configuration example of distance measurement module

[0039] 2. Operation of distance measurement sensor by indirect ToF method

[0040] 3. Detailed configuration example of distance measurement sensor

[0041] 4. Depth value calculation and processing

[0042] 5. Second Embodiment

[0043] 6. Third embodiment

[0044] 7. Configuration examples of electronic devices

[0045] 8. Computer configuration example

[0046] 9. Application examples of moving objects

[0047] <1. Schematic Configuration Example of Distance Measurement Module>

[0048] Figure 1 : is a block diagram showing a schematic configuration example in the first embodiment of the distance measurement module to which the present technology is applied.

[0049] Figure 1 The illustrated distance measurement module 11 is a distance measurement module that performs distance measurement by an indirect ToF method, and includes a light emitting source 12 , a light emitting control unit 13 , and a distance measurement sensor 14 .

[0050] The distance measurement module 11 irradiates a predetermined object as a subject with light, receives light (reflected light) obtained by reflecting the light (irradiated light) from the object, and thereby measures a depth value and a confidence level as distance information to the object, and outputs the depth value and the confidence level.

[0051] The light emission source 12 includes, for example, an infrared laser diode or the like as a light source, emits light while performing modulation at a timing corresponding to a light emission control signal supplied from the light emission control unit 13 , and irradiates a subject with the irradiation light.

[0052] The light control unit 13 controls the light source 12 by providing a light control signal of a predetermined frequency (e.g., 20 MHz, etc.) to the light source 12. In addition, the light control unit 13 also provides the light control signal to the distance measurement sensor 14 so as to drive the light receiving unit 21 according to the light emission timing of the light source 12.

[0053] Although it will be referenced later Figure 2The distance measuring sensor 14 will not be described in detail, but a pixel array unit 32 in which a plurality of pixels 31 are two-dimensionally arranged receives reflected light from an object. Then, the distance measuring sensor 14 generates and outputs a depth value and a confidence level for each pixel 31 of the pixel array unit 32.

[0054] <2. Operation of distance measurement sensor by indirect ToF method>

[0055] Next, we will refer to Figures 2 to 6 The operation of the distance measuring sensor 14 is described.

[0056] The distance measuring sensor 14 includes Figure 2 The light receiving unit 21 is shown.

[0057] The light receiving unit 21 includes a pixel array unit 32 in which each pixel 31 generates charge according to the amount of received light and outputs a signal according to the charge and is two-dimensionally arranged in a matrix in the row direction and the column direction, and a drive control circuit 33 arranged in the peripheral area of ​​the pixel array unit 32.

[0058] The drive control circuit 33 outputs a control signal for controlling the drive of the pixel 31 (eg, a distribution signal DIMIX, a selection signal ADDRESS DECODE, a reset signal RST, etc., which will be described later) based on, for example, a light emission control signal supplied from the light emission control unit 13 .

[0059] The pixel 31 includes a photodiode 51, and a first tap 52A and a second tap 52B for detecting charges photoelectrically converted by the photodiode 51. In the pixel 31, the charges generated in one photodiode 51 are distributed to the first tap 52A or the second tap 52B. Then, the charges generated in the photodiode 51 and distributed to the first tap 52A are output as a detection signal A from a signal line 53A, and the charges generated in the photodiode 51 and distributed to the second tap 52B are output as a detection signal B from a signal line 53B.

[0060] The first tap 52A includes a transfer transistor 41A, a floating diffusion (FD) unit 42A, a selection transistor 43A, and a reset transistor 44A. Similarly, the second tap 52B includes a transfer transistor 41B, a FD unit 42B, a selection transistor 43B, and a reset transistor 44B.

[0061] The operation of the pixel 31 will be described.

[0062] like Figure 3As shown, the light source 12 outputs modulated illumination light (one cycle = 2T) so that the illumination is turned on / off repeatedly with the illumination time T, and the photodiode 51 receives the reflected light with a delay of the delay time ΔT corresponding to the distance to the object. In addition, the distribution signal DIMIX_A controls the on / off of the transfer transistor 41A, and the distribution signal DIMIX_B controls the on / off of the transfer transistor 41B. The distribution signal DIMIX_A is a signal having the same phase as the illumination light, and the distribution signal DIMIX_B has a phase obtained by inverting the phase of the distribution signal DIMIX_A.

[0063] Therefore, in Figure 2 In the embodiment, when the transfer transistor 41A is turned on according to the distribution signal DIMIX_A, the charge generated by the photodiode 51 receiving the reflected light is transferred to the FD unit 42A, and when the transfer transistor 41B is turned on according to the distribution signal DIMIX_B, the charge generated by the photodiode 51 receiving the reflected light is transferred to the FD unit 42B. Therefore, during a predetermined period in which irradiation with irradiation light for the irradiation time T is periodically performed, the charge transferred via the transfer transistor 41A is sequentially accumulated in the FD unit 42A, and the charge transferred via the transfer transistor 41B is sequentially accumulated in the FD unit 42B.

[0064] Then, if the selection transistor 43A is turned on according to the selection signal ADDRESS DECODE_A after the period of accumulating the charge ends, the charge accumulated in the FD unit 42A is read via the signal line 53A, and a detection signal A corresponding to the charge amount is output from the light receiving unit 21. Similarly, if the selection transistor 43B is turned on according to the selection signal ADDRESS DECODE_B, the charge accumulated in the FD unit 42B is read via the signal line 53B, and a detection signal B corresponding to the charge amount is output from the light receiving unit 21. In addition, if the reset transistor 44A is turned on according to the reset signal RST_A, the charge accumulated in the FD unit 42A is discharged, and if the reset transistor 44B is turned on according to the reset signal RST_B, the charge accumulated in the FD unit 42B is discharged.

[0065] As described above, the pixel 31 distributes the charge generated by the reflected light received by the photodiode 51 to the first tap 52A or the second tap 52B according to the delay time Δt, and outputs the detection signal A and the detection signal B. Then, the delay time ΔT corresponds to the time in which the light emitted from the light source 12 flies to the object, is reflected by the object, and then flies to the light receiving unit 21, that is, corresponds to the distance to the object. Therefore, the distance measurement module 11 can obtain the distance to the object (depth value) based on the detection signal A and the detection signal B according to the delay time ΔT.

[0066] However, in the pixel array unit 32, due to the deviation (sensitivity difference) of the characteristics of each element such as the photodiode 51 and the pixel transistor such as the transfer transistor 41 included in each pixel 31, there is a case where the detection signal A and the detection signal B are affected differently for each pixel 31. Therefore, in the distance measurement module 11 of the indirect ToF method, a technique is adopted for removing the sensitivity difference between the taps of each pixel and improving the SN ratio by acquiring the detection signal A and the detection signal B obtained by receiving the reflected light by changing the phase in the same pixel 31.

[0067] As a method of receiving reflected light by changing the phase and calculating the depth value, for example, a detection method by using two phases (2-phase method) and a detection method by using four phases (4-phase method) will be described.

[0068] like Figure 4 As shown, the light receiving unit 21 receives the reflected light at the light receiving timings of the phases shifted by 0°, 90°, 180°, and 270° relative to the irradiation timing of the irradiation light. More specifically, the light receiving unit 21 receives the reflected light by changing the phase in a time-division manner, so that in a specific frame period, the light is received with the phase set to 0° relative to the irradiation timing of the irradiation light, in the next frame period, the light is received with the phase set to 90°, in the next frame period, the light is received with the phase set to 180°, and in the next frame period, the light is received with the phase set to 270°.

[0069] Figure 5 It is a diagram in which the exposure periods of the first tap 52A of the pixel 31 are arranged in the respective phases of 0°, 90°, 180°, and 270° make it possible to easily understand the phase difference.

[0070] like Figure 5 As shown, in the first tap 52A, the detection signal A obtained by receiving light in the same phase (phase 0°) as the irradiation light is referred to as the detection signal A0, and the detection signal A obtained by receiving light in a phase shifted by 90° from the irradiation light (phase 90°) is referred to as the detection signal A 90 , the detection signal A obtained by receiving light in a phase shifted by 180° from the irradiation light (phase 180°) is referred to as the detection signal A 180 , the detection signal A obtained by receiving light in a phase shifted by 270° from the irradiation light (phase 270°) is referred to as the detection signal A 270 .

[0071] In addition, even if the description is omitted, in the second tap 52B, the detection signal B obtained by receiving the light in the same phase (phase 0°) as the irradiation light is referred to as the detection signal B0, and the detection signal B obtained by receiving the light in the phase (phase 90°) shifted by 90° from the irradiation light is referred to as the detection signal B0. 90 , the detection signal B obtained by receiving the light in a phase shifted by 180° from the irradiation light (phase 180°) is referred to as the detection signal B 180 , the detection signal B obtained by receiving the light in a phase shifted by 270° from the irradiation light (phase 270°) is referred to as the detection signal B 270 .

[0072] Figure 6 : is a diagram illustrating a method of calculating a depth value and a confidence by using a 2-phase method and a 4-phase method.

[0073] In the indirect ToF method, the depth value d can be obtained by the following formula (1).

[0074] [Mathematical expression 1]

[0075]

[0076] In formula (1), c represents the speed of light, ΔT represents the delay time, and f represents the modulation frequency of light. The phase shift amount [rad] of the reflected light is expressed by the following formula (2).

[0077] [Mathematical expression 2]

[0078]

[0079] In the 4-phase method, detection signals A0 to A1 obtained by setting the phases to 0°, 90°, 180°, and 270° are used. 270 and detection signals B0 to B 270 I and Q in formula (2) are calculated by the following formula (3). I and Q are signals obtained by assuming that the brightness change of the irradiation light is a cos wave and converting the phase of the cos wave from the polar coordinate system to the orthogonal coordinate system (IQ plane).

[0080] I=c0-c 180 =(A0-B0)-(A 180 -B 180 )

[0081] Q=c 90 -c 270 =(A 90 -B 90 )-(A 270 -B270 )…………(3)

[0082] In the 4-phase method, for example, by taking the difference between detection signals of opposite phases in the same pixel, such as “A0-A 180 ” and “A 90 -A 270 ”, which can remove the characteristic variations between taps present in each pixel, i.e., fixed pattern noise.

[0083] In contrast, in the 2-phase method, the detection signals A0 to A 270 The detection signals B0 to B0 obtained by setting the phase to 0°, 90°, 180°, and 270° 270 The depth value d of the object is obtained by using two phases in an orthogonal relationship between them. For example, when using the detection signals A0 and B0 with a 0° phase and the detection signal A0 with a 90° phase 90 and B 90 In the case of , I and Q in formula (2) are expressed by the following formula (4).

[0084] I=c0-c 180 =(A0-B0)

[0085] Q=c 90 -c 270 =(A 90 -B 90 )…………(4)

[0086] For example, when using the detection signal A with a 180° phase 180 and B 180 And the detection signal A with a phase of 270° 270 and B 270 In the case of , I and Q in formula (2) are expressed by the following formula (5).

[0087] I=c0-c 180 =-(A 180 -B 180 )

[0088] Q=c 90 -c 270 =-(A 270 -B 270 )…………(5)

[0089] In the 2-phase method, the characteristic variation between taps present in each pixel cannot be eliminated; however, the depth value d to the object can be obtained only by detecting signals in two phases. Therefore, distance measurement can be performed at twice the frame rate of the 4-phase method. The characteristic variation between taps can be adjusted by correction parameters such as gain or offset.

[0090] In the 2-phase method and the 4-phase method, the confidence cnf can be obtained by the following formula (6).

[0091] [Mathematical expression 3]

[0092]

[0093] In the present embodiment, it does not matter whether the distance measurement module 11 uses the I signal and Q signal corresponding to the delay time ΔT calculated by the 4-phase method or the I signal and Q signal corresponding to the delay time ΔT calculated by the 2-phase method to use the depth value d and the confidence cnf. The 4-phase method or the 2-phase method may be fixedly used, or, for example, a method of appropriately selecting the 4-phase method or the 2-phase method or a mixed 4-phase method and the 2-phase method according to the movement of the object, etc. may be used. In the following, for simplicity, it is assumed that the 4-phase method is adopted.

[0094] <3. Detailed configuration example of distance measurement sensor>

[0095] Figure 7 is a block diagram of the distance measurement module 11 including a detailed configuration of the distance measurement sensor 14 .

[0096] Apart from Figure 2 In addition to the light receiving unit 21 shown, the distance measuring sensor 14 includes a distance measuring unit 22 , an ambient light calculating unit 23 , an ambient light normalizing unit 24 , an outdoor information calculating unit 25 , and a filtering unit 26 .

[0097] The distance measurement module 11 irradiates a predetermined object with light, receives light (reflected light) obtained by reflecting the light (irradiated light) from the object, and thus measures and outputs a depth value and a confidence level as distance information to the object.

[0098] Specifically, the light receiving unit 21 sets each pixel 31 of the pixel array unit 32 as a measurement target pixel, and provides RAW data, which is a detection signal corresponding to the light reception amount of the reflected light received by the measurement target pixel, to the light emission control unit 13, the distance measuring unit 22, and the ambient light calculation unit 23.

[0099] The light emission control unit 13 controls the light emission source 12 by providing a light emission control signal of a predetermined frequency to the light emission source 12, controls the exposure time based on the RAW data of the light receiving unit 21, and generates a light emission control signal for realizing the set exposure time. Therefore, the light emission control unit 13 has an AE function (automatic exposure function) based on the RAW data of the light receiving unit 21, and provides the set exposure time to the ambient light normalization unit 24.

[0100] Based on the RAW data of the measurement target pixel provided from the light receiving unit 21, the distance measuring unit 22 calculates the depth value d, which is the distance information from the distance measuring module 11 to the object in the measurement target pixel, and the confidence cnf of the depth value d, and provides the depth value d and the confidence cnf to the filtering unit 26. The method of calculating the depth value d and its confidence cnf is as described above.

[0101] The ambient light calculation unit 23 calculates an ambient light component included in the RAW data of the measurement target pixel supplied from the light receiving unit 21 , and supplies the ambient light component to the ambient light normalization unit 24 .

[0102] Figure 8 The configuration of RAW data (detection signal) obtained by one pixel is shown.

[0103] The RAW data includes an active component acv, an ambient light component amb, and a dark current component drk. The active component acv is a light component of the irradiation light emitted from the light source 12, reflected by the object, and returned. The ambient light component amb is a light component of ambient light such as sunlight. The dark current component drk is a noise component generated by a dark current generated in the light receiving unit 21, regardless of light reception.

[0104] Fig. 9 The detection signal A is shown when the distance measurement sensor 14 measures different distances D1 and D2. i and B i The ratio of the active component acv, ambient light component amb and dark current component drk in (i=any one of 0, 90, 180, 270).

[0105] Here, the ambient light component amb can be obtained by the following formula (7).

[0106] [Mathematical expression 4]

[0107]

[0108] The dark current component drk is, for example, a fixed value acquired in advance by acquiring the detection signals A and B in a state where the light receiving unit 21 is shielded.

[0109] Formula (7) is a formula for obtaining the ambient light component amb in the 4-phase method. However, even in the 2-phase method, it is possible to obtain the ambient light component amb by omitting the detection signal A which has not yet been measured. i and B i , and change the denominator of the fraction to 4, and calculate the ambient light component amb similarly.

[0110] Back to Figure 7 , the ambient light normalization unit 24 normalizes the ambient light component amb of the measurement target pixel supplied from the ambient light calculation unit 23 by using the number of pixels and the exposure time.

[0111] Specifically, the ambient light normalization unit 24 normalizes the ambient light component amb of the measurement target pixel supplied from the ambient light calculation unit 23 by using formula (8) to calculate the normalized ambient light component amb norm .

[0112] [Mathematical expression 5]

[0113]

[0114] In formula (8), Σamb represents the sum of ambient light components amb of all pixels of pixel array unit 32, and pix_n represents the number of pixels of pixel array unit 32. Furthermore, base_exp_time represents a preset basic exposure time as an initial value, and current_exp_time represents the current exposure time of the measurement target pixel provided from light emission control unit 13.

[0115] In addition, the ambient light normalization unit 24 can obtain the depth value d of the measurement target pixel from the distance measurement unit 22, and normalize the ambient light component amb by using the number of pixels, the exposure time and the depth value. In this case, the normalized ambient light component amb is calculated by the following formula (9): norm .

[0116] [Mathematical expression 6]

[0117]

[0118] By normalizing the ambient light component amb, individual adjustments based on the number of pixels, exposure time, distance, etc. become unnecessary. The ambient light normalization unit 24 provides the already calculated ambient light component amb to the outdoor information calculation unit 25. norm , where the ambient light component amb norm has been normalized.

[0119] The outdoor information calculation unit 25 calculates the ambient light component amb based on the normalized ambient light component amb. norm, and the ambient light component amb that has been provided from the ambient light normalization unit 24 norm , calculates outdoor information about whether the current environment measured by the distance measurement module 11 is outdoors, and provides the outdoor information to the filtering unit 26. The outdoor information may be a probability of being outdoors (hereinafter referred to as outdoor probability), or may be a binary value indicating outdoor or indoor. In the case where the outdoor information is represented by a binary value, it is only necessary to perform outdoor judgment by using 50% as a threshold. In the present embodiment, a description will be given assuming that the outdoor information calculation unit 25 calculates and outputs the outdoor probability as the outdoor information.

[0120] For example, the outdoor information calculation unit 25 calculates the outdoor probability α (0≤α≤1) of the measurement target pixel by the following formula (10).

[0121] α=a×amb norm +b…………(10)

[0122] a and b in formula (10) are predetermined constants determined in advance.

[0123] The filtering unit 26 performs an optimum filtering process on the distance measurement result from the distance measuring unit 22 based on the outdoor probability α which is the outdoor information supplied from the outdoor information calculating unit 25 .

[0124] Specifically, for the measurement target pixel, if the outdoor probability α as outdoor information is provided from the outdoor information calculation unit 25, and the depth value d and the confidence cnf are provided from the distance measurement unit 22, the filtering unit 26 calculates the filtered depth value d' and the confidence cnf' by the following formula (11).

[0125] d'=α·f1(d)+(1-α)·f2(d)

[0126] cnf'=α·g1(cnf)+(1-α)·g2(cnf)......(11)

[0127] Here, f1() represents a parameter set of an outdoor filter with a depth value d as input, and f2() represents a parameter set of an indoor filter with a depth value d as input. g1() represents a parameter set of an outdoor filter with a confidence level cnf as input, and g2() represents a parameter set of an indoor filter with a confidence level cnf as input. An outdoor filter or an indoor filter is a filter obtained by adjusting any filter, such as a filter for noise reduction or a filter for sharpening a boundary portion of an object for outdoor or indoor use. In the case where the outdoor information is a binary value indicating outdoor or indoor, α is 1 or 0. Therefore, either an outdoor filter or an indoor filter is selected.

[0128] The filtering unit 26 outputs the filtered depth value d′ and the confidence cnf′ as the depth value and the confidence of the measurement target pixel to the outside of the distance measurement module 11 .

[0129] Note that the filtering unit 26 may generate mapping data in which the depth value d' or the confidence cnf' is stored as the pixel value of each pixel 31 of the pixel array unit 32, and output the mapping data to a subsequent stage. In this case, a depth map in which the depth value d' is stored as the pixel value of each pixel 31 of the pixel array unit 32 and a confidence map in which the confidence cnf' is stored as the pixel value of each pixel 31 of the pixel array unit 32 are generated and output.

[0130] <4. Depth value calculation process>

[0131] Will refer to Fig.10 The depth value calculation process performed by the distance measurement module 11 is described with reference to a flowchart of FIG. For example, when a light emission control signal is supplied from the light emission control unit 13 to the light emission source 12 and the distance measurement sensor 14, the process is started.

[0132] First, in step S1, the light receiving unit 21 receives the reflected light based on the light emission control signal from the light emission control unit 13. Specifically, the light receiving unit 21 receives the reflected light by changing the phase in a time-division manner so that the light reception timing of each pixel 31 of the pixel array unit 32 is in the phases of 0°, 90°, 180°, and 270°, respectively, relative to the irradiation timing of the irradiation light. The detection signals A0 to A1 of each pixel obtained by sequentially setting the phase to 0°, 90°, 180°, and 270° are 270 and detection signals B0 to B 270 The RAW data is supplied to the light emission control unit 13 , the distance measurement unit 22 , and the ambient light calculation unit 23 .

[0133] In step S2, the distance measuring unit 22 sequentially sets each pixel 31 of the pixel array unit 32 as a measurement target pixel, calculates the depth value d and the confidence cnf of the measurement target pixel based on the RAW data of the measurement target pixel provided from the light receiving unit 21, and provides the depth value d and the confidence cnf to the filtering unit 26. The depth value d can be calculated by formula (1), and the confidence cnf can be calculated by formula (6).

[0134] In step S3, the ambient light calculation unit 23 calculates the ambient light component amb included in the RAW data of the measurement target pixel supplied from the light receiving unit 21, and supplies the ambient light component amb to the ambient light normalization unit 24. The ambient light component amb can be obtained by the following formula (7).

[0135] In step S4, the ambient light normalization unit 24 normalizes the ambient light component amb of the measurement target pixel provided from the ambient light calculation unit 23 by using the number of pixels and the exposure time. For example, the ambient light normalization unit 24 calculates the normalized ambient light component amb by formula (8): norm , and the normalized ambient light component amb norm Provided to the outdoor information calculation unit 25. Note that, as mentioned above, the normalized ambient light component amb can also be calculated by formula (9): norm .

[0136] In step S5, the outdoor information calculation unit 25 calculates the ambient light component amb based on the normalized ambient light component amb. norm , and the ambient light component amb that has been provided from the ambient light normalization unit 24 norm , calculates outdoor information about whether the current environment measured by the distance measurement module 11 is outdoors, and provides the outdoor information to the filtering unit 26. For example, the outdoor information calculation unit 25 calculates the outdoor probability α of the measurement target pixel by the above formula (10).

[0137] In step S6, the filtering unit 26 performs optimal filtering processing on the distance measurement result from the distance measurement unit 22 according to the outdoor probability α as the outdoor information provided from the outdoor information calculation unit 25. Specifically, the filtering unit 26 calculates the filtered depth value d' and the confidence cnf' by using formula (11) with respect to the depth value d and the confidence cnf of the measurement target pixel from the distance measurement unit 22. The filtered depth value d' and the confidence cnf' are output to the outside of the distance measurement module 11 as the depth value and the confidence of the measurement target pixel.

[0138] By sequentially setting the respective pixels 31 of the pixel array unit 32 as measurement target pixels, the processes of steps S2 to S6 are performed on all the pixels 31 of the pixel array unit 32 .

[0139] In step S7, the light emission control unit 13 sets the next exposure time based on the RAW data supplied from the light receiving unit 21. The process of step S7 may be performed in parallel with steps S2 to S6.

[0140] Thus, the depth value calculation process performed by the distance measurement module 11 is completed.

[0141] In the above-mentioned depth value calculation process, both the distance information (depth value d, confidence cnf) to the object and the outdoor information are calculated from the detection signal obtained in each pixel 31 of the pixel array unit 32, and the process of reflecting the outdoor information (outdoor probability) in the distance information is performed. However, in addition to the distance information, the calculated outdoor information can also be output to the outside.

[0142] Furthermore, the distance measurement sensor 14 may also perform an operation of outputting only one of the distance information and the outdoor information.

[0143] Specifically, the distance measuring sensor 14 includes, as operation modes, a first operation mode for calculating both distance information and outdoor information, a second operation mode for calculating distance information without calculating outdoor information and outputting only distance information, and a third operation mode for calculating only outdoor information without calculating distance information, and performs processing according to the operation mode specified by a setting screen or a setting control signal.

[0144] In the first operating mode, the distance measuring sensor 14 performs Fig.10 The depth value calculation process shown.

[0145] In the second operating mode, the distance measuring sensor 14 is operated by Fig.11 The light receiving unit 21, the distance measuring unit 22, and the filtering unit 26 indicated by the solid line in FIG. 2 operate, calculate the distance information without calculating the outdoor information, and output only the distance information. The filtering unit 26 performs, for example, a predetermined filtering process determined in advance.

[0146] In the third operating mode, the distance measuring sensor 14 is operated by Fig.12 The light receiving unit 21, the ambient light calculation unit 23, the ambient light normalization unit 24, and the outdoor information calculation unit 25 indicated by the solid line in FIG. operate, and calculate the outdoor information without calculating the distance information. In this case, the distance measurement sensor 14 operates as an outdoor determination sensor. Compared with an outdoor determination sensor using a general RGB sensor that receives RGB light, since the resolution of the pixel array unit 32 can be made lower, the driving power can be suppressed, and outdoor determination can be achieved with lower power consumption.

[0147] As described above, according to the distance measuring sensor 14 and the distance measuring module 11, the distance measuring sensor alone can perform the calculation of distance information and the outdoor judgment. Since the distance measuring sensor 14 and the distance measuring module 11 can perform the calculation of distance information and the outdoor judgment alone, the power consumption and the installation volume can be reduced. Since the distance measuring sensor 14 receives infrared light that cannot be imaged by the RGB camera, it is possible to detect the presence of the sun with higher accuracy than imaging by the RGB camera.

[0148] <5. Second Embodiment>

[0149] Fig.13 A block diagram showing a second embodiment of a distance measurement module is shown.

[0150] exist Fig.13 In the first embodiment, Figure 7Corresponding parts are denoted by the same reference numerals, and description thereof will be appropriately omitted.

[0151] Fig.13 The second embodiment of Figure 7 The illustrated first embodiment is configured similarly except that a subject region detection unit 81 is newly provided, and the ambient light calculation unit 23 is changed to an ambient light calculation unit 82 .

[0152] The depth value d and the confidence cnf of each pixel 31 of the pixel array unit 32 are supplied from the distance measurement unit 22 to the object region detection unit 81 .

[0153] The subject area detection unit 81 generates a confidence map in which the confidence cnf is stored as a pixel value of each pixel 31 of the pixel array unit 32. Then, the subject area detection unit 81 detects a subject area, which is a region including a subject (object) in the entire pixel area (hereinafter also referred to as a light receiving area) of the pixel array unit 32, based on the confidence map that has been generated, and provides the detected subject area as region of interest (ROI) information indicating a region of interest (i.e., a region to be focused in the light receiving area) to the ambient light calculation unit 82.

[0154] Note that the subject area detection unit 81 may also generate a depth map in which the depth value d is stored as the pixel value of each pixel 31 of the pixel array unit 32, and also detect the subject area by using the depth map. Also by using the distance information, the subject area can be detected more accurately. Alternatively, the subject area can be detected by using only the depth map without using the confidence map.

[0155] The ambient light calculation unit 82 performs processing similar to that in the first embodiment with respect to the region of interest indicated in the light receiving area by the ROI information supplied from the subject region detection unit 81. That is, the ambient light calculation unit 82 calculates the ambient light component amb included in the RAW data for each pixel 31 in the region of interest, and supplies the ambient light component amb to the ambient light normalization unit 24.

[0156] In addition, the ambient light calculation unit 82 may also receive ROI information indicating the region of interest from the outside of the distance measuring sensor 14, etc. When the ROI information is provided, the ambient light calculation unit 82 calculates the ambient light component amb included in the RAW data of the region of interest indicated by the ROI information in the light receiving area, and provides the ambient light component amb to the ambient light normalization unit 24.

[0157] Fig.14 2 is a diagram for explaining the processing of the ambient light calculation unit 82 .

[0158] Fig.14 FIG. 1A shows an example of a confidence map in which the confidence cnf of each pixel 31 provided from the distance measurement unit 22 is stored. Note that since the confidence map is actually a grayscale image, Fig.14 A is a conceptual diagram of confidence mapping.

[0159] For example, the instruction Fig.14 The area information of area 91 of B is provided as ROI information to Fig.14 Confidence map shown in A of FIG. In the case where the region information indicating region 91 is the subject region detected by the subject region detection unit 81, region 91 may be dynamically changed according to the movement of the subject. Alternatively, in the case where the region information indicating region 91 is ROI information provided from the outside, region 91 is fixed unless the ROI information is updated.

[0160] The ambient light calculation unit 82 calculates the ambient light component amb of all pixels in the area 91 in the entire area of ​​the confidence map. Alternatively, the ambient light calculation unit 82 may calculate the ambient light component amb only for predetermined sampled pixels 92 in the area 91. In this case, the sampled pixels 92 are predetermined by parameters or the like.

[0161] As described above, the ambient light calculation unit 82 acquires ROI information indicating a portion of the region of interest in the entire area of ​​the light receiving area, calculates the ambient light component amb of the region of interest, and supplies the ambient light component amb to the ambient light normalization unit 24 .

[0162] Fig.13 The ambient light normalization unit 24 and the outdoor information calculation unit 25 in the embodiment perform similar processing as the first embodiment with respect to the region of interest. That is, the ambient light normalization unit 24 normalizes the ambient light component amb of each pixel 31 in the region of interest and converts the normalized ambient light component amb into norm The outdoor information is supplied to the outdoor information calculation unit 25. The outdoor information calculation unit 25 calculates the outdoor information of each pixel 31 in the region of interest and supplies the outdoor information to the filtering unit 26.

[0163] The filtering unit 26 performs processing similar to that of the first embodiment with respect to the region of interest. That is, the filtering unit 26 performs optimal filtering processing on the distance measurement result from the distance measurement unit 22 based on the outdoor information of each pixel 31 in the region of interest. Note that with respect to the region other than the region of interest in the entire region of the light receiving area, the value from the distance measurement unit 22 may be used as it is, or processing according to the filtering processing of the region of interest, for example, average filtering processing of the filtering processing performed on the individual pixels of the region of interest, etc. may be performed.

[0164] Since the depth value calculation process in the second embodiment is similar to the reference Fig.10 The depth value calculation process in the first embodiment described is basically similar, and its detailed description will be omitted. In the depth value calculation process in the second embodiment, Fig.10 Between steps S2 and S3 in the above, the subject region detection process performed by the subject region detection unit 81 or the process of obtaining ROI information from the outside by the ambient light calculation unit 82 is added. Then, in steps S4 to S6, the ambient light component amb is calculated for the pixel 31 in the region of interest, the calculated ambient light component amb is normalized, and the outdoor information is calculated. The rest of the process is the same as that in the reference. Fig.10 The depth value calculation process of the first embodiment described is similar.

[0165] <6. Third embodiment>

[0166] Fig.15 is a block diagram of a third embodiment of a distance measurement module.

[0167] exist Fig.15 In, corresponding to Fig.13 The parts in the second embodiment shown are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0168] exist Fig.15 In the third embodiment of the present invention, an imaging sensor 101 is provided in addition to the distance measuring module 11. The imaging sensor 101 is an image sensor that receives light of RGB wavelengths and generates an image (RGB image) of a subject. Fig.15 The distance measurement module 11 and the imaging sensor 101 constitute a distance measurement system (imaging system).

[0169] The imaging sensor 101 includes a light receiving unit 111 and a signal processing unit 112 , and the signal processing unit 112 includes a demosaic processing unit 121 , a ROI determination unit 122 , and a filtering unit 123 .

[0170] Fig.15 The distance measurement module 11 shown is Fig.13 The distance measurement module 11 of the second embodiment shown is different in that the subject region detection unit 81 is omitted therein. The ROI information generated by the ROI determination unit 122 of the imaging sensor 101 is provided to the ambient light calculation unit 82. In addition, the outdoor information generated by the outdoor information calculation unit 25 is provided to the filtering unit 26, and is also provided to the filtering unit 123 of the imaging sensor 101. Another part of the distance measurement module 11 of the third embodiment is similar to that of the second embodiment described above.

[0171] The light receiving unit 111 includes a pixel array unit in which individual pixels filled with red (R), green (G) or blue (B) colors arranged in a Bayer array or the like are arranged two-dimensionally, and a signal obtained by photoelectrically converting light of an R, G or B wavelength received by each pixel is provided to the de-mosaic processing unit 121 as an imaging signal.

[0172] The demosaic processing unit 121 performs color information interpolation processing, etc. by using the pixel signal of any one pixel signal of the R signal, G signal and B signal provided from the light receiving unit 111, generates an image signal including the pixel signal of the R signal, the G signal and the B signal for each pixel, and provides the image signal to the ROI determination unit 122 and the filtering unit 123.

[0173] The ROI determination unit 122 performs a region of interest determination process for determining a region of interest on the image signal supplied from the demosaic processing unit 121. The ROI determination unit 122 performs a process similar to that of the subject region detection unit 81, except that the processing target image is not a grayscale image but an RGB image. Needless to say, the ROI determination unit 122 may determine the region of interest by a process different from that of the subject region detection unit 81. The ROI determination unit 122 supplies ROI information indicating the region of interest obtained as a result of the region of interest determination process to the filtering unit 123 and the ambient light calculation unit 82 of the distance measurement sensor 14.

[0174] The image signal is supplied from the demosaic processing unit 121, and the ROI information is supplied from the ROI determination unit 122 to the filtering unit 123. Furthermore, the outdoor information is also supplied from the outdoor information calculation unit 25 of the distance measurement sensor 14 to the filtering unit 123.

[0175] The filtering unit 123 performs optimal filtering processing on the image signal of the region of interest in the image signal from the demosaic processing unit 121 based on the outdoor information of the region of interest. Note that the region other than the region of interest in the image captured by the light receiving unit 111 may be retained as it is, or processing according to the filtering processing of the region of interest may be performed, for example, an average filtering processing of the filtering processing performed on the respective pixels of the region of interest, etc.

[0176] The filtering unit 123 outputs a filtered image signal obtained by applying a predetermined filtering process to at least the region of interest to the outside.

[0177] Since the depth value calculation process of the distance measurement module 11 is similar to the depth value calculation process of the above-described second embodiment, a description thereof will be omitted.

[0178] As described above, according to the third embodiment, the distance measurement module 11 can calculate the ambient light component amb based on the ROI information detected by the imaging sensor 101 that receives the RGB light and generates the captured image, and output the depth value d' and the confidence cnf'. In addition, since the filtering unit 123 of the imaging sensor 101 performs appropriate filtering processing on the RGB image signal based on the outdoor information, for example, the adjustment of the hue and the edge can be optimally controlled according to the scene, etc.

[0179] Figure 1 The distance measurement module 11 in the embodiment can be applied to, for example, an in-vehicle system that is installed on a vehicle and measures the distance to a target outside the vehicle. Figure 1 The distance measurement module 11 in the embodiment may be applied to a gesture recognition system, etc., which measures the distance to a target such as a user's hand and recognizes the user's gesture based on the measurement result.

[0180] <7. Configuration Example of Electronic Equipment>

[0181] The distance measurement module 11 may be installed on an electronic device such as a smartphone, a tablet terminal, a mobile phone, a personal computer, a game console, a television receiver, a wearable terminal, a digital still camera, or a digital video camera.

[0182] Fig.16 is a block diagram showing a configuration example of a smartphone as an electronic device on which a distance measurement module is mounted.

[0183] like Fig.16 As shown, the smartphone 201 is configured by connecting a distance measurement module 202, an imaging device 203, a display 204, a speaker 205, a microphone 206, a communication module 207, a sensor unit 208, a touch panel 209, and a control unit 210 via a bus 211. In addition, execution of a program by the CPU allows the control unit 210 to have functions as an application processing unit 221 and an operating system processing unit 222.

[0184] Figure 1 The distance measurement module 11 in the embodiment is applied to the distance measurement module 202. For example, the distance measurement module 202 is arranged on the front surface of the smartphone 201, and performs distance measurement on the user of the smartphone 201, so that the depth value of the surface shape of the user's face, hand, finger, etc. can be output as the distance measurement result.

[0185] The imaging device 203 is arranged on the front surface of the smartphone 201, and performs imaging with the user of the smartphone 201 as a subject to acquire an image of the user. Note that although not shown, a configuration may be adopted in which the imaging device 203 may also be provided on the rear surface of the smartphone 201.

[0186] The display 204 displays an operation screen for executing processing performed by the application processing unit 221 and the operating system processing unit 222, an image captured by the imaging device 203, etc. For example, when a call is made by using the smartphone 201, the speaker 205 and the microphone 206 output the voice of the other party and collect the voice of the user.

[0187] The communication module 207 performs communication via a communication network. The sensor unit 208 senses speed, acceleration, proximity, and the like, and the touch panel 209 acquires a touch operation of a user on an operation screen displayed on the display 204 .

[0188] The application processing unit 221 performs processing for providing various services by the smartphone 201. For example, the application processing unit 221 may perform processing for creating a face by computer graphics that virtually reproduces the expression of the user based on the depth provided from the distance measurement module 202 and displaying the face on the display 204. In addition, the application processing unit 221 may perform processing for creating three-dimensional shape data of an arbitrary solid object based on the depth provided from the distance measurement module 202, for example.

[0189] The operating system processing unit 222 performs processing for realizing basic functions and operations of the smartphone 201. For example, the operating system processing unit 222 may perform processing for authenticating the user's face and unlocking the smartphone 201 based on the depth value provided from the distance measurement module 202. In addition, based on the depth value provided from the distance measurement module 202, the operating system processing unit 222 may perform processing for recognizing a user's gesture and processing for inputting various operations according to the gesture, for example.

[0190] In the smartphone 201 configured as described above, for example, by applying the distance measurement module 11 as described above, calculation of distance measurement information and outdoor determination can be performed simultaneously. Therefore, the smartphone 201 can detect distance measurement information more accurately.

[0191] <8. Example of computer configuration>

[0192] Next, the above-described series of processing may be executed by hardware or by software. In the case where the series of processing is executed by software, a program configuring the software is installed on a general-purpose computer or the like.

[0193] Fig.17 : is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0194] In the computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, a random access memory (RAM) 303, and an electronically erasable and programmable read only memory (EEPROM) 304 are connected to each other via a bus 305. In addition, an input / output interface 306 is connected to the bus 305, and the input / output interface 306 is connected to the outside.

[0195] In the computer configured as described above, for example, the CPU 301 loads the program stored in the ROM 302 and the EEPROM 304 into the RAM 303 via the bus 305 and executes the program, and thus the above-mentioned series of processing is performed. In addition, the program executed by the computer (CPU 301) may be written in advance in the ROM 302, installed in the EEPROM 304 from the outside via the input / output interface 306, or updated.

[0196] Thus, the CPU 301 executes the processing according to the above-described flowchart or the processing executed by the configuration of the above-described block diagram. Then, the CPU 301 can output the processing result to the outside via the input / output interface 306, for example, as necessary.

[0197] In this specification, the processing performed by the computer according to the program does not necessarily have to be performed in time series in the order described in the flowchart. That is, the processing performed by the computer according to the program also includes processing performed in parallel or individually (for example, parallel processing or processing by object).

[0198] In addition, the program may be processed by one computer (processor), or may be distributedly processed by a plurality of computers. In addition, the program may be transmitted to a remote computer and executed.

[0199] <9. Application examples of moving objects>

[0200] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile object such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobility vehicle, an airplane, a drone, a ship, or a robot.

[0201] Fig.18 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0202] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. Fig.18In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an inside information detection unit 12040, and an integrated control unit 12050. In addition, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown.

[0203] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle such as an internal combustion engine or a drive motor, a drive force transmitting mechanism for transmitting the drive force to wheels, a steering mechanism for adjusting a steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0204] The body system control unit 12020 controls the operation of various devices provided on the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a rear light, a brake light, a flasher, or a fog light. In this case, a radio wave transmitted from a portable machine instead of a key or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives input of these radio waves or signals and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0205] The vehicle exterior information detection unit 12030 detects information outside the vehicle in which the vehicle control system 12000 is installed. For example, the imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receive the captured image. The vehicle exterior information detection unit 12030 may perform a process of detecting an object such as a person, a vehicle, an obstacle, a sign, a character on a road surface, etc., or a distance detection process based on the received image.

[0206] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information for distance measurement. In addition, the light received by the imaging unit 12031 can be visible light or invisible light, such as infrared light.

[0207] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver condition detector 12041 that detects the condition of the driver is connected to the in-vehicle information detection unit 12040. For example, the driver condition detector 12041 includes a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or concentration level based on the detection information input from the driver condition detector 12041, or can determine whether the driver is not dozing off.

[0208] The microcomputer 12051 can perform arithmetic operations on the control target values ​​of the driving force generation device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and can output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform coordinated control for the purpose of realizing the functions of an advanced driver assistance system (ADAS), including collision avoidance or shock absorption of the vehicle, following driving based on the distance between vehicles, driving to maintain the vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0209] In addition, the microcomputer 12051 controls the driving force generation device, steering mechanism, braking device, etc. based on the vehicle peripheral information obtained by the external vehicle information detection unit 12030 or the internal vehicle information detection unit 12040, and can perform coordinated control aimed at automatic driving, etc., which allows autonomous driving without relying on the driver's operation.

[0210] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the vehicle exterior information acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 can perform coordinated control for anti-glare such as switching from high beam to low beam by controlling the headlights according to the position of the preceding vehicle or oncoming vehicle detected by the vehicle exterior information detection unit 12030.

[0211] The audio and image output unit 12052 transmits an output signal of at least one of audio and image to an output device capable of visually or auditorily notifying information to passengers or the outside of the vehicle. Fig.18 In the example of , as examples of output devices, an audio speaker 12061, a display unit 12062, and a dashboard 12063 are shown. For example, the display unit 12062 may include at least one of an in-vehicle display or a head-up display.

[0212] Fig.19 12031 is a diagram showing an example of the installation position of the imaging unit 12031.

[0213] exist Fig.19In the figure, vehicle 12100 includes imaging units 12101, 12102, 12103, 12104, and 12105 as imaging unit 12031.

[0214] For example, the imaging units 12101, 12102, 12103, 12104, 12105 are arranged at positions such as the front nose, side mirrors, rear bumper, rear door, and upper portion of the windshield of the vehicle compartment 12100. The imaging unit 12101 arranged on the front nose and the imaging unit 12105 arranged on the upper portion of the windshield inside the compartment mainly acquire images in front of the vehicle 12100. The imaging units 12102, 12103 arranged on the side mirrors mainly acquire images on the lateral sides of the vehicle 12100. The imaging unit 12104 arranged on the rear bumper or the rear door mainly acquires images behind the vehicle 12100. The front images acquired by the imaging units 12101, 12105 are mainly used to detect front vehicles, pedestrians, obstacles, traffic lights, traffic signs, traffic lanes, etc.

[0215] Notice, Fig.19 An example of the imaging ranges of the imaging units 12101 to 12104 is shown. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or the rear door. For example, by overlapping a plurality of pieces of image data captured by the imaging units 12101 to 12104, a bird's-eye view of the vehicle 12100 viewed from above can be obtained.

[0216] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging elements, or an imaging element having pixels for phase difference detection.

[0217] For example, the microcomputer 12051 can specifically extract the nearest solid object on the driving road of the vehicle 12100, which is traveling as a preceding vehicle at a predetermined speed (e.g., 0 km / h or more) in substantially the same direction as the vehicle 12100, by determining the distance to each solid object in the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. In addition, the microcomputer 12051 can set the inter-vehicle distance to be ensured behind the preceding vehicle, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. As described above, coordinated control aimed at automatic driving, etc., which is autonomously driven without relying on the operation of the driver, can be performed.

[0218] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can classify the stereoscopic object data related to the stereoscopic object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other stereoscopic objects, and extract them so that they can be used to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles visible to the driver of the vehicle 12100 and obstacles that are almost invisible to the driver of the vehicle 12100. Then, the microcomputer 12051 determines the collision risk indicating the degree of danger of collision with each obstacle, and in the case where there is a possibility of a collision with a collision risk equal to or greater than a set value, the microcomputer 12051 can output an alarm to the driver through the audio speaker 12061 or the display unit 12062 or perform driving support for collision avoidance by performing forced deceleration or avoidance steering through the drive system control unit 12010.

[0219] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 may identify a pedestrian by determining whether there is a pedestrian in an image captured by the imaging units 12101 to 12104. For example, such pedestrian recognition is performed according to a process for extracting feature points in an image captured by the imaging units 12101 to 12104 as an infrared camera, and a process for performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether the object is a pedestrian. If the microcomputer 12051 determines that there is a pedestrian in the captured image of the imaging units 12101 to 12104 and identifies the pedestrian, the audio image output unit 12052 causes the display unit 12062 to display a square outline for emphasis so as to overlap with the identified pedestrian. In addition, the audio image output unit 12052 may cause the display unit 12062 to display an icon indicating a pedestrian at a desired position, etc.

[0220] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040 in the above configuration. Specifically, by using the distance measurement of the distance measurement module 11 as the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040, it is possible to perform processing for recognizing the driver's gesture, perform various operations (e.g., audio system, navigation system, and air conditioning system) according to the gesture, and more accurately detect the driver's condition. In addition, the distance measurement performed by the distance measurement module 11 can be used to identify the unevenness of the road surface and allow the unevenness to be reflected in the control of the suspension.

[0221] Note that the present technology can be applied to a method for amplitude modulating light projected onto an object, which is called a continuous wave method in an indirect ToF method. In addition, the structure of the photodiode 51 of the light receiving unit 21 can be applied to a distance measurement sensor having a structure in which a charge is distributed to two charge storage units, such as a distance measurement sensor having a current-assisted photon demodulator (CAPD) structure or a gate-type distance measurement sensor in which pulses of the charge of the photodiode are alternately applied to two gates. In addition, the present technology can be applied to a distance measurement sensor of a structured light method.

[0222] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0223] As long as there is no contradiction, each of the multiple present technologies described in this specification can be implemented independently. Needless to say, multiple arbitrary present technologies can be implemented in combination. For example, part or all of the present technology described in any embodiment can be combined with part or all of the present technology described in another embodiment to implement. In addition, part or all of any of the above-mentioned present technologies can be combined with another technology not described above to implement.

[0224] In addition, for example, a configuration described as one device (or processing unit) may be divided into a plurality of devices (or processing units) and configured as a plurality of devices (or processing units). Conversely, a configuration previously described as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Furthermore, it goes without saying that configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a portion of the configuration of a specific device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0225] In addition, in this specification, a system means a collection of multiple components (devices, modules (components), etc.), and it is not important whether all the components are the same. Therefore, each of multiple devices housed in separate housings and connected via a network, and one device in which multiple modules are housed in one housing are a system.

[0226] In addition, for example, the above-mentioned program can be executed in an arbitrary device. In this case, it is sufficient that the device has necessary functions (functional blocks, etc.) and can obtain necessary information.

[0227] Note that the present technology can be configured as follows. (1)

[0229] Distance measurement sensors, including

[0230] a distance measuring unit that calculates distance information to the object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting the irradiation light emitted from a predetermined light emitting source by the object,

[0231] an ambient light calculation unit that calculates an ambient light component included in a signal obtained by the light receiving unit, and

[0232] The outdoor information calculation unit calculates the outdoor information based on the ambient light component. (2)

[0234] The distance measuring sensor according to (1) further includes

[0235] A normalization unit normalizes the ambient light component calculated by the ambient light calculation unit.

[0236] The outdoor information calculation unit estimates the outdoor information based on the normalized ambient light component. (3)

[0238] The distance measuring sensor according to (2),

[0239] The normalization unit normalizes the ambient light component by using the exposure time and the number of pixels. (4)

[0241] The distance measuring sensor according to (2),

[0242] The normalization unit normalizes the ambient light component by using the exposure time, the number of pixels and the distance information. (5)

[0244] The distance measuring sensor according to any one of (1) to (4),

[0245] The ambient light calculation unit calculates the ambient light component by subtracting the dark current component from the signal obtained by the light receiving unit. (6)

[0247] The distance measuring sensor according to any one of (1) to (5),

[0248] Therein, both the distance information and the outdoor information are calculated by using the signal obtained by the light receiving unit. (7)

[0250] The distance measuring sensor according to any one of (1) to (6), wherein as an operation mode, there is provided

[0251] A first operation mode for calculating distance information and outdoor information,

[0252] a second mode of operation in which distance information is calculated without outdoor information, or

[0253] A third operation mode in which outdoor information is calculated but distance information is not calculated. (8)

[0255] The distance measuring sensor according to any one of (1) to (7),

[0256] The ambient light calculation unit calculates an ambient light component of an area of ​​interest which is a part of a light receiving area of ​​the light receiving unit. (9)

[0258] The distance measuring sensor according to (8),

[0259] The ambient light calculation unit obtains region information indicating the region of interest and calculates the ambient light component of the region of interest. (10)

[0261] The distance measuring sensor according to (9),

[0262] The ambient light calculation unit acquires region information indicating the region of interest, which has been detected by an imaging sensor that generates a captured image obtained by receiving RGB light, and calculates an ambient light component of the region of interest. (11)

[0264] The distance measuring sensor according to (8),

[0265] In addition to the distance information, the distance measurement unit also calculates the confidence information.

[0266] The distance measuring sensor further includes a region detecting unit that detects the region of interest by using at least one of the distance information or the confidence information, and

[0267] The ambient light calculation unit calculates the ambient light component of the region of interest. (12)

[0269] The distance measuring sensor according to any one of (1) to (11), further comprising

[0270] A filtering unit performs a predetermined filtering process on the distance information.

[0271] The filtering unit performs a predetermined filtering process based on the outdoor information. (13)

[0273] The distance measuring sensor according to any one of (1) to (12), further comprising

[0274] The control unit controls the exposure time based on the signal obtained by the light receiving unit. (14)

[0276] The distance measuring sensor according to any one of (1) to (13),

[0277] The outdoor information calculation unit calculates whether the outdoor information represents outdoor or indoor as the outdoor information. (15)

[0279] The distance measuring sensor according to any one of (1) to (13),

[0280] The outdoor information calculation unit calculates the probability of being outdoors as the outdoor information. (16)

[0282] Signal processing methods, including

[0283] calculating distance information to the object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting irradiation light emitted from a predetermined light emitting source by the object, by using a distance measuring sensor;

[0284] calculating, by using the distance measuring sensor, an ambient light component included in a signal obtained by the light receiving unit; and

[0285] By using a distance measurement sensor, outdoor information is calculated based on the ambient light component. (17)

[0287] Distance measurement module, including

[0288] a predetermined light source, and

[0289] Distance measurement sensor,

[0290] Distance measurement sensors include

[0291] a distance measuring unit that calculates distance information to the object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting the irradiation light emitted from a predetermined light emitting source by the object,

[0292] an ambient light calculation unit that calculates an ambient light component included in a signal obtained by the light receiving unit, and

[0293] The outdoor information calculation unit calculates the outdoor information based on the ambient light component.

[0294] Reference Mark List

[0295] 11 Distance measurement module

[0296] 12 Light Source

[0297] 13 Lighting control unit

[0298] 14 Distance measurement sensor

[0299] 21 Light receiving unit

[0300] 22 Distance measurement unit

[0301] 23 Ambient light calculation unit

[0302] 24 Ambient light normalization units

[0303] 25 Outdoor information computing unit

[0304] 26 filter units

[0305] 81 Subject area detection unit

[0306] 82 Ambient light calculation unit

[0307] 91 Area

[0308] 101 Imaging Sensor

[0309] 111 Light receiving unit

[0310] 112 Signal Processing Unit

[0311] 121 De-mosaic processing unit

[0312] 122 ROI determination unit

[0313] 123 filter unit

[0314] 201 Smartphone

[0315] 202 Distance Measurement Module

[0316] 301 CPU

[0317] 302 ROOMS

[0318] 303 RAM

Claims

1. Distance measurement sensor, including: a distance measuring unit that calculates distance information to the object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting the irradiation light emitted from a predetermined light emitting source by the object; an ambient light calculation unit that calculates an ambient light component included in the signal obtained by the light receiving unit; as well as an outdoor information calculation unit, which calculates outdoor information based on the ambient light component, The outdoor information calculation unit calculates whether the outdoor information represents outdoor or indoor as the outdoor information. Wherein, the distance measurement sensor further includes: a normalization unit, which normalizes the ambient light component calculated by the ambient light calculation unit, The outdoor information calculation unit estimates the outdoor information based on the normalized ambient light component.

2. The distance measuring sensor according to claim 1, in, The normalization unit normalizes the ambient light component by using an exposure time and a number of pixels.

3. The distance measuring sensor according to claim 1, in, The normalization unit normalizes the ambient light component by using an exposure time, a number of pixels, and the distance information.

4. The distance measuring sensor according to claim 1, in, The ambient light calculation unit calculates the ambient light component by subtracting a dark current component from the signal obtained by the light receiving unit.

5. The distance measuring sensor according to claim 1, in, Both the distance information and the outdoor information are calculated by using the signal obtained by the light receiving unit.

6. The distance measuring sensor according to claim 1, wherein: As an operating mode, it provides a first operation mode of calculating the distance information and the outdoor information, a second operation mode in which the distance information is calculated without calculating the outdoor information, or A third operation mode in which the outdoor information is calculated without calculating the distance information.

7. The distance measuring sensor according to claim 1, in, The ambient light calculation unit calculates the ambient light component of an area of ​​interest which is a part of a light receiving area of ​​the light receiving unit.

8. The distance measuring sensor according to claim 7, in, The ambient light calculation unit acquires region information indicating the region of interest and calculates the ambient light component of the region of interest.

9. The distance measuring sensor according to claim 8, in, The ambient light calculation unit acquires region information indicating the region of interest, the region information having been detected by an imaging sensor that generates a captured image obtained by receiving RGB light, and calculates the ambient light component of the region of interest.

10. The distance measuring sensor according to claim 7, in, In addition to the distance information, the distance measurement unit also calculates confidence information. The distance measuring sensor further includes a region detecting unit that detects the region of interest by using at least one of the distance information or the confidence information, and The ambient light calculation unit calculates the ambient light component of the region of interest.

11. The distance measuring sensor according to claim 1, further comprising A filtering unit performs a predetermined filtering process on the distance information. The filtering unit performs the predetermined filtering process based on the outdoor information.

12. The distance measuring sensor according to claim 1, further comprising A control unit controls an exposure time based on the signal obtained by the light receiving unit.

13. The distance measuring sensor according to claim 1, in, The outdoor information calculation unit calculates a probability of being outdoors as the outdoor information.

14. A signal processing method comprising: calculating distance information to the object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting irradiation light emitted from a predetermined light emitting source by the object, by using a distance measuring sensor; calculating, by using the distance measuring sensor, an ambient light component included in the signal obtained by the light receiving unit; as well as calculating outdoor information based on the ambient light component by using the distance measuring sensor, Wherein, the outdoor information indicates whether the outdoor information is outdoor or indoor, wherein the method further comprises: The ambient light component is normalized, The outdoor information is estimated based on the normalized ambient light component.

15. Distance measurement module, including: A predetermined light source; as well as Distance measurement sensor, The distance measuring sensor comprises a distance measuring unit that calculates distance information to an object based on a signal obtained by a light receiving unit that receives reflected light obtained by reflecting the irradiation light emitted from the predetermined light source by the object, an ambient light calculation unit that calculates an ambient light component included in the signal obtained by the light receiving unit, and an outdoor information calculation unit, which calculates outdoor information based on the ambient light component, The outdoor information calculation unit calculates whether the outdoor information represents outdoor or indoor as the outdoor information. Wherein, the distance measurement sensor also includes a normalization unit, which normalizes the ambient light component calculated by the ambient light calculation unit, The outdoor information calculation unit estimates the outdoor information based on the normalized ambient light component.

Images