Optical device, optical system, movable body, and distance measuring method

Abstract

To provide a technique advantageous for improving measuring accuracy.SOLUTION: An optical device comprises a distance sensor and obtains distance information on the basis of output from the distance sensor. The optical device comprises a processor that, on the basis of the movement of the distance sensor in an optical axis direction of the distance sensor, performs reduction processing of reducing the influence of the movement on the distance information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an optical device, an optical system, a moving object, and a distance measuring method. [Background technology]

[0002] A Time of Flight (ToF) distance measuring device is known, which measures distance by emitting light from a light source and detecting the light reflected by an object. Patent Document 1 describes that a distance measurement frame for obtaining a distance image is composed of multiple subframes, and each subframe is composed of multiple microframes. The pixels constituting the distance measurement frame have pixel values ​​indicating the distance from the distance measuring device to the object. This distance is calculated from the time from when the distance measuring device emits light to when it receives reflected light. The multiple subframes have different measurement periods for measuring the reflected light. The pixels in each subframe have a pixel value corresponding to the number of photons received during the measurement period (subframe period) of that subframe, and the pixel value is expressed in multiple bits. Each pixel in the multiple microframes constituting each subframe has a pixel value indicating whether or not a photon was received during the measurement period (microframe period) assigned to that microframe, and the pixel value is expressed in one bit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-174180 Summary of the Invention [Problem to be solved by the invention]

[0004] In an optical device that obtains distance information using a distance sensor, if the distance sensor vibrates or moves in the optical axis direction (the direction in which the distance is measured), the measurement accuracy may decrease.

[0005] An object of the present invention is to provide a technique that is advantageous for improving measurement accuracy. [Means for solving the problem]

[0006] One aspect of the present invention relates to an optical device, the optical device may include a distance sensor, and may be configured to obtain distance information based on an output of the distance sensor. The optical device may also include a processor that performs a reduction process to reduce an effect of movement of the distance sensor in an optical axis direction of the distance sensor on the distance information. [Effects of the Invention]

[0007] According to the present invention, a technique that is advantageous for improving measurement accuracy is provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an optical device according to a first embodiment. [Figure 2] FIG. 4 is a diagram illustrating an example of operation timing of the ToF method in the optical device according to the first embodiment. [Figure 3] 5A to 5C are diagrams illustrating the operation of the optical device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a comparative example for explaining the operation of the optical device according to the first embodiment. [Figure 5] 5A to 5C are diagrams illustrating the operation of the optical device according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating the configuration of an optical device according to a second embodiment. [Figure 7] FIG. 10 is a diagram illustrating the configuration of an optical device according to a third embodiment. [Figure 8] 10A to 10C are diagrams illustrating the operation of the optical device according to the third embodiment. [Figure 9] FIG. 10 is a diagram illustrating the configuration of an optical device according to a fourth embodiment. [Figure 10A] 10A to 10C are diagrams illustrating the operation of the optical device according to the fourth embodiment. [Figure 10B] 10A to 10C are diagrams illustrating the operation of the optical device according to the fourth embodiment. [Figure 11A]FIG. 10 is a diagram showing a comparative example for explaining the operation of the optical device according to the fourth embodiment. [Figure 11B] FIG. 10 is a diagram showing a comparative example for explaining the operation of the optical device according to the fourth embodiment. [Figure 12] 10A to 10C are diagrams illustrating the operation of the optical device according to the fourth embodiment. [Figure 13] FIG. 10 is a diagram illustrating the configuration of an optical device according to a fifth embodiment. [Figure 14] FIG. 10 is a diagram illustrating the configuration of a fifth embodiment. [Figure 15] 4 is a flowchart showing the operation of the optical device according to the first embodiment. [Figure 16] FIG. 1 is a diagram showing an example of the configuration of an image sensor that can be incorporated into a distance sensor. [Figure 17] FIG. 2 is a diagram showing an example of the configuration of a pixel of an image sensor. [Figure 18] FIG. 1 is a diagram showing an example of the configuration of an optical system and a moving object according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] The configuration and operation of an optical device 100 according to a first embodiment will be described with reference to FIGS. 1, 2, 3, 4, 5, and 15. The optical device 100 is configured to obtain distance information indicating the distance between the optical device 100 and an object 110. The optical device 100 may be understood as a distance measuring device. The object 110 may be any object that reflects light. While FIG. 1 shows a single object 110, the optical device 100 may be configured to obtain distance information for multiple objects within a field of view (a measurement area). The optical device 100 may include a distance sensor 102 and may be configured to obtain distance information based on an output from the distance sensor 102. The optical device 100 may include a processor 120 that performs a reduction process to reduce the effect of movement (e.g., the amount of movement) of the distance sensor 102 in the optical axis direction AX of the distance sensor 102 on the distance information.

[0011] The optical device 100 may also include a light-emitting unit 101 that emits light in the optical axis direction of the distance sensor 102. When an object 110 is present within the field of view of the optical device 100, the light emitted from the light-emitting unit 101 may be reflected by the object 110 and enter the distance sensor 102 as reflected light. The optical axis direction of the distance sensor 102 is the direction in which the distance between the optical device 100 and the object 110 is measured. The distance sensor 102 may include a photoelectric conversion device such as an image sensor and an optical system OPT that forms an image of the reflected light on the light-receiving surface of the photoelectric conversion device. The optical axis direction AX may be understood as the optical axis direction of the optical system OPT. The optical system OPT may be replaceable. All or a part of the optical system OPT may be shared with an optical system for the light-emitting unit 101 to emit light into the field of view of the optical device 100.

[0012] The optical device 100 may include a detection unit 106 that detects movement (e.g., movement amount) of the optical device 100 or the distance sensor 102 in the optical axis direction AX and outputs movement data MD indicating the movement. The detection unit 106 may include at least one of an acceleration sensor and a displacement sensor. If the distance sensor 102 vibrates in the optical axis direction AX, the movement of the distance sensor 102 in the optical axis direction AX may include the vibration. If the distance sensor 102 moves in a direction intersecting the optical axis direction AX, the movement of the distance sensor 102 in the optical axis direction AX refers to a component parallel to the optical axis direction AX. The processor 120 may perform a reduction process based on the output (movement data MD) of the detection unit 106 to reduce the effect of the movement of the distance sensor 102 in the optical axis direction AX on the distance information DD. The processor 120 may generate the distance information DD by correcting information obtained by the output of the distance sensor 102 based on an output indicating the detection result by the detection unit 106 during a measurement period in which the distance sensor 102 performs measurements.

[0013] A more specific configuration example will be described below. The optical device 100 may include a communication IF (interface) unit 107 for communicating with other devices. The optical device 100 may receive control information and transmit distance information DD via the communication IF unit 107. The processor 120 may include, for example, a register unit 104, a timing control unit 103, and a signal processing unit 105. The signal processing unit 105 may include a memory 108 that stores multiple subframes SFD output from the distance sensor 102. The signal processing unit 105 may also include a calculation unit 109 that generates distance information DD based on the multiple subframes SFD stored in the memory 108 and the output (movement data MD) of the detection unit 106. The calculation unit 109 may perform a reduction process to reduce the influence of movement of the distance sensor 102 in the optical axis direction AX on the distance information, based on the output (movement data MD) of the detection unit 106. In other words, the calculation unit 109 corrects the information obtained from the multiple subframes SFD held in the memory 108 based on the output (movement data MD) of the detection unit 106, and generates distance information DD based on the corrected information.

[0014] The light emitting unit 101 may include a light source that emits light in response to a light emission control signal LE, which is a first timing control signal generated by the timing control unit 103. Note that the timing control unit 103 generating the light emission control signal LE is equivalent to the timing control unit 103 activating the light emission control signal LE. The light source may be, for example, a semiconductor laser diode. The light source may emit light having a predetermined pulse width in response to the light emission control signal LE supplied from the timing control unit 103. The light emitting unit 101 may include an optical member (not shown), such as a diffuser, so that the light emitted from the light source may be diffused and irradiated in a predetermined two-dimensional range.

[0015] The distance sensor 102 has one or more photoelectric conversion elements and can output multiple subframes (subframe data) SFDs with different measurement periods. A ranging frame (ranging frame data) is generated based on the multiple subframes. Here, the measurement period is the exposure period of the photoelectric conversion elements, i.e., the period during which a signal is generated by photoelectric conversion. The pixels of each subframe have a pixel value corresponding to the amount of light (e.g., the number of photons) received during the measurement period of that subframe, and the pixel value can be expressed by multiple bits. The exposure period of the photoelectric conversion elements during the subframe period for acquiring each subframe can be controlled by an exposure control signal EX, which is a second timing control signal generated by the timing control unit 103. Note that generating the exposure control signal EX by the timing control unit 103 is equivalent to activating the exposure control signal EX by the timing control unit 103. The distance sensor 102 can include, for example, a CMOS sensor or a SPAD sensor. For example, by arranging a plurality of photoelectric conversion elements two-dimensionally, such as in a CMOS image sensor or a SPAD image sensor, it is possible to obtain two-dimensional distance information, that is, a distance image.

[0016] The timing control unit 103 activates the exposure control signal EX multiple times based on the activation timing of the light emission control signal LE during a ranging frame period for generating ranging frames. The register unit 104 can store control information for controlling the operation of the optical device 100.

[0017] The memory 108 stores or holds multiple subframes SFD output from the distance sensor 102. The calculation unit 109 corrects information obtained from the multiple subframes SFD stored in the memory 108 based on the output (movement data MD) of the detection unit 106, and generates distance information DD based on the corrected information. The following describes an example in which the distance sensor 102 has multiple photoelectric conversion elements and outputs an image having pixel values ​​corresponding to the amount of light received by each pixel (e.g., the number of photons) during each subframe period. In each subframe, a pixel having a pixel value equal to or greater than a predetermined value indicates that it received light emitted from the light-emitting unit 101 and reflected by the object 110 during the exposure period for that subframe. The calculation unit 109 can generate distance information DD indicating the distance between the optical device 100 and the object 110 based on the time difference between the light-emission period and the exposure period during each subframe. This method is called a ToF (Time of Flight) method.

[0018] FIG. 2 is a diagram illustrating the operation timing of the ToF system in the optical device 100 of the first embodiment. In FIG. 2, a period T1 for obtaining each of the ranging frames F1, F2, etc. is a ranging frame period. The ranging frame period is made up of multiple (here, m) subframe periods T2. In each of the multiple subframe periods T2, subframes SF1, SF2, etc. are obtained. The ranging frames F1, F2, etc. are also collectively referred to as ranging frames F, and the subframes SF1, SF2, etc. SFm are also collectively referred to as subframes SF. A ranging frame F from which one piece of distance information is generated is generated based on multiple subframes SF.

[0019] Each subframe period T2 defines the temporal relationship between the light emission timing of the light-emitting unit 101 and the exposure period (measurement period) during which the distance sensor 102 performs an exposure operation. Therefore, the distance sensor 102 operates at a timing corresponding to the light emission timing of the light-emitting unit 101. Furthermore, the pixel (pixel signal) of each subframe SF has a pixel value corresponding to the amount of light (e.g., the number of photons) received during the exposure period (measurement period) of that subframe SF, and the pixel value can be expressed by multiple bits. Each subframe SF can be generated based on multiple microframes. More specifically, the pixel value of each pixel in each subframe SF can be generated by calculating the sum of the pixel values ​​of multiple microframes for each pixel. The pixel in each microframe has a pixel value indicating whether or not a unit amount of light (e.g., photons) has been received during the measurement period (microframe period) assigned to that microframe, and the pixel value can be expressed by one bit.

[0020] 15 is a flowchart showing the operation of the optical device 100 of the first embodiment. This operation can be understood as a distance measurement method for measuring distance using the distance sensor 102. In step S1501, control information for setting the light emission timing of the light emitting unit 101 (i.e., the timing for activating the light emission control signal LE) and the exposure timing (i.e., the timing for activating the exposure control signal EX) can be stored in the register unit 104.

[0021] Steps S1502 to S1509 are steps for acquiring one or more ranging frames F. Steps S1502 to S1505 are steps for acquiring one sub-frame SF. In step S1502, the timing control unit 103 activates the light emission control signal LE, thereby causing the light emitting unit 101 to emit light. In step S1503, the timing control unit 103 activates the exposure control signal EX for a set exposure period (measurement period), thereby causing the distance sensor 102 to perform an exposure operation. In step S1504, the signal processing unit 105 acquires a sub-frame SFD, which is data of one sub-frame SF, from the distance sensor 102 and stores it in the memory 108. In step S1505, the signal processing unit 105 acquires from the detection unit 106 the detection result (movement data MD) by the detection unit 106 during the exposure period (measurement period), and stores it in the calculation unit 109, for example. Note that steps S1504 and S1505 may be performed in parallel. In step S1506, the signal processing unit 105 determines whether all subframes SF for generating one ranging frame F have been acquired. If acquisition of all subframes SF has not been completed, steps S1502 to S1505 are further performed. On the other hand, if acquisition of all subframes SF has been completed, step S1507 is performed. In step S1507, the signal processing unit 105 (calculation unit 109) generates distance information (second distance information) based on the plurality of subframe data SFD (first distance information) stored in memory 108 and the detection results by the detection unit 106 corresponding to each of the plurality of subframes SF. Step S1507 may be performed after steps S1504 and S1505 and before step S1506.

[0022] In step S1509, the signal processing unit 105 determines whether all ranging frames F have been acquired, and executes steps S1502 to S1508 if acquisition of all ranging frames F has not been completed. If acquisition of all ranging frames F has been completed, the signal processing unit 105 ends the operation shown in FIG.

[0023] Step S1504 may be understood as an example of an acquisition step of acquiring first distance information using the distance sensor during a certain period of time. Step S1505 may be understood as an example of a detection step of detecting movement of the distance sensor AX in the optical axis direction AX of the distance sensor 102 during that period of time. Steps S1507 and S1508 may be understood as generation steps of generating second distance information based on the first distance information acquired in the acquisition step and the results detected in the detection step.

[0024] FIG. 3 schematically illustrates the operation of the optical device 100 when the optical device 100 does not move in the optical axis direction AX during the period when the ranging frames are acquired. FIG. 3(a) illustrates the positional relationship between the optical device 100 and the object 110 in the sub-frames SF1 to SF8. In this example, the optical device 100 does not move in the optical axis direction AX in the sub-frames SF1 to SF8. Distance D is the distance between the optical device 100 and the object 110. FIG. 3(b) illustrates the light emission timing of the light-emitting unit 101, the incident timing of reflected light entering the distance sensor 102, and the exposure timing of the distance sensor 102 in each of the sub-frames SF1 to SF8. In FIG. 3(b), "emitted light" indicates the light emission timing of the light-emitting unit 101, and "reflected light" indicates the incident timing of reflected light from the object 110 entering the distance sensor 102. 3(b), in the bars shown to the right of "SF1" to "SF8," which indicate the subframe periods for acquiring subframes SF1 to SF8, white bars indicate exposure periods and black bars indicate non-exposure periods. Also, "Δt" indicates the time difference between the light emission timing of the light emitting unit 101 and the timing at which the reflected light is incident on the distance sensor 102 by the timing control unit 103. Also, "signal value" indicates the pixel value of a pixel in subframes SF1 to SF8, and "distance" is a value obtained by converting the exposure period of subframes SF1 to SF8 into a distance.

[0025] In the example of FIG. 3, the distance sensor 102 detects reflected light from the object 110 during the period for acquiring subframes SF5 and SF6. The signal values ​​in subframes SF5 and SF6 are greater than the signal values ​​in the other subframes. The calculation unit 109 can determine the distance d300 between the optical device 100 and the object 110 based on the signal values ​​in subframes SF1 to SF8. For example, the calculation unit 109 can determine the rising positions of the signal values ​​in subframes SF1 to SF8 and identify the subframe in which the object 110 was detected (subframe SF5 in this example) from the rising positions. The calculation unit 109 can then determine the distance (distance information) from the center position of the exposure period of that subframe.

[0026] FIG. 4 schematically illustrates an operation (comparative example) of the optical device 100 when the optical device 100 moves in the optical axis direction AX during acquisition of distance measurement frames but does not perform the reduction process (correction process). FIG. 4(a) illustrates the positional relationship between the optical device 100 and the object 110 in subframes SF1 to SF8. In this example, as indicated by movement amounts dz1 to dz8, the optical device 100 moves in the optical axis direction AX in subframes SF1 to SF8. In this example, dz1 and dz7 are 0. Movement of the optical device 100 can be caused by, for example, hand shake, shaking of a platform on which the optical device 100 is mounted, or movement of a moving object such as a vehicle on which the optical device 100 is mounted. In this example, the distance between the optical device 100 and the object 110 is closest in subframe SF4. The notation method in FIG. 4(b) is the same as that in FIG. 3(b). 4(b), as the optical device 100 moves in the optical axis direction AX, the time difference between the timing at which the light emitting unit 101 emits light and the timing at which the reflected light enters the distance sensor 102 shifts from Δt. dt2 to dt6 and dt8 are values ​​obtained by converting the amounts of movement dz2 to dz6 and dz8, respectively, into time (values ​​obtained by dividing the amount of movement by the luminous flux).

[0027] Unlike the example in Fig. 3, in the example in Fig. 4, reflected light is detected by the distance sensor 102 in subframe SF4 as well, and a significant signal value is obtained. When the distance (distance information) is determined based on the subframe SF in which the signal value rises, the result is distance d400. Compared to distance d300, an error occurs in the distance measurement result by the distance equivalent to one subframe.

[0028] Fig. 5 schematically shows the reduction process (correction process) in the optical device 100 of the first embodiment. The positional relationship between the optical device 100 and the object 110 in the sub-frames SF1 to SF8 is the same as the example shown in Fig. 4(a). The light emission timing of the light-emitting unit 101, the exposure timing of the distance sensor 102, and the incidence timing of reflected light on the distance sensor in each of the sub-frames SF1 to SF8 are the same as the example shown in Fig. 4(a).

[0029] The detection unit 106 detects movement (e.g., movement amount) of the distance sensor 102 in the optical axis direction AX during the exposure period (measurement period) of the distance sensor 102, and can provide movement data MD (dz1 to dz8) indicating the detection results to the calculation unit 109. The calculation unit 109 generates distance information (second distance information) based on the data of the plurality of subframes SF1 to SF8 (first distance information) stored in the memory 108 and the detection results (dz1 to dz8) by the detection unit 106 corresponding to the plurality of subframes SF1 to SF8, respectively. The calculation unit 109 obtains corrected distance values, for example, by adding movement amounts dz2 to dz8 based on the movement amount dz1 during the period required to obtain the subframe SF1 to the distance obtained from each subframe SF. For example, if a signal value rises in subframe SF4, indicating that object 110 was detected in subframe SF4, the movement amount dz4 detected when subframe SF4 was acquired is added to the distance d500 corresponding to subframe SF4. The final distance (corrected distance) obtained is d501. This cancels out or reduces the influence (error factor) of the movement amount of optical device 100 in the optical axis direction AX, thereby reducing distance measurement errors. For example, assuming that the exposure period of each subframe SF is 600 ps (equivalent to 9 cm) and the shift in exposure time for each subframe SF is 300 ps (equivalent to 4.5 cm), a movement amount of a few centimeters in amplitude, equivalent to camera shake or body shaking, can be corrected in SF4.

[0030] While the above example assumes that the optical device moves continuously in the optical axis direction over multiple subframe periods, the reduction process is also useful when the optical device moves intermittently, for example, only during one subframe period. In the above example, the movement amount of the optical device in the optical axis direction is calculated based on subframe SF1, but any subframe SF may be used as the reference for the calculation. The photoelectric conversion element of the distance sensor 102 is not limited to a specific photoelectric conversion element. However, using a SPAD sensor including an avalanche photodiode is advantageous in that it is essentially free of readout noise, thereby shortening the subframe period required to ensure the same accuracy compared to using other sensors such as a CMOS sensor. Therefore, an optical device using a SPAD sensor is advantageous in shortening the ranging frame period, and when the movement speed of the optical device in the optical axis direction is the same, the movement amount data required for correction can be reduced.

[0031] An optical device 100 according to a second embodiment will be described below with reference to FIG. 6. Matters not mentioned in the description of the second embodiment may follow those of the first embodiment. FIG. 6 illustrates the configuration of the optical device 100 according to the second embodiment. The optical device 100 according to the second embodiment may include a memory 600 that stores the output of the detection unit 106 (movement amount data indicating the amount of movement of the optical device 100 or the distance sensor 102 in the optical axis direction AX). The processor 120 may perform a reduction process based on the output of the detection unit 106 stored in the memory 600. For example, after acquiring multiple subframes for generating a ranging frame, the processor 120 may generate corrected distance information based on the multiple subframes stored in the memory 108 and the movement amount data stored in the memory 600. The second embodiment is advantageous, for example, when there is a large delay in the data output from the detection unit 106.

[0032] An optical device 100 according to a third embodiment will be described below with reference to FIGS. 7 and 8. Matters not mentioned in the description of the third embodiment may follow those of the first embodiment. FIG. 7 shows the configuration of the optical device 100 according to the third embodiment. In the third embodiment, the detection unit 106 provides its output (movement data MD) to the processor 120 (timing control unit 103). The processor 120 (timing control unit 103) sends second timing signals to the distance sensor 102, which respectively control multiple measurement periods for acquiring multiple sub-frames SF, based on the output (movement data MD) of the detection unit 106. More specifically, in the example shown in FIG. 7, the detection unit 106 provides its output (movement data MD) to the timing control unit 103. Furthermore, the timing control unit 103 sends an exposure control signal EX to the distance sensor 102 as a second timing signal, which respectively controls multiple measurement periods for acquiring multiple sub-frames SF, based on the output (movement data MD) of the detection unit 106.

[0033] FIG. 8 schematically illustrates the reduction process (correction process) in the optical device 100 of the third embodiment. FIG. 8(a) illustrates the positional relationship between the optical device 100 and the object 110 in sub-frames SF1 to SF8. In this example, the optical device 100 moves in the optical axis direction AX in sub-frames SF1 to SF8, as indicated by movement amounts dz1 to dz8. In this example, dz1 and dz7 are 0. The notation method in FIG. 8(b) is the same as that in FIG. 3(b). As shown in FIG. 8(b), as the optical device 100 moves in the optical axis direction AX, the time difference between the timing at which the light-emitting unit 101 emits light and the timing at which the reflected light enters the distance sensor 102 shifts from Δt. dt2 to dt6 and dt8 are values ​​obtained by converting the movement amounts dz2 to dz6 and dz8, respectively, into time (values ​​obtained by dividing the movement amounts by the luminous flux). The exposure periods indicated by the white bars are adjusted by dt2 to dt6 and dt8 relative to the exposure periods when the correction amount is set to 0.

[0034] The distance measurement method performed in the optical device 100 of the third embodiment may include a measurement step and a generation step. In the measurement step, the distance sensor 102 may perform measurements at multiple measurement timings determined based on the movement of the distance sensor 102 in the optical axis direction AX of the distance sensor 102. In the generation step, distance information may be generated based on the output of the distance sensor 102 in the measurement step.

[0035] In the example of FIG. 8, the distance sensor 102 detects reflected light from the object 110 during the period for acquiring subframes SF5 and SF6, and the signal values ​​in subframes SF5 and SF6 are larger than those in the other subframes. The calculation unit 109 can determine the distance d800 between the optical device 100 and the object 110 based on the signal values ​​in subframes SF1 to SF8. For example, the calculation unit 109 can determine the rising positions of the signal values ​​in subframes SF1 to SF8 and identify the subframe in which the object 110 was detected (subframe SF5 in this example) from the rising positions. The calculation unit 109 can then determine the distance (distance information) based on the center position of the exposure period of that subframe. In the example shown in FIG. 8, even though the optical device 100 is moving in the optical axis direction, the distance sensor 102 detects reflected light only in subframes SF5 and SF6, similar to the example of FIG. 3, which illustrates an example in which the optical device 100 moves in the optical axis direction. This makes it possible to cancel out or reduce the influence (error factor) of the amount of movement of the optical device 100 in the optical axis direction AX, thereby reducing distance measurement errors.

[0036] The exposure period may be adjusted at a predetermined cycle (for example, at a certain timing within each exposure period when the correction amount is set to 0), or when calculation of dt1 to dt8 is completed, or at another timing. The same adjustment amount may be applied to at least one subframe period. The same adjustment amount may be applied to all subframe periods for one ranging frame.

[0037] The configuration and operation of the optical device 100 of the fourth embodiment will be described below with reference to FIGS. 9, 10A, 10B, 11A, 11B, and 12. Matters not mentioned in the description of the fourth embodiment may conform to at least one of the first to third embodiments. FIG. 9 shows the configuration of the optical device 100 of the fourth embodiment. The optical device 100 of the fourth embodiment includes a second detection unit 112 that detects information regarding at least one of the movement of the distance sensor 102 in a direction perpendicular to the optical axis direction AX and the rotation of the distance sensor 102 (such as the tilt with respect to the optical axis direction). The processor 120 can generate distance information DD by correcting information obtained from the output of the distance sensor 102 based on the outputs of the first detection unit 106 and the second detection unit 112 during a measurement period in which the distance sensor 102 performs measurement. The processor 120 or the signal processing unit 105 may include, for example, a correction unit 111 that performs shake correction on a plurality of subframe SFDs stored in the memory 108 based on the output of the second detection unit 112. The calculation unit 109 may perform the aforementioned reduction process on the plurality of subframes for which shake correction has been performed by the correction unit 111. The second detection unit 112 may include, for example, a gyro sensor and a circuit that processes the output of the gyro sensor.

[0038] 10A and 10B schematically illustrate the operation of the optical device 100 when no information is detected by the second detection unit 112 during the period when a ranging frame is acquired. In the following explanation, for simplicity, it is assumed that there are six subframes SF and two pixels in the distance sensor 102 of the optical device 100. Also, for simplicity, it is assumed that there is no movement of the optical device 100 (distance sensor 102) in the optical axis direction AX. FIG. 10A illustrates the positional relationship between the pixel groups and the target object 910 in subframes SF1 to SF6. Here, it is assumed that an object O_a is located in the area viewed by pixel P_a, an object O_b is located in the area viewed by pixel P_b, and the distances between the optical device 100 and the objects O_a and O_b are D_a and D_b, respectively. First, consider the case where no information is detected by the second detection unit 112 over six subframe periods.

[0039] The left column of FIG. 10B illustrates the light-emitting timing of the light-emitting unit 101 in each of subframes SF1 to SF6, the timing at which reflected light enters pixel P_a, the exposure timing of pixel P_a, and the signal value of pixel P_a. Pixel P_a detects reflected light in subframe SF6. The right column of FIG. 10B illustrates the light-emitting timing of the light-emitting unit 101 in each of subframes SF1 to SF6, the timing at which reflected light enters pixel P_b, the exposure timing of pixel P_b, and the signal value of pixel P_b. Pixel P_b detects reflected light in subframes SF4 and SF5. For pixel P_a, the calculation unit 109 identifies the subframe in which object 110 was detected from the rising position (subframe SF6 in this example), and can determine the distance (distance information) d1000 based on the center position of the exposure period of that subframe. For pixel P_b, the calculation unit 109 can identify the subframe (in this example, subframe SF4) in which the object 110 was detected from the rising position, and determine the distance (distance information) d1001 based on the center position of the exposure period of that subframe.

[0040] 11A and 11B schematically show an operation (comparative example) of the optical device 100 in a case where the second detection unit 112 detects information during the period when a ranging frame is acquired, but the correction unit 111 does not perform correction. FIG. 11A illustrates the positional relationship between the pixel groups and the object 910 in subframes SF1 to SF6. Compared to FIG. 10A, it is assumed that the second detection unit 112 detects that the optical device 100 has moved in a direction perpendicular to the optical axis direction AX only in subframe SF4. It is assumed that the amount of movement of the optical device 100 is equal to or less than the size of the light-receiving area of ​​the distance sensor 102.

[0041] The left column of FIG. 11B illustrates the light-emitting timing of the light-emitting unit 101 in each of subframes SF1 to SF6, the timing at which reflected light enters pixel P_a, the exposure timing of pixel P_a, and the signal value of pixel P_a. Compared to FIG. 10B, pixel P_a also detects light in subframe SF4. The right column of FIG. 11B illustrates the light-emitting timing of the light-emitting unit 101 in each of subframes SF1 to SF6, the timing at which reflected light enters pixel P_b, the exposure timing of pixel P_b, and the signal value of pixel P_b. Pixel P_b does not detect reflected light in subframe SF4. This is because object O_b has entered the viewing area of ​​pixel P_a due to at least one of the movement and rotation of the optical device 100. As a result, when the distance value is calculated from the rising position, the estimated distance value is distance d1100 for pixel P_a, and distance d1101 for pixel P_b, resulting in an error in the distance value to be estimated.

[0042] The middle part of Fig. 12 shows a schematic diagram of the operation (comparative example) of the optical device 100 when information is detected by the second detection unit 112 during the period when a ranging frame is acquired, but correction is not performed by the correction unit 111 based on this information. The bottom part of Fig. 12 shows a schematic diagram of the operation of the optical device 100 when information is detected by the second detection unit 112 during the period when a ranging frame is acquired, and correction is performed by the correction unit 111 based on this information.

[0043] In the lower part of Fig. 12, the signal value (black circle) of pixel P_b in sub-frame SF4 illustrates the result of correction performed by the correction unit 111. Also, in the lower part of Fig. 12, the signal value of pixel P_a in sub-frame SF4 has no signal value to replace, so a method of replacing it with the average value of the values ​​of the previous and next sub-frames, for example, can be applied. As a result, the estimated distance values ​​are distance d1200 for pixel P_a and distance d1201 for pixel P_b, and therefore the same result can be obtained as when the distance sensor 102 does not move in a direction perpendicular to the optical axis direction AX or rotate.

[0044] For simplicity, an example is shown in which the distance sensor 102 is composed of two pixels and horizontal movement occurs only in the pixel arrangement direction, but the distance sensor 102 may have multiple pixels arranged two-dimensionally.

[0045] The configuration and operation of the optical device 100 of the fifth embodiment will be described below with reference to FIGS. 13 and 14. Matters not mentioned in the description of the fifth embodiment may conform to at least one of the first to fourth embodiments. In the fifth embodiment, the detection unit 106 is replaced with a detection unit 106'. The detection unit 106' may be incorporated into the processor 120, or more specifically, into the signal processing unit 105.

[0046] The detection unit 106′ can be configured to detect the movement of the distance sensor 102 in the optical axis direction AX, for example, based on the plurality of subframes SFD stored by the memory 108. In other words, the detection unit 106′ can be configured to detect the movement of the distance sensor 102 in the optical axis direction AX by image processing.

[0047] 14 illustrates two consecutive subframes SD1400 and SF1401 among the multiple subframes SF that constitute the distance frame. The detection unit 106′ can generate movement amount data MD indicating the movement of the distance sensor 102 in the optical axis direction AX from images of the object 110 that appear in both the subframes SD1400 and SF1401. The subframe SF1400 is the subframe immediately preceding the subframe SF1401. If the object 110 is a planar object, for example, its image appears in both the two consecutive subframes. Therefore, it is possible to calculate a scaling factor from two or more consecutive subframes, and then calculate the movement amount of the distance sensor 102 in the optical axis direction AX based on this scaling factor.

[0048] For example, an image O1400 of the object 110 may be present in sub-frame SF1400. If the optical device 100 moves in the optical axis direction AX in sub-frame SF1401, causing the image O1400 to change to image O1401, the magnification ratio r1400 can be calculated based on that change. The detection unit 106′ calculates the magnification ratio and can use that magnification ratio to calculate the amount of movement of the distance sensor 102 in the optical axis direction AX. Even with this configuration, the reduction process can be performed, as in the first to fourth embodiments.

[0049] For simplicity, we have assumed a flat object 110 here, but by using AI or other means to detect objects, it is possible to estimate the magnification ratio for objects other than those with simple shapes and calculate the amount of movement of the distance sensor 102 in the optical axis direction AX.

[0050] FIG. 16 shows a configuration example of an image sensor IM00 that can be incorporated into the distance sensor 102. The image sensor IM00 may be configured on a single semiconductor substrate, or may be configured by stacking multiple semiconductor substrates (semiconductor layers). The image sensor IM00 may include a pixel array IM20 having multiple pixels IM21 arranged to form multiple rows and multiple columns. The image sensor IM00 may also include a readout circuit IM12 that reads out signals generated by each pixel IM21 of the pixel array IM20, and a control pulse generation unit IM15. The image sensor IM00 may also include a horizontal scanning circuit unit IM11, multiple signal lines IM13, a vertical scanning circuit unit IM10, multiple signal lines IM16, and an output circuit IM14.

[0051] The vertical scanning circuit unit IM10 may be configured to receive a first control pulse supplied from the control pulse generation unit IM15, generate a second control pulse, and supply the second control pulse to each pixel IM21. The vertical scanning circuit unit IM10 may include logic circuits such as a shift register and an address decoder. The horizontal scanning circuit unit IM11 may be configured to supply a column selection signal to the readout circuit IM12. The horizontal scanning circuit unit IM11 may include logic circuits such as a shift register and an address decoder.

[0052] FIG. 17 shows a configuration example of one pixel IM21 in FIG. 16. The pixel IM21 may include an avalanche photodiode (APD) 201 as a photoelectric conversion element, and the image sensor IM00 may be configured as an APD image sensor. The APD 201 generates charge pairs in response to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. A voltage VH (second voltage) higher than the voltage VL supplied to the anode may be supplied to the cathode of the APD 201. A reverse bias voltage (predetermined voltage) that can cause the APD 201 to perform avalanche multiplication may be supplied between the anode and cathode. With such a reverse bias voltage supplied between the anode and cathode, charges generated by incident light may cause avalanche multiplication, generating an avalanche current.

[0053] A mode in which an APD is operated with a voltage between the anode and cathode greater than the breakdown voltage is called Geiger mode. A mode in which an APD is operated with a voltage between the anode and cathode close to or less than the breakdown voltage is called linear mode. An APD operated in Geiger mode is called a SPAD. For example, the voltage VL (first voltage) is −30 V and the voltage VH (second voltage) is 1 V. The APD 201 may be operated in either linear mode or Geiger mode.

[0054] The quench element 202 can be arranged to connect a power supply that supplies voltage VH to the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has the function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation). The quench element 202 also has the function of returning the voltage supplied to the APD 201 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0055] The pixel 1121 may include a waveform shaping unit 210, a processing circuit 211, and a selection circuit 212. The waveform shaping unit 210 may shape the potential change of the cathode of the APD 201 obtained when photons are detected, and output a pulse signal. For example, an inverter circuit may be used as the waveform shaping unit 210. In FIG. 4, the waveform shaping unit 210 is configured with one inverter, but the waveform shaping unit 210 may include a series connection of multiple inverters or may include another circuit that has a waveform shaping effect.

[0056] The processing circuit 211 can count the pulse signals output from the waveform shaping unit 210 during each subfield period and hold the count value as a signal value. The processing circuit 211 can also be configured to reset the signal held in the processing circuit 211 by receiving a control pulse pRES via a drive line 213. The selection circuit 212 can receive a control pulse pSEL via a drive line 214 from the vertical scanning circuit unit IM10 and switch between electrical connection and disconnection between the processing circuit 211 and the signal line IM13. The selection circuit 212 can include, for example, a buffer circuit for outputting a signal.

[0057] An application example of the optical device 100 will be described below. FIG. 18(a) is a block diagram showing a schematic configuration of an optical system according to an embodiment. FIG. 18(b) is a block diagram showing a schematic configuration of a moving object according to an embodiment. FIG. 18(a) shows an example of an optical system related to an in-vehicle camera. The optical system 1300 includes an image capture device 1310. The optical system 1300 includes an image processing unit 1312 that performs image processing on multiple pieces of image data acquired by the image capture device 1310. The optical system 1300 also includes a distance acquisition unit 1316 that calculates the distance to an object and a collision determination unit 1318 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the distance acquisition unit 1316 is an embodiment of the optical device 100 configured to acquire distance information to the object using a Time of Flight (ToF) method. The collision determination unit 1318 may determine the possibility of a collision using any of the distance information. The distance acquisition unit 1316 may be implemented by dedicated hardware or a software module. The distance acquisition unit 1316 may also be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. The distance acquisition unit 1316 may also be realized by a combination of these.

[0058] The optical system 1300 is connected to a vehicle information acquisition device 1320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The optical system 1300 is also connected to an ECU 1330, which is a control device that outputs a control signal to generate a braking force on the vehicle based on the determination result of a collision determination unit 1318. The optical system 1300 is also connected to an alarm device 1340 that issues an alarm to the driver based on the determination result of the collision determination unit 1318. For example, if the determination result of the collision determination unit 1318 indicates a high collision possibility, the ECU 1330 controls the drive device (mechanical device) 1360 by applying the brakes, releasing the accelerator, suppressing engine output, or other vehicle control to avoid the collision and mitigate damage. The alarm device 1340 warns the user by sounding an alarm, displaying alarm information on a screen of a car navigation system, or vibrating a seat belt or steering wheel.

[0059] In this embodiment, the surroundings of a vehicle (moving object 1301), for example, the front or rear, are imaged by an optical system 1300. FIG. 18(b) shows an optical system for imaging the area in front of the vehicle (imaging range 1350). A vehicle information acquisition device 1320 sends instructions to the optical system 1300 or the imaging device 1310. This configuration can further improve the accuracy of distance measurement.

[0060] While the above describes an example of control to prevent collisions with other vehicles, the optical system 1300 can also be applied to autonomous driving control to follow other vehicles and autonomous driving control to prevent vehicles from drifting out of their lanes. Furthermore, the optical system 1300 can be applied not only to vehicles such as automobiles, but also to moving bodies (mobile devices) such as ships, aircraft, and industrial robots. The moving body includes one or both of a driving force generating unit that generates a driving force primarily used to move the moving body and a rotating body primarily used to move the moving body. The driving force generating unit can be an engine, a motor, or the like. The rotating body can be a tire, a wheel, a ship's screw, an aircraft's propeller, or the like. In addition to moving bodies, the optical system 1300 can be applied to a wide range of devices that use object recognition, such as intelligent transport systems (ITS). (Disclosures) The present specification and accompanying drawings disclose the following: (Item 1) An optical device comprising a distance sensor, and obtaining distance information based on an output of the distance sensor, 1. An optical device comprising: a processor that performs a reduction process for reducing an influence of movement of the distance sensor in an optical axis direction of the distance sensor on the distance information, based on the movement of the distance sensor in the optical axis direction. (Item 2) a detection unit that detects movement of the distance sensor in the optical axis direction; the processor performs the reduction process based on an output of the detection unit. 2. The optical device according to item 1, (Item 3) a memory that stores an output of the detection unit; the processor performs the reduction process based on the output of the detection unit stored in the memory. 3. The optical device according to item 2, (Item 4) The detection unit includes at least one of an acceleration sensor and a displacement sensor. 4. The optical device according to item 2 or 3, (Item 5) the processor generates the distance information by correcting information obtained by the output of the distance sensor based on the output of the detection unit during a measurement period in which the distance sensor performs measurement. 5. The optical device according to any one of items 2 to 4. (Item 6) the distance sensor generates a ranging frame composed of a plurality of subframes having different measurement periods; 6. The optical device according to item 5, (Item 7) the distance sensor includes a photoelectric conversion element, and the measurement period is an exposure period of the photoelectric conversion element; 7. The optical device according to item 6, (Item 8) the distance sensor generates a ranging frame including a plurality of subframes each having a different measurement period; the processor sends, to the distance sensor, timing signals that control a plurality of measurement periods for acquiring the plurality of subframes, based on the output of the detection unit; 3. The optical device according to item 2, (Item 9) the distance sensor includes a photoelectric conversion element, and the measurement period is an exposure period of the photoelectric conversion element; 9. The optical device according to item 8, characterized in that (Item 10) the distance sensor includes a photoelectric conversion element, and the distance sensor generates a distance measurement frame that is configured from a plurality of subframes having different exposure periods; the processor detects movement of the distance sensor in the optical axis direction based on the plurality of sub-frames; 2. The optical device according to item 1, (Item 11) a second detection unit configured to detect at least one of a movement of the distance sensor in a direction perpendicular to the optical axis direction and a rotation of the distance sensor; the processor generates the distance information by correcting information obtained by the output of the distance sensor based on outputs of the detection unit and the second detection unit during the measurement period. 6. The optical device according to item 5, (Item 12) The distance sensor includes a SPAD sensor. 12. The optical device according to any one of items 1 to 11. (Item 13) Further comprising a light emitting unit, The distance sensor operates at a timing corresponding to a light emission timing of the light emitting unit. 13. The optical device according to any one of items 1 to 12. (Item 14) An optical device according to any one of items 1 to 13; a signal processing unit that processes a signal output by the optical device; An optical system comprising: (Item 15) A moving body including the optical device according to any one of items 1 to 13, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the optical device. (Item 16) A distance measurement method for measuring distance using a distance sensor, comprising: an acquisition step of acquiring first distance information using the distance sensor during a certain period of time; a detecting step of detecting a movement of the distance sensor in an optical axis direction of the distance sensor during the period; a generating step of generating second distance information based on the first distance information acquired in the acquiring step and the result of detection in the detecting step; A distance measuring method comprising: (Item 17) A distance measurement method for measuring distance using a distance sensor, comprising: a measuring step of causing the distance sensor to perform measurements at a plurality of measurement timings determined based on a movement of the distance sensor in the optical axis direction of the distance sensor; a generating step of generating distance information based on the output of the distance sensor in the measuring step; A distance measuring method characterized by: (others) The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0061] 100: Optical device, 102: Distance sensor, 106: Detection unit, 120: Processor

Claims

1. An optical device comprising a distance sensor, and obtaining distance information based on an output of the distance sensor, 1. An optical device comprising: a processor that performs a reduction process for reducing an influence of movement of the distance sensor in an optical axis direction of the distance sensor on the distance information, based on the movement of the distance sensor in the optical axis direction.

2. a detection unit that detects movement of the distance sensor in the optical axis direction; the processor performs the reduction process based on an output of the detection unit.

2. The optical device according to claim 1.

3. a memory that stores an output of the detection unit; the processor performs the reduction process based on the output of the detection unit stored in the memory.

3. The optical device according to claim 2.

4. the detection unit includes at least one of an acceleration sensor and a displacement sensor; 3. The optical device according to claim 2.

5. the processor generates the distance information by correcting information obtained by the output of the distance sensor based on the output of the detection unit during a measurement period in which the distance sensor performs measurement.

3. The optical device according to claim 2.

6. the distance sensor generates a ranging frame composed of a plurality of subframes having different measurement periods; 6. The optical device according to claim 5.

7. the distance sensor includes a photoelectric conversion element, and the measurement period is an exposure period of the photoelectric conversion element; 7. The optical device according to claim 6.

8. the distance sensor generates a ranging frame including a plurality of subframes each having a different measurement period; the processor sends, to the distance sensor, timing signals that control a plurality of measurement periods for acquiring the plurality of subframes, based on the output of the detection unit; 3. The optical device according to claim 2.

9. the distance sensor includes a photoelectric conversion element, and the measurement period is an exposure period of the photoelectric conversion element; 9. The optical device according to claim 8.

10. the distance sensor includes a photoelectric conversion element, and the distance sensor generates a distance measurement frame that is configured from a plurality of subframes having different exposure periods; the processor detects movement of the distance sensor in the optical axis direction based on the plurality of sub-frames; 2. The optical device according to claim 1.

11. a second detection unit configured to detect at least one of a movement of the distance sensor in a direction perpendicular to the optical axis direction and a rotation of the distance sensor; the processor generates the distance information by correcting information obtained by the output of the distance sensor based on outputs of the detection unit and the second detection unit during the measurement period.

6. The optical device according to claim 5.

12. The distance sensor includes a SPAD sensor.

12. An optical device according to any one of claims 1 to 11.

13. Further comprising a light emitting unit, The distance sensor operates at a timing corresponding to a light emission timing of the light emitting unit.

12. An optical device according to any one of claims 1 to 11.

14. An optical device according to any one of claims 1 to 11; a signal processing unit that processes a signal output by the optical device; An optical system comprising:

15. A moving body comprising the optical device according to any one of claims 1 to 11, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the optical device.

16. A distance measurement method for measuring distance using a distance sensor, comprising: an acquisition step of acquiring first distance information using the distance sensor during a certain period of time; a detecting step of detecting a movement of the distance sensor in an optical axis direction of the distance sensor during the period; a generating step of generating second distance information based on the first distance information acquired in the acquiring step and the result of detection in the detecting step; A distance measuring method comprising:

17. A distance measurement method for measuring distance using a distance sensor, comprising: a measuring step of causing the distance sensor to perform measurements at a plurality of measurement timings determined based on a movement of the distance sensor in the optical axis direction of the distance sensor; a generating step of generating distance information based on the output of the distance sensor in the measuring step; A distance measuring method characterized by:

Images