Imaging device, control method, and computer program

Abstract

To provide an imaging device that can maintain distance measurement accuracy while suppressing the shooting time. [Solution] The imaging device comprises a light-emitting unit that emits pulsed light, an imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax, a control unit that controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range, and a distance measuring unit that calculates a distance value to a subject and the reliability of the distance value based on the at least first image signal and the second image signal, wherein the control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit so as to change the magnitude of the distance range of the at least first image signal and the second image signal generated by the imaging unit based on the reliability.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method, and a computer program.

Background Art

[0002] There is a photographing method using a camera called a range gate camera. It is a technique for clearly imaging only the subject within the target distance range by emitting pulsed light in front of the camera for a predetermined time and exposing only the light reflected within the target distance range to the imaging element inside the camera. Hereinafter, the above technique is referred to as range gate control.

[0003] With this range gate control, even in bad weather, a subject at a specific distance can be clearly imaged. Also, in range gate control, by controlling the emission time of the pulsed light, the exposure time, and the exposure timing of the camera, the distance range that can be photographed in one exposure (hereinafter referred to as the range width) can be divided or integrated.

[0004] When range gate control is performed with an integrated range width, since it contains a lot of ambient light, the subject will be photographed unclearly. However, since the number of distance divisions (hereinafter referred to as the range number) when photographing the entire photographing range is small, the time until the photographing of the desired distance range is completed can be shortened. Thereby, object recognition processing can be executed at an early timing.

[0005] On the other hand, when range gate control is performed with a divided range width, the ambient light decreases and the subject can be clearly photographed, but since the range number increases, the time until the photographing of the desired distance range is completed is extended.

[0006] For example, in Patent Document 1, an image obtained by range gate control is analyzed, and if there is an image to be focused on as a result of the analysis, for example, an image containing a subject, only the range width of that distance range is divided. Thereby, a technique for clearly photographing a subject while suppressing an extension of the time required to photograph the entire photographing range is described.

[0007] In recent years, stereo camera systems using the triangulation principle have been installed in many automobiles as devices for measuring the distance to a subject, enabling advanced driver-assistance systems (ADAS) functions. Examples of stereo camera systems include dual-lens stereo camera systems and monocular stereo camera systems (for example, image plane phase-difference camera ranging systems).

[0008] A stereo camera ranging system uses two cameras placed parallel to each other at a predetermined distance apart. It calculates the amount of parallax shift of the subject captured by each camera and uses this to calculate the distance to the subject.

[0009] The image plane phase-difference camera ranging system captures images using a single camera equipped with an image sensor called an image plane phase-difference element. It detects the amount of shift due to parallax in the image signal generated when light passing through the imaging optical system is incident on multiple pixels formed on the image sensor. Based on this amount of shift, it calculates the distance to the subject.

[0010] This range-measuring camera system may not be able to accurately calculate the distance to the subject in adverse weather conditions. Specifically, adverse weather conditions refer to situations where particles such as rain, fog, and snow obstruct the visibility of the subject.

[0011] In the aforementioned distance measuring camera system, if the camera captures light diffusely reflected by the particles in front of the subject, the subject cannot be photographed clearly. As a result, parallax cannot be calculated accurately, and the distance to the subject cannot be measured with precision. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Patent No. 6416085 [Overview of the project] [Problems that the invention aims to solve]

[0013] On the other hand, Patent Document 1 analyzes images acquired by range gate control and, if an image contains a subject, divides only the range width of that distance range. This allows for clear imaging of the subject while suppressing the extension of the time required to capture the entire shooting range.

[0014] However, the technology described in Patent Document 1 has the problem that, since the range width is updated based on the image analysis results, it is necessary to perform the image analysis process before the range width update process.

[0015] The objective of the present invention is to provide an imaging device that can maintain distance measurement accuracy while suppressing the shooting time. [Means for solving the problem]

[0016] One side view of the present invention, A light-emitting unit that emits pulsed light, An imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax, A control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range. The system includes a distance measuring unit that calculates a distance value to a subject and the reliability of the distance value based on at least the first image signal and the second image signal. The control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit to change the magnitude of the distance range of at least the first image signal and the second image signal generated by the imaging unit, based on the reliability. [Effects of the Invention]

[0017] According to the present invention, it is possible to provide an imaging device that can maintain distance measurement accuracy while suppressing the shooting time. [Brief explanation of the drawing]

[0018] [Figure 1]It is a functional block diagram showing a configuration example of the camera 100, the light emitter 200, and the moving body 300 according to Embodiment 1. [Figure 2] (A) is a top view of the photoelectric conversion element 102 as viewed from the light incident direction, and (B) is a cross-sectional view of the pixel group 400 in the I-I' cross-section in FIG. 2(A). [Figure 3] (A) to (D) are diagrams for explaining the relationship between the subject distance and the incident light in the imaging surface phase difference method. [Figure 4] It is a diagram showing an example of the relationship between the progress of the reflected light and the exposure timing in range gate control. [Figure 5] It is a diagram showing an example of the light emission and exposure (charge accumulation) control operation in one frame time in range gate control. [Figure 6A] It is a flowchart showing an example of the processing in the control method according to Embodiment 1. [Figure 6B] It is a flowchart showing an example of the subsequent processing of FIG. 6A. [Figure 7] It is a diagram showing an example of the relationship between the set value k and the range width according to Embodiment 1. [Figure 8] It is a diagram showing an example of the division of the range width according to Embodiment 1. [Figure 9] It is a diagram showing an example of a method for comparing the parallax reliability in each distance range obtained by range gate shooting with different set values k. [Figure 10] It is a functional block diagram showing a configuration example of the camera 100, the light emitter 200, and the moving body 300 according to Embodiment 2. [Figure 11] It is a flowchart showing an example of the processing according to Embodiment 2.

Embodiments for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and duplicate explanations are omitted or simplified.

[0020] <Embodiment 1> Figure 1 is a functional block diagram showing an example configuration of the camera 100, light emitter 200, and mobile body 300 according to Embodiment 1. The camera 100 functions as an imaging device.

[0021] Furthermore, in this embodiment, the mobile body 300 will be described using an example of a vehicle such as an automobile. However, the mobile body can be any mobile device, such as a train, ship, airplane, robot, drone, AGV (Automated Guided Vehicle), or AMR (Autonomous Mobile Robot).

[0022] Furthermore, some of the functional blocks shown in Figure 1 are realized by having a computer (not shown) included in the camera 100, light emitter 200, and mobile body 300 execute a computer program stored in memory (not shown) which is a storage medium.

[0023] However, some or all of these can be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0024] Furthermore, the components shown in Figure 1 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. The above explanation regarding Figure 1 also applies to Figure 10.

[0025] The camera 100 includes an imaging optical system 101, a photoelectric conversion element 102, an image processing unit 103, a distance measuring unit 104, a camera control unit 105, a memory unit 106, a communication unit 107, etc. The photoelectric conversion element 102 functions as an imaging unit.

[0026] The imaging optical system 101 can form an image (optical image) of the subject on the photoelectric conversion element 102, and has an exit pupil at a predetermined distance from the photoelectric conversion element 102.

[0027] The photoelectric conversion element 102 is, for example, a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor. The photoelectric conversion element 102 includes, for example, a pixel region in which pixels having a photoelectric conversion function are arranged in two dimensions.

[0028] In this embodiment, the photoelectric conversion element 102 uses an image sensor with an image plane phase difference distance measurement method as a stereo camera system. That is, each pixel region has one microlens and two photoelectric conversion units, and the subject image with parallax formed on the photoelectric conversion element 102 is photoelectrically converted to generate a first image signal and a second image signal, respectively.

[0029] In this embodiment, the image-plane phase-difference type photoelectric conversion element 102 has one microlens and two photoelectric conversion units in each pixel region, but it is not limited to this, as long as it has at least two photoelectric conversion units. For example, the image-plane phase-difference type may be realized with a quad-pixel structure having one microlens and four photoelectric conversion units.

[0030] Thus, the photoelectric conversion element 102, acting as the imaging unit, generates at least a first image signal and a second image signal having a predetermined parallax. However, the stereo camera system is not limited to this, and for example, a stereo camera consisting of two cameras having parallax may be used.

[0031] The image processing unit 103 performs image processing on the image signal generated by the photoelectric conversion element 102, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal. The output of the image processing unit 103 is supplied to the distance measuring unit 104, the ECU 301 (Electric Control Unit) of the mobile body 300, and the camera control unit 105.

[0032] The distance measuring unit 104 performs image recognition based on the first and second image signals supplied from the image processing unit 103 to recognize surrounding objects such as people and vehicles. It also calculates the distance to the object based on the phase shift of the images corresponding to the parallax between the first and second image signals supplied from the image processing unit 103. In other words, the distance measuring unit 104 calculates the distance value to the subject and the reliability of the distance value based on at least the first and second image signals.

[0033] The camera control unit 105 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the camera 100.

[0034] Furthermore, the camera control unit 105 functions as a control means, and by repeatedly outputting a reference signal to the camera 100 at predetermined intervals, it controls the length of the exposure period (charge accumulation time) for each frame and the timing of the control signals.

[0035] In the following description, exposure refers to the operation of charge accumulation, photoelectric conversion, or imaging in the photoelectric conversion element 102 from start to finish. In this embodiment, charge accumulation, photoelectric conversion, and imaging are used interchangeably, and the operations of charge accumulation, photoelectric conversion, and imaging include, for example, the operation of counting the photoelectrically converted signal in the counter 211 circuit when the photoelectric conversion element 102 is an APD.

[0036] Furthermore, the camera control unit 105 sets predetermined values ​​to the light emission control unit 202 via the communication unit 107 and the communication unit 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing synchronized with the reference signal, thereby controlling the light emission period of the light emitter 200.

[0037] In this way, a reference signal synchronized with the reference signal transmitted to the photoelectric conversion element 102 is also transmitted to the light emitter 200, and the light emitter 200 performs light emission control in synchronization with the reference signal. This makes it possible to synchronize the exposure timing of the photoelectric conversion element 102 with the light emission timing of the light emitter 200.

[0038] Here, the camera control unit 105 functions as a control unit that controls the emission time of the light-emitting unit and the exposure (charge accumulation) time of the imaging unit so that the imaging unit exposes (charge accumulates) the reflected light of pulsed light within a predetermined distance range.

[0039] The storage unit 106 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 107 is equipped with wireless and wired interfaces and outputs the generated image signals to the outside of the camera 100 and receives various signals from the outside.

[0040] Furthermore, in this embodiment, the communication unit 107 is connected to the communication unit 203 of the light emitter 200, and is also responsible for transmitting the aforementioned reference signal and control commands from the camera control unit 105 to the light emitter 200.

[0041] The light emitter 200 has a light-emitting unit 201 that emits pulsed light, a light-emitting control unit 202, and a communication unit 203. The light-emitting unit 201 is, for example, a near-infrared LED positioned in front of the mobile body 300, and is configured in combination with a lens.

[0042] The wavelength of the pulsed light emitted by the light-emitting unit 201 is not limited to the near-infrared region, but may be in the visible light region, for example. Furthermore, the light-emitting unit 201 may function as a light-emitting device on the mobile body 300, for example, to change the light intensity of a headlight. The light-emitting unit 201 outputs pulsed light for a predetermined emission time in response to a pulse signal output to the light emission control unit 202.

[0043] The light emission control unit 202 receives a reference signal transmitted by the camera control unit 105 of the camera 100 via the communication unit 203, generates a pulse signal at a predetermined timing using the reference signal as a reference, and outputs it to the light emission unit 201.

[0044] Here, the light emission control unit 202 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next.

[0045] The camera control unit 105 sets a predetermined value to the light emission control unit 202 via the communication unit 107 and 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing based on the reference signal, and the light emission period of the light emitter 200 is controlled. In this way, the light emission control unit 202 controls the light emission based on the same signal as the reference signal input to the camera 100.

[0046] The communication unit 203 communicates with the communication unit 107 of the camera 100, receives setting information and reference signals from the camera control unit 105 to the light emission control unit 202, and transmits them to the light emission control unit 202.

[0047] The ECU301 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the vehicle control unit 302.

[0048] The output of the distance measuring unit 104 is supplied to the camera control unit 105, and also to the vehicle control unit 302 and the display unit 303 via the ECU 301. The vehicle control unit 302 functions as a movement control means that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 301. The display unit 303 functions as a display means and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 300.

[0049] In this embodiment, the ECU 301 receives distance measurement result information from the distance measurement unit 104 and can perform vehicle stopping control (such as automatic braking) according to the content of the distance measurement result. The ECU 301 also receives distance measurement processing data from the distance measurement unit 104 and transmits it to the display unit 303.

[0050] Based on the output of the ECU 301, the display unit 303 displays various information to the driver of the mobile unit 300, such as images generated by the photoelectric conversion element 102, distance measurement results from the distance measurement unit 104, and the vehicle's driving status, for example, using a GUI.

[0051] Furthermore, the image processing unit 103, distance measuring unit 104, etc. shown in Figure 1 do not necessarily have to be mounted on the mobile body 300. For example, they may be installed on an external terminal provided separately from the mobile body 300 for remotely controlling the mobile body 300 or for monitoring the movement of the mobile body.

[0052] Here, the ranging principle using the image plane phase difference method with the photoelectric conversion element 102 will be explained using Figures 2 and 3. Figure 2(A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence, and (B) is a cross-sectional view of Figure 2(A) II'.

[0053] As shown in Figure 2(A), the photoelectric conversion element 102 is composed of multiple 2x2 pixel groups 400 arranged on a matrix. Each pixel group 400 has four infrared pixels (IR) that detect IR light. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412. The arrangement of each pixel in the pixel group 400 is not limited to this and may be changed depending on the wavelength band of the light emitted from the light emitter 200.

[0054] That is, the pixel group 400 may have, for example, an infrared pixel (IR) for detecting IR light, a red pixel for detecting the color red, a green pixel for detecting green light, and a blue pixel for detecting blue light. Alternatively, the pixel group 400 may have two green pixels for detecting the color green, a red pixel for detecting the color red, and a blue pixel for detecting blue light. In the above pixel arrangement, the two green pixels are arranged diagonally.

[0055] Figure 2(B) is a cross-sectional view of the pixel group 400 in the I-I' section of Figure 2(A), as shown above. Each pixel consists of a light guide layer 414 including a microlens 413 and a light receiving layer 415.

[0056] The light guide layer 414 consists of a microlens 413 for efficiently guiding light incident on the pixels to the light receiving layer 415, a color filter that transmits light in the wavelength band detected by each pixel, and a light guide member having wiring for pixel readout and pixel driving. The light receiving layer 415 converts the light incident on the light guide layer 414 into an electrical signal. The light receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0057] Figures 3(A) to 3(D) illustrate the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 3(A) is a schematic diagram showing the exit pupil 500 of the imaging optical system 101 and the light incident on the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 of the infrared pixel IR of the photoelectric conversion element 102. The photoelectric conversion element 102 has multiple pixels, but for simplicity, we will describe a single infrared pixel IR.

[0058] The microlens 413 of the infrared pixel IR is arranged so as to be optically conjugate to the exit pupil 500 and the light-receiving layer 415. As a result, light passing through the first pupil region 510, which is a partial pupil region contained within the exit pupil 500, is incident on the first photoelectric conversion unit 411, and light passing through the second pupil region 520 is incident on the second photoelectric conversion unit 412. do.

[0059] Each pixel's first photoelectric conversion unit 411 converts the received light into photoelectric signals and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the photoelectric conversion element 102. The first image signal shows the intensity distribution of the first image formed on the photoelectric conversion element 102 by light that has mainly passed through the first pupil region 510.

[0060] Each pixel's second photoelectric conversion unit 412 converts the received light into photoelectric signals and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 contained in the photoelectric conversion element 102. The second image signal shows the intensity distribution of the second image formed on the photoelectric conversion element 102 by light that has mainly passed through the second pupil region 520.

[0061] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 3(B) to (D).

[0062] Figures 3(B), (C), and (D) are schematic diagrams showing the photoelectric conversion element 102 and the imaging optical system 101. In the figures, 511 represents the first light passing through the first pupil region 510, and 521 represents the second light passing through the second pupil region 520.

[0063] Figure 3(B) shows the state when the image is in focus, with the first light 511 and the second light 521 converging on the imaging optical system 101. At this time, the parallax between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is 0.

[0064] Figure 3(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a negative value.

[0065] Figure 3(D) shows the state where the image is defocused in the positive w-axis direction. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a positive value.

[0066] A comparison of Figures 3(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax is proportional to the amount of defocus.

[0067] Therefore, the amount of disparity between the first image signal and the second image signal can be detected using a region-based matching method, such as the block matching method, and converted into a defocus amount. The block matching method is a method in which a region with a high degree of similarity (hereinafter referred to as the disparity matching region) from the other image is matched with a selected region from one image, and the positional difference between that region and the other is defined as the disparity.

[0068] Furthermore, by using the imaging formula of the imaging optical system 101, the amount of defocus on the image side can be converted into the distance to the object. The imaging formula of the imaging optical system 101 is given by the following equation 1, where f is the focal length of the photoelectric conversion element 102, Ipp is the distance from the principal point on the image side to the photoelectric conversion element 102, ΔL is the amount of defocus, and D is the distance to the object.

[0069]

number

[0070] Next, Figure 4 shows an example of the relationship between the propagation of reflected light and exposure timing in range gate control, illustrating the relationship between the propagation of synchrotron radiation and its reflected light from the light emitter 200 and the exposure timing of the camera 100. In Figure 4, the horizontal axis represents distance and the vertical axis represents time.

[0071] Figure 4 illustrates a method for acquiring an image (range-gated image) of the target distance by synchronizing the light emission timing and exposure timing according to the target distance (range-gate control). A camera that acquires an image of the target distance range using this range-gate control is called a range-gate camera.

[0072] First, let's explain the horizontal axis. Fog 610 exists between distance x1 and distance x2, and a vehicle 620 is located at distance x3. Also, in Figure 4, range gate control is used, with the position of camera 100 as the starting point, and range gate images are acquired from there within a predetermined distance range.

[0073] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 620 is located within the target distance range R. Furthermore, the subject image within the target distance range R is photoelectrically converted by the photoelectric conversion element 102 and held as an electric charge in each pixel.

[0074] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission at the light emitter 200, and time tf is defined as the end time of light emission. In this case, the light emission period is tf. Also, when acquiring a range gate image within the target distance range R, with the position of camera 100 as the starting point, the exposure start time (charge accumulation start time) is defined as time t1, and the exposure end time (charge accumulation end time) is defined as time t2.

[0075] Time t1 is the moment when the light emitted from the light emitter 200 at time 0 returns to the camera 100 as reflected light from the vehicle 620. Also, t2 is the moment when the light emitted from the light emitter 200 returns to the camera 100 as reflected light from a point that has traveled a distance R from distance D to the target distance range.

[0076] Furthermore, time t3 is defined as the timing when the first reflected light returns to camera 100, and time t4 is defined as the timing when the last reflected light from fog 610 returns to camera 100.

[0077] In range gate control, exposure (charge accumulation) is not performed during the period from time t3 to time t4 when the reflected light from the fog 610 reaches the camera 100, and exposure (charge accumulation) is performed only during the period from time t1 to time t2 when the reflected light from distance D reaches the camera within the target distance range R. This makes it possible to remove the fog 610 while clearly acquiring an image of the vehicle 620.

[0078] Here, we will explain the time it takes for reflected light from an object at distance x to return to camera 100. Let time tr be the timing when the emitted light, which began to be emitted at time 0, strikes an object at distance x and returns to the imaging unit as reflected light.

[0079] At this time, the relationship between the timing time tr of the reflected light returning and the distance x from the object being imaged is given by the following equation 2. Time tr=2x / speed of light c (approximately 3×10^8m / s) (Formula 2)

[0080] As shown in Figure 4, when the imaging range is defined as the target distance range R from distance D, the exposure (charge accumulation) timing time t1 at the starting point of the target distance range R can be obtained by substituting distance D into distance x in equation (2) above, as shown in equation 3 below. Time t1=2D / speed of light c...(Formula 3)

[0081] Furthermore, the exposure (charge accumulation) timing time t2 at the endpoint of the target distance range R can be obtained by substituting the distance D+R into the distance x in equation 2 above and adding the time tf, as shown in equation 4 below. t2=tf+2(D+R) / speed of light c (Equation 4)

[0082] By controlling the time tr from light emission to exposure (charge accumulation) according to the target distance x (the distance to be imaged), range gate control is realized that allows for clear imaging of subjects at the target distance even if there is fog or other obstructions between the camera and the target distance.

[0083] Figure 5 shows an example of the light emission and exposure (charge accumulation) control operation for one frame in range gate control, illustrating the control operation for obtaining a range gate image per frame.

[0084] In Figure 5, the vertical synchronization signal indicates the frame period of the image, with the period between one low pulse and the next low pulse being one frame time. The light emission control indicates the light emission timing of the light emission unit 201, and the exposure (charge accumulation) control of the photoelectric conversion element indicates the length of the exposure (charge accumulation) period for each frame of the photoelectric conversion element 102 output by the camera control unit 105, as well as the timing of the control signal.

[0085] In range gate control, the light emission period is controlled in a pulsed manner by the camera control unit 105, and exposure (charge accumulation) of the photoelectric conversion element 102 is performed only for reflected light from a specific target distance range R.

[0086] Here, let tf be the emission period from the start to the end of emission, t1 be the time from the start of emission to the start of exposure (charge accumulation), and t2 be the time from the start of emission to the end of exposure (charge accumulation). In this case, t1 represents the period from the start of emission until the light reaches a specific target distance range R and the reflected light returns to camera 100, and the time from t1 to t2 is the period during which the reflected light from the specific target distance range R is exposed.

[0087] For range gate control to function correctly, it is necessary to synchronize the timing of light emission start and exposure start according to a predetermined target distance range. In this embodiment, the camera control unit 105 synchronizes the timing by transmitting the same reference signal to the photoelectric conversion element 102 and the light emission control unit 202.

[0088] As shown in the timing chart of Figure 5, the period from the start of one emission to the start of the next emission constitutes a range gate operation cycle. During one range gate operation cycle, the light received by the photoelectric conversion element 102 is converted into an electric charge and held within the photoelectric conversion element 102.

[0089] In this state, the next range gate operation cycle is performed, and the light newly received by the photoelectric conversion element 102 is converted into an electric charge and added to the charge held within the photoelectric conversion element 102. The interval between one emission and the next emission is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 100.

[0090] As shown in Figure 5, within one frame time, the range gate operation cycle is performed a predetermined number of times. The charge that was added and held last within one frame time is sent to the memory in the photoelectric conversion element 102, and the charge that was added and held thereafter is reset.

[0091] In this embodiment, since the exposure (charge accumulation) period is synchronized with the light emission from the light emitter 200, it is possible to obtain a clear image for the target distance range even under adverse weather conditions such as fog. On the other hand, in range gate control, distance can only be measured in units of the target distance range R, which depends on the exposure time of the camera 100 and the light emission time of the light emitter 200.

[0092] To improve distance measurement accuracy, it is necessary to reduce the target distance range R. One possible method to achieve this is to shorten the exposure time of the camera 100 and the light emission time of the light emitter 200. However, this is difficult due to the physical control limitations of the camera 100 and the light emitter 200.

[0093] Furthermore, reducing the target distance range R and attempting to perform range gate imaging over long distances increases the number of exposures for camera 100, resulting in increased power consumption and longer time required to measure distances over long distances.

[0094] Next, the stereo range gate camera distance measurement process in this embodiment will be explained using Figures 6A, 6B, 7, and 8.

[0095] Figure 6A is a flowchart showing an example of processing in the control method according to Embodiment 1, and Figure 6B is a flowchart showing a continuation of the processing example in Figure 6A. The flowcharts in Figures 6A and 6B control a light-emitting unit that emits pulsed light and an imaging unit that generates a first image signal and a second image signal having a predetermined parallax.

[0096] Furthermore, the CPU and other components within the camera control unit 105 execute the computer program stored in memory, thereby sequentially performing the operations at each step of the flowcharts shown in Figures 6A and 6B.

[0097] For the sake of clarity, the process will be explained separately for the first step S101 and subsequent steps, and for the nth step (n>=2) S101 and subsequent steps.

[0098] First, let's explain the processing from step S101 onwards. In step S101, the camera control unit 105 defines the set value k for range gate control of the light emitter 200, the first photoelectric conversion unit 411, and the second photoelectric conversion unit 412.

[0099] The set value k is a value corresponding to the distance range width in the processing flow of Figures 6A and 6B, and the range width is determined by the light emission time of the light emitter 200, the exposure (charge accumulation) time of the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412, and the exposure (charge accumulation) timing.

[0100] Figure 7 is a diagram showing an example of the relationship between the set value k and the range width according to Embodiment 1, and shows the set value k and the range width corresponding to the set value k in this process, but is not limited to this. From Figure 7, it is assumed that only k values ​​of 1, 2, and 3 are defined, and here the set value k = 1 (range width 30m) is defined.

[0101] In step S102, range gate imaging is performed at the set value k, and phase-difference distance measurement is performed on the imaging plane. Distance values ​​and parallax reliability are also calculated for each range. Specifically, in step S102, the camera control unit 105 performs range gate control of the light emitter 200, the first photoelectric converter 411, and the second photoelectric converter 412 so that the set value k=1 (range width 30m), and acquires the first image signal and the second image signal for each distance range.

[0102] Here, step S102 functions as a distance measurement step (distance measurement unit) that calculates the distance value to the subject and the reliability of the distance value based on the first image signal and the second image signal.

[0103] Subsequently, the distance measuring unit 104 performs phase difference distance measurement on the imaging plane based on the first and second image signals in each distance range and calculates the distance value for each distance range. The distance value is obtained, for example, by calculating the amount of disparity for each pixel based on the first and second image signals using the block matching method described above, and then converting the amount of disparity into a distance value.

[0104] The distance measuring unit 104 also calculates the variation in luminance values ​​for each parallax matching area in the block matching method. The variation in luminance values ​​is an indicator of the accuracy of the calculated parallax amount, and is hereinafter referred to as the parallax reliability.

[0105] Parallax reliability can be evaluated, for example, by the ratio of the standard deviation of the luminance values ​​in the parallax matching area to the mean value. When the change in pixel values ​​(so-called contrast) within the matching area is large, the standard deviation becomes large.

[0106] Furthermore, a larger amount of light incident on a pixel results in a larger average value. A larger amount of light incident on a pixel also results in more light shot noise. In other words, the average value has a positive correlation with the amount of noise. The ratio of the average value to the standard deviation (standard deviation / average value) corresponds to the ratio of contrast to noise.

[0107] If the contrast is sufficiently large relative to the noise, it can be estimated that the error in calculating the amount of parallax is small. In other words, the higher the parallax confidence, the smaller the error in the calculated amount of parallax, and the more accurate the amount of parallax. If the number of pixels in the matching area is n, the brightness values ​​of each pixel are x1, x2...xn, and the average value of the brightness values ​​is x, then the parallax confidence d can be calculated, for example, using equation 5 below.

[0108]

number

[0109] Thus, in this embodiment, the reliability is calculated based on the variation in brightness values ​​in a pixel group having at least a first image signal and a second image signal, or the variation in brightness values ​​and the average value of the brightness values.

[0110] In step S103, the distance measuring unit 104 determines whether the proportion of the parallax matching area where the calculated parallax confidence is less than or equal to the threshold d1 is a% or more of the total parallax matching area (for example, whether 50% or more of the parallax matching area is less than or equal to the threshold d1 of the total parallax matching area).

[0111] In other words, the system determines whether a predetermined proportion or more of the range-gated images within each distance range contain areas with low parallax reliability, such as the road surface. Here, the threshold d1 refers to the parallax reliability on the road surface.

[0112] If, in step S103, it is determined that the percentage of parallax matching areas where the parallax confidence is less than or equal to the threshold d1 is less than a%, the process proceeds to step S104. In step S104, the average parallax confidence value dave is calculated for each range. That is, the average parallax confidence value dave for each distance range is calculated based on the parallax confidence in each parallax matching area calculated by the distance measuring unit 104 for each distance range.

[0113] On the other hand, if in step S103 it is determined that the ratio of parallax matching regions with a parallax reliability of less than or equal to the threshold d1 is a% or more for the entire parallax matching region, the process proceeds to step S105. In step S105, regions with a parallax reliability of less than or equal to the threshold d1 are removed. That is, the distance measuring unit 104 removes the image signals in the parallax matching regions with a parallax reliability of less than or equal to the threshold d1 from the first image signal and the second image signal for each distance range.

[0114] In other words, in this embodiment, instead of using the first and second image signals whose reliability is less than or equal to a predetermined value, the size of the distance range of the first and second image signals generated by the imaging unit is changed based on the first and second image signals whose reliability is greater than a predetermined value.

[0115] The process then proceeds to step S104, where the average disparity confidence dave is calculated for each range. Specifically, the average disparity confidence dave for each distance range is calculated based on the first and second image signals in the remaining disparity matching region removed in step S105.

[0116] Thus, step S105 prevents the calculation of the average parallax confidence dave from the parallax confidence of the road surface, which has low parallax confidence, from being affected when, for example, there is a range gate image in which the subject and the road surface are mixed. In other words, in the above case, it prevents the parallax confidence of the distance range in which the subject is captured from being calculated as low due to the influence of the road surface, etc.

[0117] In step S106, it is determined whether this is the first time step S106 has been processed, or whether xm has been traveled since the start of the previous step S106. Since this is the first step S106, the process proceeds to step S107 in Figure 6B. In step S107, the distance measuring unit 104 compares the average parallax confidence value dave for each distance range with the threshold d2 and determines whether the average parallax confidence value dave > threshold d2.

[0118] The threshold d2 is the minimum average parallax confidence value required to determine that the distance value of the subject calculated in this embodiment is an accurate value. If the result in step S107 is No, that is, if it is determined that there is a distance range in which the average parallax confidence value dave is less than or equal to the threshold d2, the process proceeds to step S108.

[0119] If the result in step S107 is "Yes," that is, if the average disparity confidence value dave across all distance ranges is determined to be greater than the threshold d2, the process proceeds to step S109.

[0120] In step S108, it is determined whether it is the first time step S108 or if dave(n) > dave(nn-1). Since it is the first time step S108, the process proceeds from step S108 to step S110.

[0121] In step S110, the range width of the relevant range is switched to set the value k = k + 1 (where k <= 3). That is, the distance measuring unit 104 updates k to k + 1 (where k <= 3) only for ranges (distance ranges) where the average parallax reliability value dave <= threshold d2 for each distance range, and sets the value k from 1 to 2 as shown in Figure 8.

[0122] Figure 8 shows an example of range width division according to Embodiment 1. In other words, when k=1, the range (distance range) width is 30m, but when k=2, each range (distance range) is divided into two, and the range width (distance range) becomes 15m.

[0123] When range gate imaging is performed with a range width of 15m after division, ambient light such as diffuse reflection from fog is reduced compared to range gate imaging with the range width before division, resulting in a range gate image with improved subject quality. Furthermore, since phase-difference distance measurement can be performed on the image sensor based on the improved range gate image, the calculated parallax reliability is also improved.

[0124] Thus, in step S110 of this embodiment, when the reliability is less than or equal to a predetermined threshold d2, the emission time of the light-emitting unit and the exposure time of the imaging unit are controlled to reduce the distance range.

[0125] In step S112, the camera control unit 105 determines whether, for example, the driver of the mobile body has performed an operation to end the movement of the mobile body 300. If it is determined that an operation to end the movement has been performed, the stereo range gate camera distance measurement processing flow shown in Figures 6A and 6B is terminated. If it is determined that an operation to end the movement has not been performed, the process proceeds to step S101 and the second stereo range gate camera distance measurement processing is started.

[0126] On the other hand, if the result in step S107 is Yes, then in step S109, it is determined whether the average parallax confidence value dave for each range is less than the threshold d3. That is, in step S109, the average parallax confidence value dave for each range calculated by the distance measuring unit 104 is compared with the threshold d3, and it is determined whether there is a distance range where the average parallax confidence value dave is less than the threshold d3.

[0127] If the result in step S109 is "No," that is, if the average parallax confidence value dave is greater than the threshold d3 within a certain distance range, the process proceeds to step S111.

[0128] On the other hand, if it is determined Yes in step S109, that is, if the average value dave of the parallax reliability for each distance range is all smaller than the threshold value d3, the process proceeds to step S112. Note that the threshold value d3 is an average value of the parallax reliability sufficient for determining that the subject distance value calculated in the present embodiment is an accurate value.

[0129] Note that if an attempt is made to make the average value of the parallax reliability larger than the threshold value d3, it is necessary to further divide the range width, but the time until the shooting of the desired distance range is completed is extended. Therefore, the extension of the shooting time is prevented by step S111 described later. Here, the threshold values d1, d2, and d3 have the relationship shown in the following equation 6. d1 < d2 < d3 ··· (Equation 6)

[0130] In step S111, the range width of the corresponding range is switched and the set value k = k - 1 (where k >= 1) is set. That is, the distance measuring unit 104 updates k to k - 1 only for the distance range in which it is determined that the average value dave of the parallax reliability > the threshold value d3 (where k >= 1). That is, in this case, as shown in FIG. 8, the set value k remains 1 (range width 30 m).

[0131] Thus, in step S111 of the present embodiment, based on the reliability, the control unit controls the light emitting unit and the imaging unit so that the distance range of the image captured by the imaging unit in one exposure is increased.

[0132] Here, steps S110, S111, etc. function as control steps (control unit) for controlling the light emission time of the light emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light in a predetermined distance range. Also, in the control step (control unit), based on the reliability, the light emission time of the light emitting unit and the exposure time of the imaging unit are controlled so as to change the size of the distance range of at least the first image signal and the second image signal generated by the imaging unit.

[0133] In step S112, as described above, the camera control unit 105 determines whether an operation to terminate the movement of the mobile body 300 has been performed. If an operation to terminate movement has been performed, the stereo range gate camera distance measurement process shown in Figures 6A and 6B is terminated. If it is determined that an operation to terminate movement has not been performed, the process proceeds to step S101 and the second stereo range gate camera distance measurement process is started.

[0134] Next, the processing from step S101 onwards for the nth time (n>=2) will be explained using Figures 6A, 6B, 7, and 8. The processing in step S101 is as described above. Here, since it is the nth time step S101, the camera control unit 105 sets the set value k from the (n-1)th time step S112.

[0135] In step S102, as described above, the camera control unit 105 performs range gate control with a set value k and acquires first and second image signals for each distance range. Based on the acquired first and second image signals for each distance range, the distance measuring unit 104 performs imaging plane phase difference distance measurement and calculates the distance value for each distance range.

[0136] Steps S103, S104, and S105 are as described above. If there is a% or more of the disparity matching area in the first and second image signals of each distance range where the disparity confidence is less than or equal to the threshold d1, the image signal is removed in step S105. Then, in step S104, the average disparity confidence value dave for each distance range is calculated.

[0137] In step S106, the camera control unit 105 determines whether xm has traveled since the start of the (n-1)th step S106. If xm has traveled since the start of the (n-1)th step S106, the process proceeds to step S107 in Figure 6B; otherwise, the process proceeds to step S101.

[0138] In step S107, as described above, the distance measuring unit 104 determines whether the average parallax confidence value dave > threshold d2. If it is determined that there is a range in which the average parallax confidence value dave is less than or equal to the threshold d2, the process proceeds to step S108.

[0139] Step S108 is a process performed by the distance measuring unit 104, which determines whether the average parallax confidence value dave(n) for each distance range in the nth measurement is greater than the average parallax confidence value dave(n-1) for each distance range in the n-1 measurement.

[0140] If the average disparity confidence value dave(n) calculated in the nth step S104 is greater than the average disparity confidence value dave(n-1) calculated in the n-1th step S104, the process proceeds to step S110.

[0141] If the average parallax confidence value dave(n) for each distance range calculated in the nth step S104 is less than or equal to the average parallax confidence value dave(n-1) for each distance range calculated in the (n-1)th step S104, the process proceeds to step S111.

[0142] Figure 9 shows a method for comparing the average parallax confidence value dave. Figure 9 is a diagram illustrating an example of a method for comparing parallax confidence in each distance range obtained by range gate imaging with different setting values ​​k. Here, the average parallax confidence value of the range gate image at 30m in front of camera 100, calculated in the first step S104, is defined as dave(1).

[0143] Furthermore, the average parallax confidence value of the range gate image at 15m in front of camera 100, calculated in the second step S104, is denoted as dave(2), and the average parallax confidence value of the range gate image at 15m to 30m is denoted as dave(3). Since the comparison of the average parallax confidence values ​​is performed within the same range, the magnitude of dave(1) and (dave(2)+dave(3)) / 2 is compared.

[0144] If (dave(2)+dave(3)) / 2 is larger than dave(1), it can be considered that the influence of ambient light has decreased due to the division of the range width, and the average parallax confidence has improved. Therefore, it can be predicted that the average parallax confidence can be further improved by dividing the range width even more.

[0145] On the other hand, if (dave(2) + dave(3)) / 2 is less than or equal to dave(1), then the distance range is less affected by ambient light and is likely to include many subjects with low average parallax reliability, such as road surfaces.

[0146] Therefore, it is predicted that dividing the range width will not significantly improve the average parallax reliability. Thus, it is considered desirable to integrate the range width to shorten the time required to complete imaging within the desired distance range.

[0147] Step S110 is a process performed by the distance measuring unit 104, in which, for each distance range in the (n-1)th measurement, the set value k is updated to k+1 only for distance ranges where the average parallax confidence value dave for each distance range in the nth measurement is large. This divides the range width as shown in Figure 8. However, k <= 3.

[0148] Step S111 is a process performed by the distance measuring unit 104, in which, for each distance range in the n-1 measurement, the set value k is updated to k-1 only for distance ranges where the average parallax confidence value dave for each distance range in the n measurement is equal to or less than or equal to the average parallax confidence value dave for each distance range in the n measurement. This integrates the range widths (provided k>=1).

[0149] In step S112, the camera control unit 105 determines whether an operation to terminate the movement of the mobile body 300 has been performed. If an operation to terminate movement has been performed, the stereo range gate camera distance measurement process shown in Figures 6A and 6B is terminated. If it is determined that an operation to terminate movement has not been performed, the process proceeds to step S101 and the (n+1)th stereo range gate camera distance measurement process is started.

[0150] If, in step S107, the average disparity confidence value dave across all distance ranges is determined to be greater than the threshold d2, the process proceeds to step S109.

[0151] In step S109, the average parallax confidence value dave calculated by the distance measuring unit 104 is compared with the threshold d3 to determine whether there are any distance ranges where the average parallax confidence value dave < threshold d3. If there are distance ranges where the average parallax confidence value dave is greater than the threshold d3, the process proceeds to step S111. If the average parallax confidence value dave for each distance range is less than the threshold d3, the process proceeds to step S112.

[0152] In step S111, the distance measuring unit 104 determines that the average parallax confidence value dave for each distance range is a threshold. For distances greater than d3, the set value k is updated to k-1, and the range width is consolidated.

[0153] In step S112, as described above, the camera control unit 105 determines whether an operation to terminate the movement of the mobile body 300 has been performed. If an operation to terminate movement has been performed, the stereo range gate camera distance measurement process shown in Figures 6A and 6B is terminated. If it is determined that an operation to terminate movement has not been performed, the process proceeds to step S101 and the (n+1)th stereo range gate camera distance measurement process is started.

[0154] In this embodiment, by dividing or merging the range width based on the parallax confidence calculated from the range gate image, it is possible to maintain a parallax confidence level above a predetermined level while suppressing the extension of the time required to capture the entire shooting range.

[0155] <Embodiment 2> Next, Embodiment 2 of the present invention will be described. Embodiment 1 described the stereo range gate camera distance measurement process. Embodiment 2 will describe a configuration and processing flow that determines the visibility conditions around the vehicle and performs stereo range gate camera distance measurement if visibility is poor, and stereo camera distance measurement if visibility is good.

[0156] Figure 10 is a functional block diagram showing an example configuration of the camera 100, light emitter 200, and mobile body 300 according to Embodiment 2. The only difference from the block diagram of Embodiment 1 shown in Figure 1 is that the camera 100 is equipped with a visibility defect detection unit 108; the other configurations are the same and therefore will not be explained.

[0157] The visibility impairment detection unit 108 acquires and determines the visibility conditions around the vehicle and outputs the determination result to the camera control unit 105. The visibility impairment detection unit 108 also functions as a visibility condition determination unit that determines the visibility conditions around the imaging unit.

[0158] Poor visibility refers to conditions such as bad weather like rain or fog, or dark conditions like nighttime. Bad weather can be determined using known techniques, such as the method described in Japanese Patent Publication No. 2008-33872.

[0159] In the method described in Japanese Patent Publication No. 2008-33872, a camera attached to the moving vehicle captures images of the area illuminated and unilluminated by the vehicle's headlights, and determines whether visibility is poor based on the difference in brightness. Alternatively, darkness may be determined by using a light-receiving sensor mounted on the moving vehicle, for example, if the amount of light around the vehicle falls below a predetermined threshold.

[0160] Next, Figure 11 is a flowchart showing an example of processing according to Embodiment 2. Note that the CPU and other components of the computer within the camera control unit 105 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart of Figure 11.

[0161] The process will be explained separately for the first step S201 and subsequent steps, and for the nth step (n>=2) S201 and subsequent steps. First, the process for the first step S201 and subsequent steps will be explained.

[0162] In step S201, it is determined whether the process of step S201 is being performed for the first time, or whether Ym has been traveled since step S201 of the (n-1)th time. If yes, proceed to step S202; otherwise, proceed to step S207. In this case, it is the first time step S201, so proceed to step S202.

[0163] In step S202, the visibility impairment determination unit 108 acquires the visibility conditions around the vehicle. As mentioned above, the visibility conditions are determined using the method described in Japanese Patent Application Publication No. 2008-33872 or using a light receiving sensor attached to the moving object.

[0164] In step S203, the system determines whether visibility around the vehicle is poor based on the acquired visibility conditions. If visibility is determined to be poor, the system proceeds to step S204. If visibility is determined to be good, the system proceeds to step S205.

[0165] In step S204, the stereo range gate camera distance measurement described in Embodiment 1 is performed. As mentioned above, this distance measurement method performs exposure control synchronized with the light emission, making it possible to accurately calculate the distance to the subject within the distance range even in poor visibility conditions. Furthermore, it is possible to maintain a parallax reliability of a predetermined level or higher while suppressing the extension of the time required to capture the entire image reflection.

[0166] On the other hand, if the result in step S201 is "No", then in step S207, it is determined whether poor visibility was determined in the (n-1)th step S203. If poor visibility was determined, the process proceeds to step S204; if good visibility was determined, the process proceeds to step S205.

[0167] In step S205, stereo camera ranging (for example, the image plane phase-difference camera ranging described above) is performed. Unlike stereo range gate camera ranging, this ranging method does not perform light emission or exposure control synchronized with the light emission.

[0168] By exposing the camera to reflected light regardless of distance, it is possible to capture images of subjects at various distances in a single exposure, and then measure the distance to each subject based on the obtained images. In other words, stereo camera ranging can measure the distance to subjects with less power consumption than stereo range gate camera ranging. However, as mentioned above, the ranging accuracy decreases in poor visibility conditions.

[0169] Thus, in step S204, the emission time of the light-emitting unit and the exposure time of the imaging unit are controlled so that the imaging unit is exposed to reflected pulsed light within a predetermined distance range. On the other hand, in step S205, the distance to the subject is calculated based on at least a first image signal and a second image signal generated from the imaging unit without using pulsed light.

[0170] In other words, in steps S202 to S205, the system switches between executing the process in step S204 or executing the predetermined step in step S205 based on the determination result of the visibility condition determination unit.

[0171] Step S206 determines whether an operation to terminate the movement of the mobile body 300 has been performed. If an operation to terminate movement has been performed, the processing flow shown in Figure 11 is terminated. If an operation to terminate movement has not been performed, the second step S201 is started.

[0172] Next, we will explain the processing from step S201 onwards for the nth time (n>=2) using Figure 11. In step S201, it is determined whether Ym has been traveled since the (n-1)th step S201. If Ym has been traveled, the process proceeds to step S202; otherwise, it proceeds to step S207.

[0173] Steps S202, S203, S204, S205, and S206 are processed as described above, so their explanation is omitted.

[0174] In step S207, it is determined whether poor visibility was determined in the (n-1)th step S203. If poor visibility was determined, the process proceeds to step S204 to perform stereo range gate camera distance measurement. On the other hand, if good visibility was determined in step S207, the process proceeds to step S205 to perform stereo camera distance measurement.

[0175] In this embodiment, the visibility conditions around the vehicle can be determined, and appropriate distance measurement processing can be performed according to the visibility conditions.

[0176] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0177] Furthermore, the present invention includes, for example, a system that implements the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0178] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the distance measuring camera system, etc., via a network or various storage media.

[0179] The computer (or CPU, MPU, etc.) in the distance measuring camera system, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0180] (Configuration 1) An imaging device comprising: a light-emitting unit that emits pulsed light; an imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax; a control unit that controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range; and a distance measuring unit that calculates a distance value to a subject and the reliability of the distance value based on the at least first image signal and the second image signal, wherein the control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit so as to change the magnitude of the distance range of the at least first image signal and the second image signal generated by the imaging unit based on the reliability.

[0181] (Configuration 2) The imaging device according to Configuration 1, characterized in that the reliability is calculated based on the variation in brightness values ​​in a pixel group having at least the first image signal and the second image signal, or the variation in brightness values ​​and the average value of the brightness values.

[0182] (Configuration 3) The imaging apparatus according to Configuration 1 or 2, characterized in that the control unit controls the emission time of the emission unit and the exposure time of the imaging unit to reduce the distance range when the reliability is below a predetermined threshold.

[0183] (Configuration 4) The imaging device according to any one of Configurations 1 to 3, characterized in that the control unit controls the light-emitting unit and the imaging unit based on the reliability so as to increase the distance range of the image captured by the imaging unit in one exposure.

[0184] (Configuration 5) An imaging device according to any one of Configurations 1 to 4, comprising a field of view determination unit for determining the field of view around the imaging unit, wherein the control unit switches between controlling the emission time of the emission unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range, or executing a process to calculate the distance to a subject based on the at least first image signal and the second image signal generated from the imaging unit without using the pulsed light.

[0185] (Configuration 6) The imaging apparatus according to any one of Configurations 1 to 5, characterized in that the control unit changes the size of the distance range of the at least first image signal and the second image signal generated by the imaging unit based on the reliability of at least the first image signal and the second image signal which is greater than a predetermined value.

[0186] (Method) A control method for controlling a light-emitting unit that emits pulsed light and an imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax, comprising: a control step of controlling the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range; and a distance measurement step of calculating a distance value to a subject and a reliability of the distance value based on the at least first image signal and the second image signal, wherein the control step controls the emission time of the light-emitting unit and the exposure time of the imaging unit so as to change the magnitude of the distance range of the at least first image signal and the second image signal generated by the imaging unit based on the reliability.

[0187] (Program) A computer program for controlling each part of the imaging device described in any one of configurations 1 to 6 by computer. [Explanation of Symbols]

[0188] 100: Camera 200: Light emitter 300: Mobile

Claims

1. A light-emitting unit that emits pulsed light, An imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax, A control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range. The system includes a distance measuring unit that calculates a distance value to a subject and the reliability of the distance value based on at least the first image signal and the second image signal. The imaging apparatus is characterized in that the control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit to change the magnitude of the distance range of at least the first image signal and the second image signal generated by the imaging unit, based on the reliability.

2. The imaging apparatus according to claim 1, characterized in that the reliability is calculated based on the variation in brightness values ​​in a pixel group having at least the first image signal and the second image signal, or the variation in brightness values ​​and the average value of the brightness values.

3. The imaging apparatus according to claim 1, characterized in that the control unit controls the emission time of the emission unit and the exposure time of the imaging unit to reduce the distance range when the reliability is below a predetermined threshold.

4. The imaging apparatus according to claim 1, characterized in that the control unit controls the light-emitting unit and the imaging unit based on the reliability so as to increase the distance range of the image captured by the imaging unit in one exposure.

5. The system includes a field of view determination unit that determines the field of view conditions around the imaging unit, Based on the determination result of the field of view determination unit, the control unit controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range. The imaging apparatus according to claim 1, characterized in that it can switch whether to perform a process of calculating the distance to a subject based on the at least first image signal and the second image signal generated from the imaging unit without using the pulsed light.

6. The imaging apparatus according to claim 1, characterized in that the control unit changes the size of the distance range of the at least first image signal and the second image signal generated by the imaging unit based on the reliability of the at least first image signal and the second image signal which is greater than a predetermined value.

7. A control method for controlling a light-emitting unit that emits pulsed light and an imaging unit that generates at least a first image signal and a second image signal having a predetermined parallax, A control step that controls the emission time of the light-emitting unit and the exposure time of the imaging unit so that the imaging unit exposes the reflected light of the pulsed light within a predetermined distance range, The system includes a distance measurement step that calculates a distance value to a subject and the reliability of the distance value based on at least the first image signal and the second image signal, The control step is characterized by controlling the emission time of the light-emitting unit and the exposure time of the imaging unit to change the magnitude of the distance range of at least the first image signal and the second image signal generated by the imaging unit, based on the reliability.

8. A computer program for controlling each part of the imaging apparatus described in any one of claims 1 to 6 by computer.

Images