Imaging device, imaging method, and computer program

Abstract

To provide an imaging device that can reduce the power consumption of the image sensor. [Solution] The imaging device comprises a light-emitting unit that emits pulsed light, an imaging unit that generates an image signal by exposing the reflected light from a subject when the pulsed light strikes the subject, and a control unit that controls the light emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit, wherein the control unit changes the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range.

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging method, a computer program, and the like.

Background Art

[0002] In recent years, as a device for measuring the distance to a subject, a distance measurement camera system using the triangulation principle has been installed in many automobiles in order to realize the functions of an advanced driving assistance system (ADAS). Examples of the distance measurement camera system include a stereo camera system and an imaging surface phase difference camera system.

[0003] The stereo camera system arranges two cameras in parallel at a predetermined interval, detects the amount of deviation due to the parallax of the subject imaged in the images taken by each camera, and calculates the distance to the subject based on this.

[0004] The imaging surface phase difference distance measurement camera system performs imaging with a single camera equipped with an imaging element called an imaging surface phase difference element, and detects the deviation due to the parallax of the image signals generated when light passing through the imaging optical system is incident on a plurality of pixels formed on the imaging element. Then, the distance to the subject is calculated based on this deviation.

[0005] These distance measurement camera systems may incorrectly calculate the distance to the subject in bad weather. Specifically, bad weather refers to a situation where particles such as rain, fog, and snow obstruct the visibility of the subject.

[0006] In the above camera system, if the camera captures the light diffusely reflected by the particles in front of the subject, an image signal other than the subject may be generated, and the distance to the subject may not be accurately calculated due to the influence of the image signal.

[0007] On the other hand, there is a camera imaging technique called a range gate camera. This technique involves emitting pulsed light in front of the camera at predetermined intervals, and the image sensor inside the camera exposes only the light reflected within the target distance range, thereby clearly imaging only subjects within that range. Hereafter, this technique will be referred to as range gate control. This range gate control makes it possible to clearly image subjects at a predetermined distance, even in bad weather.

[0008] Patent Document 1 proposes an imaging device that combines a distance measuring camera system with range gate control, enabling long-distance photography and distance measurement even in adverse weather conditions. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Patent No. 6293134 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, when combining a distance measuring camera system with range gate control, calculations are required not only for distance measurement by the camera but also for image capture using range gate control. This results in an increase in the total computational processing load of the system. Consequently, the size of the computational processing circuitry required for this processing increases, and the overall power consumption of the system tends to rise.

[0011] Therefore, the present invention aims to provide an imaging device that can reduce the power consumption of the image sensor. [Means for solving the problem]

[0012] One side view of the present invention, A light-emitting unit that emits pulsed light, The imaging unit generates an image signal by exposing the reflected light that strikes the subject with the pulsed light and reflecting it, The system includes a control unit that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the light-emitting unit to reflected light from a subject within a predetermined distance range from the imaging unit. The control unit is characterized by changing the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an imaging device that can reduce the power consumption of the image sensor. [Brief explanation of the drawing]

[0014] [Figure 1] This is a functional block diagram showing an example configuration of the camera 100, light emitter 200, and mobile body 300 according to Embodiment 1. [Figure 2] (A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence, and (B) is a cross-sectional view of the pixel group 400 in the I-I' section in Figure 2(A). [Figure 3] Figures (A) to (D) illustrate the relationship between subject distance and incident light in the image plane phase-detection imaging method. [Figure 4] This figure illustrates an example of the relationship between the propagation of reflected light and exposure timing in range gate control. [Figure 5] This figure shows an example of the light emission and exposure (charge accumulation) control operation for one frame in range gate control. [Figure 6] This figure shows an example of the relationship between the range range and the field of view during imaging in the range gate control according to Embodiment 1. [Figure 7] (A) and (B) are diagrams showing the image range obtained during imaging using range gate control according to Embodiment 1. [Figure 8] This flowchart shows an example of the processing of the imaging method according to Embodiment 1. [Figure 9] This flowchart shows an example of the processing of the imaging method according to Embodiment 2. [Modes for carrying out the invention]

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same member or element is designated by the same reference number, and redundant descriptions are omitted or simplified.

[0016] <Embodiment 1> FIG. 1 is a functional block diagram showing a configuration example of a camera 100, a light emitter 200, and a moving body 300 according to Embodiment 1. Note that the camera 100 functions as an imaging device.

[0017] In addition, in this embodiment, an example of a vehicle such as an automobile is used as the moving body 300 for explanation. However, the moving body may be any movable device such as a train, a ship, an airplane, a robot, a drone, an AGV (Automated Guided Vehicle), or an AMR (Autonomous Mobile Robot).

[0018] Note that some of the functional blocks shown in FIG. 1 are realized by causing a computer (not shown) included in the camera 100, the light emitter 200, and the moving body 300 to execute a computer program stored in a memory as a storage medium (not shown).

[0019] However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) or the like can be used.

[0020] In addition, each of those shown in FIG. 1 does not have to be built in the same housing, and may be configured by separate devices connected via signal paths. Note that the above description regarding FIG. 1 also applies to FIG. 10 in the same manner.

[0021] The camera 100 includes an imaging optical system 101, a photoelectric conversion element 102, an image processing unit 103, a distance measuring unit 104, a camera control unit 105, a memory unit 106, a communication unit 107, etc. The photoelectric conversion element 102 functions as an imaging unit that generates an image signal by exposing the reflected light, which is reflected when pulsed light strikes the subject.

[0022] The imaging optical system 101 can form an image (optical image) of the subject on the photoelectric conversion element 102, and has an exit pupil at a predetermined distance from the photoelectric conversion element 102.

[0023] The photoelectric conversion element 102 is, for example, a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor. The photoelectric conversion element 102 includes, for example, a pixel region in which pixels having a photoelectric conversion function are arranged in two dimensions.

[0024] In this embodiment, the photoelectric conversion element 102 uses an image sensor with an image plane phase difference distance measurement method as a stereo camera system. That is, each pixel region has one microlens and two photoelectric conversion units, and the subject image with parallax formed on the photoelectric conversion element 102 is photoelectrically converted to generate a first image signal and a second image signal, respectively.

[0025] In this embodiment, the image-plane phase-difference type photoelectric conversion element 102 has one microlens and two photoelectric conversion units in each pixel region, but it is not limited to this, as long as it has at least two photoelectric conversion units. For example, the image-plane phase-difference type may be realized with a quad-pixel structure having one microlens and four photoelectric conversion units.

[0026] In this way, the photoelectric conversion element 102, acting as the imaging unit, generates a first image signal and a second image signal having a predetermined parallax. However, the stereo camera system is not limited to this, and for example, a stereo camera consisting of two cameras having parallax may be used.

[0027] Furthermore, as mentioned above, image sensors using the image plane phase-difference method have two photoelectric conversion units, but this is not limited to this; at least one photoelectric conversion unit is sufficient. For example, the image plane phase-difference method can be implemented using a quad-pixel structure.

[0028] The image processing unit 103 performs image processing on the image signal generated by the photoelectric conversion element 102, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal. The output of the image processing unit 103 is supplied to the distance measuring unit 104, the ECU 301 (Electric Control Unit) of the mobile body 300, and the camera control unit 105.

[0029] The distance measuring unit 104 performs image recognition based on the first and second image signals supplied from the image processing unit 103, thereby recognizing surrounding objects such as people and vehicles. It also calculates the distance to the object based on the phase shift of the images corresponding to the parallax between the first and second image signals supplied from the image processing unit 103.

[0030] Specifically, the distance measuring unit 104 calculates the distance value to the subject and the reliability of the distance value based on the first image signal and the second image signal. The image processing unit 103 and the distance measuring unit 104 function as arithmetic processing units that perform predetermined arithmetic processing (image processing, image recognition, distance measurement, etc.) on the image obtained from the image signal.

[0031] The camera control unit 105 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the camera 100.

[0032] Furthermore, the camera control unit 105 functions as a control unit and transmits a reference signal to the camera 100 at predetermined intervals to control the length of the exposure period (charge accumulation time) for each frame and the timing of the control signals.

[0033] In the following description, exposure refers to the operation of charge accumulation, photoelectric conversion, or imaging in the photoelectric conversion element 102 from start to finish. In this embodiment, charge accumulation, photoelectric conversion, and imaging are used interchangeably, and the operations of charge accumulation, photoelectric conversion, and imaging include, for example, the operation of counting the photoelectrically converted signal in the counter 211 circuit when the photoelectric conversion element 102 is an APD.

[0034] Furthermore, the camera control unit 105 sets predetermined values ​​to the light emission control unit 202 via the communication unit 107 and the communication unit 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing synchronized with the reference signal, thereby controlling the light emission period of the light emitter 200.

[0035] In this way, a reference signal synchronized with the reference signal transmitted to the photoelectric conversion element 102 is also transmitted to the light emitter 200, and the light emitter 200 performs light emission control in synchronization with the reference signal. This makes it possible to synchronize the exposure timing of the photoelectric conversion element 102 with the light emission timing of the light emitter 200.

[0036] Here, the camera control unit 105 functions as a control unit that executes control steps to control the emission timing of the light-emitting unit and the exposure (charge accumulation) timing of the imaging unit so that the imaging unit exposes (charge accumulates) the reflected pulse light from a subject within a predetermined distance range.

[0037] The storage unit 106 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 107 is equipped with wireless and wired interfaces and outputs the generated image signals to the outside of the camera 100 and receives various signals from the outside.

[0038] Furthermore, in this embodiment, the communication unit 107 is connected to the communication unit 203 of the light emitter 200, and is also responsible for transmitting the aforementioned reference signal and control commands from the camera control unit 105 to the light emitter 200.

[0039] The light emitter 200 has a light-emitting unit 201 that emits pulsed light, a light-emitting control unit 202, and a communication unit 203. The light-emitting unit 201 is, for example, a near-infrared LED positioned in front of the mobile body 300, and is configured in combination with a lens.

[0040] The wavelength of the pulsed light emitted by the light-emitting unit 201 is not limited to the near-infrared region, but may be in the visible light region, for example. Furthermore, the light-emitting unit 201 may function as a light-emitting device on the mobile body 300, for example, to change the light intensity of a headlight. The light-emitting unit 201 outputs pulsed light for a predetermined emission time in response to a pulse signal output to the light emission control unit 202.

[0041] The light emission control unit 202 receives a reference signal transmitted by the camera control unit 105 of the camera 100 via the communication unit 203, generates a pulse signal at a predetermined timing using the reference signal as a reference, and outputs it to the light emission unit 201.

[0042] Here, the light emission control unit 202 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next.

[0043] The camera control unit 105 sets a predetermined value to the light emission control unit 202 via the communication unit 107 and 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing based on the reference signal, and the light emission period of the light emitter 200 is controlled. In this way, the light emission control unit 202 controls the light emission based on the same signal as the reference signal input to the camera 100.

[0044] The communication unit 203 communicates with the communication unit 107 of the camera 100, receives setting information and reference signals from the camera control unit 105 to the light emission control unit 202, and transmits them to the light emission control unit 202.

[0045] The ECU301 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the vehicle control unit 302.

[0046] The output of the distance measuring unit 104 is supplied to the camera control unit 105, and also to the vehicle control unit 302 and the display unit 303 via the ECU 301. The vehicle control unit 302 functions as a movement control unit that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 301. The display unit 303 functions as a display unit and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 300.

[0047] In this embodiment, the ECU 301 receives distance measurement result information from the distance measurement unit 104 and can perform vehicle stopping control (such as automatic braking) according to the content of the distance measurement result. The ECU 301 also receives distance measurement processing data from the distance measurement unit 104 and transmits it to the display unit 303.

[0048] Based on the output of the ECU 301, the display unit 303 displays various information to the driver of the mobile unit 300, such as images generated by the photoelectric conversion element 102, distance measurement results from the distance measurement unit 104, and the vehicle's driving status, for example, using a GUI.

[0049] Furthermore, the image processing unit 103, distance measuring unit 104, etc. shown in Figure 1 do not necessarily have to be mounted on the mobile body 300. For example, they may be installed on an external terminal provided separately from the mobile body 300 for remotely controlling the mobile body 300 or for monitoring the movement of the mobile body.

[0050] Here, the ranging principle using the image plane phase difference method with the photoelectric conversion element 102 will be explained using Figures 2 and 3. Figure 2(A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence, and (B) is a cross-sectional view of Figure 2(A) II'.

[0051] As shown in Figure 2(A), the photoelectric conversion element 102 is composed of multiple 2x2 pixel groups 400 arranged on a matrix. Each pixel group 400 has four infrared pixels (IR) that detect IR light. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412. The arrangement of each pixel in the pixel group 400 is not limited to this and may be changed depending on the wavelength band of the light emitted from the light emitter 200.

[0052] That is, the pixel group 400 may have, for example, an infrared pixel (IR) for detecting IR light, a red pixel for detecting the color red, a green pixel for detecting green light, and a blue pixel for detecting blue light. Alternatively, the pixel group 400 may have two green pixels for detecting the color green, a red pixel for detecting the color red, and a blue pixel for detecting blue light. In the above pixel arrangement, the two green pixels are arranged diagonally.

[0053] Figure 2(B) is a cross-sectional view of the pixel group 400 in the I-I' section of Figure 2(A), as shown above. Each pixel consists of a light guide layer 414 including a microlens 413 and a light receiving layer 415.

[0054] The light guide layer 414 consists of a microlens 413 for efficiently guiding light incident on the pixels to the light receiving layer 415, a color filter that transmits light in the wavelength band detected by each pixel, and a light guide member having wiring for pixel readout and pixel driving. The light receiving layer 415 converts the light incident on the light guide layer 414 into an electrical signal. The light receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0055] Figures 3(A) to 3(D) illustrate the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 3(A) is a schematic diagram showing the exit pupil 500 of the imaging optical system 101 and the light incident on the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 of the infrared pixel IR of the photoelectric conversion element 102. The photoelectric conversion element 102 has multiple pixels, but for simplicity, we will describe a single infrared pixel IR.

[0056] The microlens 413 of the infrared pixel IR is arranged so as to be optically conjugate to the exit pupil 500 and the light-receiving layer 415. As a result, light passing through the first pupil region 510, which is a partial pupil region contained within the exit pupil 500, is incident on the first photoelectric conversion unit 411, and light passing through the second pupil region 520 is incident on the second photoelectric conversion unit 412. do.

[0057] Each pixel's first photoelectric conversion unit 411 converts the received light into photoelectric signals and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the photoelectric conversion element 102. The first image signal shows the intensity distribution of the first image formed on the photoelectric conversion element 102 by light that has mainly passed through the first pupil region 510.

[0058] Each pixel's second photoelectric conversion unit 412 converts the received light into photoelectric signals and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 contained in the photoelectric conversion element 102. The second image signal shows the intensity distribution of the second image formed on the photoelectric conversion element 102 by light that has mainly passed through the second pupil region 520.

[0059] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 3(B) to (D).

[0060] Figures 3(B), (C), and (D) are schematic diagrams showing the photoelectric conversion element 102 and the imaging optical system 101. In the figures, 511 represents the first light passing through the first pupil region 510, and 521 represents the second light passing through the second pupil region 520.

[0061] Figure 3(B) shows the state when the image is in focus, with the first light 511 and the second light 521 converging on the photoelectric conversion element 102. At this time, the parallax between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is 0.

[0062] Figure 3(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a negative value.

[0063] Figure 3(D) shows the state where the image is defocused in the positive w-axis direction. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a positive value.

[0064] A comparison of Figures 3(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax is proportional to the amount of defocus.

[0065] Therefore, the amount of disparity between the first image signal and the second image signal can be detected using a region-based matching method, such as the block matching method, and converted into a defocus amount. The block matching method is a method in which a region with a high degree of similarity (hereinafter referred to as the disparity matching region) from the other image is matched with a selected region from one image, and the positional difference between that region and the other is defined as the disparity.

[0066] Furthermore, by using the imaging formula of the imaging optical system 101, the amount of defocus on the image side can be converted into the distance to the object. The imaging formula of the imaging optical system 101 is given by the following equation 1, where f is the focal length of the photoelectric conversion element 102, Ipp is the distance from the principal point on the image side to the photoelectric conversion element 102, ΔL is the amount of defocus, and D is the distance to the object.

[0067]

number

[0068] Next, Figure 4 shows an example of the relationship between the propagation of reflected light and exposure timing in range gate control, illustrating the relationship between the propagation of synchrotron radiation and its reflected light from the light emitter 200 and the exposure timing of the camera 100. In Figure 4, the horizontal axis represents distance and the vertical axis represents time.

[0069] Figure 4 illustrates a method for acquiring an image (range-gated image) of the target distance by synchronizing the light emission timing and exposure timing according to the target distance (range-gate control). A camera that acquires a target distance image using this range-gate control is called a range-gate camera.

[0070] First, let's explain the horizontal axis. Fog 610 exists between distance x1 and distance x2, and a vehicle 620 is located at distance x3. Also, in Figure 4, range gate control is used, with the position of camera 100 as the starting point, and range gate images are acquired from there within a predetermined distance range.

[0071] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 620 is located within the target distance range R. Furthermore, the subject image within the target distance range R is photoelectrically converted by the photoelectric conversion element 102 and held as an electric charge in each pixel.

[0072] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission at the light emitter 200, and time tf is defined as the end time of light emission. In this case, the light emission period is tf. Also, when acquiring a range gate image within the target distance range R, with the position of camera 100 as the starting point, the exposure start time (charge accumulation start time) is defined as time t1, and the exposure end time (charge accumulation end time) is defined as time t2.

[0073] Time t1 is the moment when the light emitted from the light emitter 200 at time 0 returns to the camera 100 as reflected light from the vehicle 620. Also, t2 is the moment when the light emitted from the light emitter 200 returns to the camera 100 as reflected light from a point that has traveled a distance R from distance D to the target distance range.

[0074] Furthermore, the timing at which the first reflected light returns to camera 100 is defined as time t3, and the timing at which the last reflected light from fog 610 returns to camera 100 is defined as time t4.

[0075] In range gate control, exposure (charge accumulation) is not performed during the period from time t3 to time t4 when the reflected light from the fog 610 reaches the camera 100, and exposure (charge accumulation) is performed only during the period from time t1 to time t2 when the reflected light from distance D reaches the camera within the target distance range R. This makes it possible to remove the fog 610 while clearly acquiring an image of the vehicle 620.

[0076] Here, we will explain the time it takes for reflected light from an object at distance x to return to camera 100. Let time tr be the timing when the emitted light, which began to be emitted at time 0, strikes an object at distance x and returns to the imaging unit as reflected light.

[0077] At this time, the relationship between the timing time tr of the reflected light returning and the distance x from the object being imaged is given by the following equation 2. Time tr=2x / speed of light c (approximately 3×10^8m / s) (Formula 2)

[0078] As shown in Figure 4, when the imaging range is defined as the target distance range R from distance D, the exposure (charge accumulation) timing time t1 at the starting point of the target distance range R can be obtained by substituting distance D into distance x in equation (2) above, as shown in equation 3 below. Time t1=2D / speed of light c...(Equation 3)

[0079] Furthermore, the exposure (charge accumulation) timing time t2 at the endpoint of the target distance range R can be obtained by substituting the distance D+R into the distance x in equation 2 above and adding the time tf, as shown in equation 4 below. t2=tf+2(D+R) / speed of light c (Equation 4)

[0080] By controlling the time tr from light emission to exposure (charge accumulation) according to the target distance x (the distance at which imaging is desired), range gate control is achieved, enabling clear imaging of subjects at the target distance even if there is fog or other obstructions between the camera and the target distance.

[0081] Figure 5 shows an example of the light emission and exposure (charge accumulation) control operation for one frame in range gate control, illustrating the control operation for obtaining a range gate image per frame.

[0082] In Figure 5, the vertical synchronization signal indicates the frame period of the image, with the period between one low pulse and the next low pulse being one frame time. The light emission control indicates the light emission timing of the light emission unit 201, and the exposure (charge accumulation) control of the photoelectric conversion element indicates the length of the exposure (charge accumulation) period for each frame of the photoelectric conversion element 102 output by the camera control unit 105, as well as the timing of the control signal.

[0083] In range gate control, the light emission period is controlled in a pulsed manner by the camera control unit 105, and exposure (charge accumulation) of the photoelectric conversion element 102 is performed only for reflected light from a specific target distance range R.

[0084] Here, let tf be the emission period from the start to the end of emission, t1 be the time from the start of emission to the start of exposure (charge accumulation), and t2 be the time from the start of emission to the end of exposure (charge accumulation). In this case, t1 represents the period from the start of emission until the light reaches a specific target distance range R and the reflected light returns to camera 100, and the time from t1 to t2 is the period during which the reflected light from the specific target distance range R is exposed.

[0085] For range gate control to function correctly, it is necessary to synchronize the timing of light emission start and exposure start according to a predetermined target distance range. In this embodiment, the camera control unit 105 synchronizes the timing by transmitting the same reference signal to the photoelectric conversion element 102 and the light emission control unit 202.

[0086] As shown in the timing chart of Figure 5, the period from the start of one emission to the start of the next emission constitutes a range gate operation cycle. During one range gate operation cycle, the light received by the photoelectric conversion element 102 is converted into an electric charge and held within the photoelectric conversion element 102.

[0087] In this state, the next range gate operation cycle is performed, and the light newly received by the photoelectric conversion element 102 is converted into an electric charge and added to the charge held within the photoelectric conversion element 102. The interval between one emission and the next emission is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 100.

[0088] As shown in Figure 5, within one frame time, the range gate operation cycle is performed a predetermined number of times. The charge that was last added and held within one frame time is sent to the memory in the photoelectric conversion element 102, and the charge that was subsequently added and held is reset.

[0089] In this embodiment, since the exposure (charge accumulation) period is synchronized with the light emission from the light emitter 200, it is possible to obtain a clear image for a targeted distance range even under adverse weather conditions such as fog.

[0090] Next, when imaging using range gate control, the method for calculating the pixel range (operating area) required for the photoelectric conversion element 102 to perform a predetermined operation will be explained using Figures 6, 7, and 8.

[0091] Figure 6 shows an example of the relationship between the range range and the field of view during imaging in the range gate control according to Embodiment 1. The horizontal axis represents the horizontal distance from the camera 100, and the vertical axis represents the vertical height from the ground surface. Figures 7(A) and 7(B) show the image range obtained during imaging with the range gate control according to Embodiment 1.

[0092] The orientation and position of the camera 100 are such that the optical axis of the imaging optical system 101 is parallel to the horizontal direction and its vertical height from the ground surface is Hc. φc is the vertical field of view of the camera 100, and it has a field of view that is symmetrical with respect to the optical axis of the imaging optical system 101.

[0093] φ1 is the field of view that can be captured when imaging with distance D1 and range range R1 using range gate control, and the range of φ1 is given by the following equation 5. φ1=atan(Hc / D1)+atan(φc / 2)...(Formula 5)

[0094] At this time, the range of reflected light that passes through the imaging optical system 101 and enters the pixel area of ​​the photoelectric conversion element 102 is, as shown in Figure 7(A), the pixel area corresponding to φ1, which is narrower vertically downwards than the vertical range of the pixel area corresponding to φc.

[0095] In Figures 7(A) and (B), the upward direction represents the upward direction of the subject, i.e., the vertical height direction, and the downward direction on the light-receiving surface of the photoelectric conversion element 102. This is because the imaging optical system 101 causes the image of the subject to be formed upside down on the light-receiving surface of the photoelectric conversion element 102.

[0096] φ2 is the field of view that can be captured when imaging with distance D2 and range range R2 using range gate control, and the range of φ2 is given by the following equation 6. φ2=atan(Hc / D2)+atan(φc / 2)...(Formula 6)

[0097] At this time, the range of reflected light that passes through the imaging optical system 101 and enters the pixel area of ​​the photoelectric conversion element 102 is, as shown in Figure 7(B), the pixel area corresponding to φ2, which is narrower vertically downwards than the vertical range of the pixel area corresponding to φc.

[0098] Since D2 > D1, equations 5 and 6 show that φ2 < φ1. Therefore, the pixel range (operating area) of the photoelectric conversion element 102, which is at a greater distance (D2), is narrower than the distance D1.

[0099] Therefore, in this embodiment, the operating area (pixel range) for performing a predetermined operation in the imaging unit is determined based on the distance of the subject on the side closer to the imaging unit within a predetermined distance range. That is, the larger the distance of the subject on the side closer to the imaging unit within the predetermined distance range, the smaller the operating area (pixel range) for performing the predetermined operation in the imaging unit.

[0100] Furthermore, the greater the distance of the subject closer to the imaging unit within a predetermined distance range, the smaller the range of the operating area (pixel range) for performing a predetermined operation in the imaging unit, in the vertical direction of the operating area.

[0101] Furthermore, the greater the distance of the subject closer to the imaging unit within a predetermined distance range, the greater the operating area (pixel range) for performing a predetermined operation in the imaging unit will be located in the upper part of the image relative to the subject. Here, the upper part of the image relative to the subject refers to the area located towards the top of the image relative to the subject, as shown in Figure 7(B).

[0102] Figure 8 is a flowchart showing an example of the imaging method according to Embodiment 1. The CPU and other components within the camera control unit 105 execute a computer program stored in memory, which sequentially performs the operations of each step in the flowchart in Figure 8.

[0103] Furthermore, in Figure 8, for example, the processing flow shown in Figure 8 is repeatedly executed at a fixed frame period. That is, the imaging unit is driven at a fixed period regardless of the size of the operating area. Note that the following explanation describes the case where distance D1 and range R1 are captured, but the same processing flow applies when capturing distance D2 and range R2.

[0104] First, in step S11, the camera control unit 105 determines the vertical height Hc of the camera 100 from the ground surface. Here, Hc is the vertical height of the camera 100 from the ground surface when the camera 100 is mounted on a stationary mobile body 300 on the ground surface. This value is read from a predetermined value that has been stored in the storage unit 106 beforehand. Alternatively, the height Ho of the subject may be estimated based on the image recognition results of the previous field.

[0105] Next, in step S12, the camera control unit 105 determines the emission time of the light emitter 200, the exposure (charge accumulation) time of the camera 100, and the exposure (charge accumulation) timing for capturing images at distance D1 and range range R1 using range gate control. That is, in step S12, the emission timing of the light emitter and the exposure timing of the imaging unit are controlled so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit.

[0106] Next, in step S13, the camera control unit 105 determines the operating area (pixel range) for performing a predetermined operation from among all the pixels of the photoelectric conversion element 102. Here, the range φ1 that can be photographed using range gate control is calculated using the above-mentioned equation 5, with distance D1 and range range R1, and the range corresponding to φ1 is used as the operating area (pixel range).

[0107] In other words, in step S13, the operating area for performing a predetermined operation in the imaging unit is changed based on a predetermined distance range. Here, steps S12 and S13 function as control steps.

[0108] The above-mentioned predetermined operation includes at least one operation in the photoelectric conversion element 102, such as charge storage, photoelectric conversion, A / D conversion, avalanche multiplication, noise reduction, dark current reduction, and readout. In this embodiment, the operating area (pixel range) for performing the above-mentioned predetermined operation in the imaging unit is changed based on a predetermined distance range.

[0109] Next, in step S14, the camera control unit 105 performs predetermined operations such as photoelectric conversion and reading using range gate control within the light emission time, exposure time, exposure timing determined in step S12, and the operating area (pixel range) determined in step S13. That is, it operates the light emitter 200 and the photoelectric conversion element 102 using range gate control. As a result, the image signal is acquired by the photoelectric conversion element 102 and input to the image processing unit 103.

[0110] Next, in step S15, the camera control unit 105 performs a waiting process for a duration equivalent to the operating time of the non-operating pixel region among the multiple pixels of the photoelectric conversion element 102, before starting the next image capture.

[0111] This makes it possible to reduce the average pixel range (operating area) required for the photoelectric conversion element 102 to perform a predetermined operation per unit time when imaging is performed multiple times at predetermined time intervals. In other words, it becomes possible to reduce the power consumption of the photoelectric conversion element 102.

[0112] In this embodiment, the power consumption of the photoelectric conversion element 102 can be reduced by varying the operating area (pixel range) for performing predetermined operations in the imaging unit according to the distance D of range gate imaging. Furthermore, since the amount of the image signal output from the photoelectric conversion element 102 is reduced, the power consumption of the image processing unit 103 and the distance measuring unit 104, which receive the image signal and perform calculations, can be reduced.

[0113] <Embodiment 2> Embodiment 2 of the present invention will be described below. Note that the same parts as in Embodiment 1 will not be described.

[0114] Figure 9 is a flowchart showing an example of the imaging method according to Embodiment 2. The CPU and other components within the camera control unit 105 execute a computer program stored in memory, which sequentially performs the operations of each step in the flowchart in Figure 9.

[0115] Steps S11 to S14 are the same as in Embodiment 1, but in this embodiment, step S15 in Embodiment 1 is omitted, and after step S14 is completed, the process returns to step S11 and repeats the processing flow shown in Figure 9. That is, in this embodiment, the imaging unit is driven at a cycle corresponding to the size of the operating area.

[0116] This makes it possible to start the next image capture as soon as the predetermined operation in the above operating area (pixel range) of the photoelectric conversion element 102 is completed when multiple images are taken, thereby improving the frame rate of the camera 100.

[0117] Furthermore, for example, the orientation of camera 100 may not be parallel to the horizontal direction but may have an angle, and the vertical height Hc may be measured in real time when the mobile body 300 is moving, rather than when it is stationary.

[0118] In other words, the attitude changes of the mobile body 300 may be measured in real time using an attitude sensor or the like, and the attitude and height of the camera 100 may be corrected based on that. Also, the vertical field of view of the camera 100 may be asymmetrical with respect to the optical axis.

[0119] Furthermore, with range gate control, the operating area (pixel range) for performing a predetermined operation in the imaging unit may be multiple distances or multiple range ranges. Even in that case, the shooting angle of view should be calculated by considering the camera 100's orientation, vertical angle of view, operating area (pixel range) due to range gate control, etc., using the same approach as described above.

[0120] Furthermore, the operating area (pixel range) for performing a predetermined operation in the photoelectric conversion element 102 is not limited to the determination method described above. For example, it may be selected from a plurality of predetermined operating areas (pixel ranges) according to the imaging distance of range gate control.

[0121] In the above embodiment, the operating area (pixel range) for performing a predetermined operation (e.g., photoelectric conversion or readout operation) in the photoelectric conversion element 102 is determined to be changed based on a predetermined distance range. However, in this case, the size of the calculation processing area for performing calculations within the readout pixel range may be made even smaller than the above pixel range.

[0122] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0123] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0124] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the distance measuring camera system, etc., via a network or various storage media.

[0125] The computer (or CPU, MPU, etc.) in the distance measuring camera system, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0126] (Configuration 1) An imaging device comprising: a light-emitting unit that emits pulsed light; an imaging unit that exposes the reflected light from a subject when the pulsed light strikes the subject and generates an image signal; and a control unit that controls the light emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit, wherein the control unit changes the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range.

[0127] (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the predetermined operation includes at least one of the following in the imaging unit: charge accumulation, photoelectric conversion, A / D conversion, avalanche multiplication, noise reduction, dark current reduction, and readout.

[0128] (Configuration 3) The imaging apparatus according to Configuration 1 or 2, characterized in that the control unit determines the operating area based on the distance of the subject on the side closer to the imaging unit within the predetermined distance range.

[0129] (Configuration 4) The imaging apparatus according to Configuration 3, characterized in that the control unit makes the operating area smaller as the distance to the subject on the side closer to the imaging unit within the predetermined distance range increases.

[0130] (Configuration 5) The imaging device according to Configuration 4, characterized in that the control unit makes the operating area smaller in the vertical direction of the operating area as the distance of the subject closer to the imaging unit within the predetermined distance range increases.

[0131] (Configuration 6) The imaging apparatus according to Configuration 5, characterized in that the control unit sets the operating area to a range that is closer to the upper part of the subject in the image as the distance of the subject on the side closer to the imaging unit within the predetermined distance range increases.

[0132] (Configuration 7) The imaging apparatus according to any one of Configurations 1 to 6, characterized in that the control unit drives the imaging unit at a constant period regardless of the size of the operating area.

[0133] (Configuration 8) The imaging device according to any one of Configurations 1 to 6, characterized in that the control unit drives the imaging unit at a period corresponding to the size of the operating area.

[0134] (Method) An imaging method comprising controlling a light-emitting unit that emits pulsed light and an imaging unit that exposes the reflected light, which is reflected when the pulsed light strikes a subject, to generate an image signal, An imaging method characterized by having a control step that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the imaging unit to reflected light from a subject within a predetermined distance range from the imaging unit, and changes the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range.

[0135] (Program) A computer program for controlling each part of the imaging device described in any one of configurations 1 to 8 by computer. [Explanation of Symbols]

[0136] 100: Camera 101: Imaging Optical System 102: Photoelectric conversion element 103: Image Processing Unit 104: Distance measurement section 105: Camera Control Unit 106: Storage section 107: Communications Department 200: Light emitter 201: Light-emitting part 202: Light emission control unit 203: Communications Department 300: Mobile 301: ECU 302: Vehicle Control Unit 303: Display section 400: Pixel group 411: First photoelectric conversion unit 412: Second photoelectric conversion unit 413: Microlens 414: Light guide layer 415: Light-receiving layer 500: Exit pupil 510: First pupil region 520: Second Eye Region 511: The First Light 521: The Second Light 610: Fog 620: Vehicle

Claims

1. A light-emitting unit that emits pulsed light, The imaging unit generates an image signal by exposing the reflected light that strikes the subject with the pulsed light and reflecting it, The system includes a control unit that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the light-emitting unit to reflected light from a subject within a predetermined distance range from the imaging unit. The imaging apparatus is characterized in that the control unit changes the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range.

2. The imaging apparatus according to claim 1, characterized in that the predetermined operation includes at least one of the following in the imaging unit: charge accumulation, photoelectric conversion, A / D conversion, avalanche multiplication, noise reduction, dark current reduction, and readout.

3. The imaging apparatus according to claim 1, characterized in that the control unit determines the operating area based on the distance of the subject on the side closer to the imaging unit within the predetermined distance range.

4. The imaging apparatus according to claim 3, characterized in that the control unit makes the operating area smaller as the distance of the subject closer to the imaging unit within the predetermined distance range increases.

5. The imaging apparatus according to claim 4, characterized in that the control unit makes the operating area smaller in the vertical direction of the operating area as the distance of the subject closer to the imaging unit within the predetermined distance range increases.

6. The imaging apparatus according to claim 5, characterized in that the control unit sets the operating area to a range that is closer to the upper part of the subject in the image as the distance of the subject on the side closer to the imaging unit within the predetermined distance range increases.

7. The imaging apparatus according to claim 1, characterized in that the control unit drives the imaging unit at a constant period regardless of the size of the operating area.

8. The imaging apparatus according to claim 1, characterized in that the control unit drives the imaging unit at a period corresponding to the size of the operating area.

9. An imaging method for controlling a light-emitting unit that emits pulsed light and an imaging unit that exposes the reflected light, which is reflected when the pulsed light strikes a subject, and generates an image signal, An imaging method characterized by having a control step that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the imaging unit to reflected light from a subject within a predetermined distance range from the imaging unit, and changes the operating area for performing a predetermined operation in the imaging unit based on the predetermined distance range.

10. A computer program for controlling each part of the imaging apparatus described in any one of claims 1 to 8 by computer.

Images