Imaging device, mobile device, imaging method, and computer program

Abstract

To provide an imaging device that can reduce the amount of computational processing required. [Solution] The imaging device comprises a light-emitting unit that emits pulsed light, an imaging unit that generates an image signal by exposing the reflected light from a subject when the pulsed light strikes it, and a control unit that controls the light emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit, and an arithmetic processing unit that performs predetermined arithmetic processing on an image obtained from the image signal, wherein the control unit determines the arithmetic processing range for performing the arithmetic processing based on the predetermined distance range.

Description

Technical Field

[0001] The present invention relates to an imaging device, a moving body, an imaging method, a computer program, and the like.

Background Art

[0002] In recent years, a distance measurement camera system using the triangulation principle has been mounted on many automobiles in order to realize the function of an advanced driver assistance system (ADAS) as a device for measuring the distance to a subject. Examples of the distance measurement camera system include a stereo camera system and an imaging surface phase difference camera system.

[0003] The stereo camera system arranges two cameras in parallel at a predetermined interval, detects the amount of displacement due to the parallax of the subject imaged in the images taken by each camera, and calculates the distance to the subject based on this.

[0004] The imaging surface phase difference distance measurement camera system performs imaging with a single camera provided with an imaging element called an imaging surface phase difference element, and detects the displacement due to the parallax of the image signals generated by light incident on a plurality of pixels formed on the imaging element after passing through the imaging optical system, and calculates the distance to the subject based on this.

[0005] These distance measurement camera systems may erroneously calculate the distance to the subject in bad weather. Specifically, bad weather refers to a situation where particles such as rain, fog, and snow obstruct the visibility of the subject.

[0006] In the above camera system, if the camera captures the light diffusely reflected by the particles in front of the subject, an image signal other than the subject may be generated, and the distance to the subject may not be accurately calculated due to the influence of the image signal.

[0007] On the other hand, there is a camera imaging technique called a range gate camera. This technique involves emitting pulsed light in front of the camera at predetermined intervals, and the image sensor inside the camera exposes only the light reflected within the target distance range, thereby clearly imaging only subjects within that range. Hereafter, this technique will be referred to as range gate control. This range gate control makes it possible to clearly image subjects at a predetermined distance, even in bad weather.

[0008] Patent Document 1 proposes an imaging device that combines a distance measuring camera system with range gate control, enabling long-distance photography and distance measurement even in adverse weather conditions. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Patent No. 6293134 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, when combining a distance measuring camera system with range gate control, calculations are required not only for distance measurement by the distance measuring camera but also for image capture using range gate control. This presents a challenge in that the total computational processing load of the system increases. Consequently, the size of the computational processing circuit required for this processing increases, and the power consumption of the computational processing circuit tends to increase.

[0011] Therefore, one of the objectives of the present invention is to provide an imaging device that can reduce the amount of computational processing required. [Means for solving the problem]

[0012] In an imaging device, the present invention relates to: A light-emitting unit that emits pulsed light, The imaging unit generates an image signal by exposing the reflected light that strikes the subject with the pulsed light and reflecting it, A control unit that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the imaging unit to reflected light from a subject within a predetermined distance range from the imaging unit, A calculation processing unit that performs predetermined calculations on an image obtained from the aforementioned image signal, Equipped with, The control unit determines the calculation processing range for performing the calculation based on the predetermined distance range. It is characterized by the following. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an imaging device that can reduce the amount of computational processing required. [Brief explanation of the drawing]

[0014] [Figure 1] This is a functional block diagram showing an example configuration of the camera 100, light emitter 200, and mobile body 300 according to Embodiment 1. [Figure 2] (A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence, and (B) is a cross-sectional view of the pixel group 400 in the I-I' section in Figure 2(A). [Figure 3] Figures (A) to (D) illustrate the relationship between subject distance and incident light in the image plane phase-detection imaging method. [Figure 4] This figure illustrates an example of the relationship between the propagation of reflected light and exposure timing in range gate control. [Figure 5] This figure shows an example of the light emission and exposure (charge accumulation) control operation for one frame in range gate control. [Figure 6] This figure shows an example of the relationship between the range range and the field of view during imaging in the range gate control according to Embodiment 1. [Figure 7] (A) and (B) are diagrams showing the image range obtained during imaging using range gate control according to Embodiment 1. [Figure 8] This flowchart shows an example of the processing of the imaging method according to Embodiment 1. [Figure 9]It is a diagram showing an example of the relationship between the range and the shooting angle during imaging in range gate control according to Embodiment 2. [Figure 10] (A) and (B) are diagrams showing the image ranges obtained during imaging in range gate control according to Embodiment 2. [Figure 11] It is a flowchart showing an example of the processing of the imaging method according to Embodiment 2.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are given the same reference numerals, and duplicate descriptions are omitted or simplified.

[0016] <Embodiment 1> FIG. 1 is a functional block diagram showing a configuration example of the camera 100, the light emitter 200, and the moving body 300 according to Embodiment 1. Note that the camera 100 functions as an imaging device.

[0017] Also, in the present embodiment, an example of a vehicle such as an automobile is used as the moving body 300 for explanation. However, the moving body may be any movable device such as a train, a ship, an airplane, a robot, a drone, an AGV (Automated Guided Vehicle), or an AMR (Autonomous Mobile Robot).

[0018] Note that some of the functional blocks shown in FIG. 1 are realized by causing a computer (not shown) included in the camera 100, the light emitter 200, and the moving body 300 to execute a computer program stored in a memory as a storage medium (not shown).

[0019] However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) can be used.

[0020] Furthermore, the components shown in Figure 1 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. The above explanation regarding Figure 1 also applies to Figure 10.

[0021] The camera 100 includes an imaging optical system 101, a photoelectric conversion element 102, an image processing unit 103, a distance measuring unit 104, a camera control unit 105, a memory unit 106, a communication unit 107, etc. The photoelectric conversion element 102 functions as an imaging unit that generates an image signal by exposing the reflected light, which is reflected when pulsed light strikes the subject.

[0022] The imaging optical system 101 can form an image (optical image) of the subject on the photoelectric conversion element 102, and has an exit pupil at a predetermined distance from the photoelectric conversion element 102.

[0023] The photoelectric conversion element 102 is, for example, a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor. The photoelectric conversion element 102 includes, for example, a pixel region in which pixels having a photoelectric conversion function are arranged in two dimensions.

[0024] In this embodiment, the photoelectric conversion element 102 uses an image sensor with an image plane phase difference distance measurement method as a stereo camera system. That is, each pixel region has one microlens and two photoelectric conversion units, and the subject image with parallax formed on the photoelectric conversion element 102 is photoelectrically converted to generate a first image signal and a second image signal, respectively.

[0025] In this embodiment, the image-plane phase-difference type photoelectric conversion element 102 has one microlens and two photoelectric conversion units in each pixel region, but it is not limited to this, as long as it has at least two photoelectric conversion units. For example, the image-plane phase-difference type may be realized with a quad-pixel structure having one microlens and four photoelectric conversion units.

[0026] In this way, the photoelectric conversion element 102, acting as the imaging unit, generates a first image signal and a second image signal having a predetermined parallax. However, the stereo camera system is not limited to this, and for example, a stereo camera consisting of two cameras having parallax may be used.

[0027] Furthermore, as mentioned above, image sensors using the image plane phase-difference method have two photoelectric conversion units, but this is not limited to this; at least one photoelectric conversion unit is sufficient. For example, the image plane phase-difference method can be implemented using a quad-pixel structure.

[0028] The image processing unit 103 performs image processing on the image signal generated by the photoelectric conversion element 102, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal. The output of the image processing unit 103 is supplied to the distance measuring unit 104, the ECU 301 (Electric Control Unit) of the mobile body 300, and the camera control unit 105.

[0029] The distance measuring unit 104 performs image recognition based on the first and second image signals supplied from the image processing unit 103, thereby recognizing surrounding objects such as people and vehicles. It also calculates the distance to the object based on the phase shift of the images corresponding to the parallax between the first and second image signals supplied from the image processing unit 103.

[0030] Specifically, the distance measuring unit 104 calculates the distance value to the subject and the reliability of the distance value based on the first image signal and the second image signal. The image processing unit 103 and the distance measuring unit 104 function as arithmetic processing units that perform predetermined arithmetic processing (image processing, image recognition, distance measurement, etc.) on the image obtained from the image signal.

[0031] The camera control unit 105 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the camera 100.

[0032] Furthermore, the camera control unit 105 functions as a control unit and transmits a reference signal to the camera 100 at predetermined intervals to control the length of the exposure period (charge accumulation time) for each frame and the timing of the control signals.

[0033] In the following description, exposure refers to the operation of charge accumulation, photoelectric conversion, or imaging in the photoelectric conversion element 102 from start to finish. In this embodiment, charge accumulation, photoelectric conversion, and imaging are used interchangeably, and the operations of charge accumulation, photoelectric conversion, and imaging include, for example, the operation of counting the photoelectrically converted signal in the counter 211 circuit when the photoelectric conversion element 102 is an APD.

[0034] Furthermore, the camera control unit 105 sets predetermined values ​​to the light emission control unit 202 via the communication unit 107 and the communication unit 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing synchronized with the reference signal, thereby controlling the light emission period of the light emitter 200.

[0035] In this way, a reference signal synchronized with the reference signal transmitted to the photoelectric conversion element 102 is also transmitted to the light emitter 200, and the light emitter 200 performs light emission control in synchronization with the reference signal. This makes it possible to synchronize the exposure timing of the photoelectric conversion element 102 with the light emission timing of the light emitter 200.

[0036] Here, the camera control unit 105 functions as a control unit that executes control steps to control the emission timing of the light-emitting unit and the exposure (charge accumulation) timing of the imaging unit so that the imaging unit exposes (charge accumulates) the reflected pulse light from a subject within a predetermined distance range.

[0037] The storage unit 106 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 107 is equipped with wireless and wired interfaces and outputs the generated image signals to the outside of the camera 100 and receives various signals from the outside.

[0038] Furthermore, in this embodiment, the communication unit 107 is connected to the communication unit 203 of the light emitter 200, and is also responsible for transmitting the aforementioned reference signal and control commands from the camera control unit 105 to the light emitter 200.

[0039] The light emitter 200 has a light-emitting unit 201 that emits pulsed light, a light-emitting control unit 202, and a communication unit 203. The light-emitting unit 201 is, for example, a near-infrared LED positioned in front of the mobile body 300, and is configured in combination with a lens.

[0040] The wavelength of the pulsed light emitted by the light-emitting unit 201 is not limited to the near-infrared region, but may be in the visible light region, for example. Furthermore, the light-emitting unit 201 may function as a light-emitting device on the mobile body 300, for example, to change the light intensity of a headlight. The light-emitting unit 201 outputs pulsed light for a predetermined emission time in response to a pulse signal output to the light emission control unit 202.

[0041] The light emission control unit 202 receives a reference signal transmitted by the camera control unit 105 of the camera 100 via the communication unit 203, generates a pulse signal at a predetermined timing using the reference signal as a reference, and outputs it to the light emission unit 201.

[0042] Here, the light emission control unit 202 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next.

[0043] The camera control unit 105 sets a predetermined value to the light emission control unit 202 via the communication unit 107 and 203, so that a pulse signal is output to the light emission unit 201 at a predetermined timing based on the reference signal, and the light emission period of the light emitter 200 is controlled. In this way, the light emission control unit 202 controls the light emission based on the same signal as the reference signal input to the camera 100.

[0044] The communication unit 203 communicates with the communication unit 107 of the camera 100, receives setting information and reference signals from the camera control unit 105 to the light emission control unit 202, and transmits them to the light emission control unit 202.

[0045] The ECU301 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control each part of the vehicle control unit 302.

[0046] The output of the distance measuring unit 104 is supplied to the camera control unit 105, and also to the vehicle control unit 302 and the display unit 303 via the ECU 301. The vehicle control unit 302 functions as a movement control unit that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 301. The display unit 303 functions as a display unit and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 300.

[0047] In this embodiment, the ECU 301 receives distance measurement result information from the distance measurement unit 104 and can perform vehicle stopping control (such as automatic braking) according to the content of the distance measurement result. The ECU 301 also receives distance measurement processing data from the distance measurement unit 104 and transmits it to the display unit 303.

[0048] Based on the output of the ECU 301, the display unit 303 displays various information to the driver of the mobile unit 300, such as images generated by the photoelectric conversion element 102, distance measurement results from the distance measurement unit 104, and the vehicle's driving status, for example, using a GUI.

[0049] Furthermore, the image processing unit 103, distance measuring unit 104, etc. shown in Figure 1 do not necessarily have to be mounted on the mobile body 300. For example, they may be installed on an external terminal provided separately from the mobile body 300 for remotely controlling the mobile body 300 or for monitoring the movement of the mobile body.

[0050] Here, the ranging principle using the image plane phase difference method with the photoelectric conversion element 102 will be explained using Figures 2 and 3. Figure 2(A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence, and (B) is a cross-sectional view of Figure 2(A) II'.

[0051] As shown in Figure 2(A), the photoelectric conversion element 102 is composed of multiple 2x2 pixel groups 400 arranged on a matrix. Each pixel group 400 has four infrared pixels (IR) that detect IR light. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412. The arrangement of each pixel in the pixel group 400 is not limited to this and may be changed depending on the wavelength band of the light emitted from the light emitter 200.

[0052] That is, the pixel group 400 may have, for example, an infrared pixel (IR) for detecting IR light, a red pixel for detecting the color red, a green pixel for detecting green light, and a blue pixel for detecting blue light. Alternatively, the pixel group 400 may have two green pixels for detecting the color green, a red pixel for detecting the color red, and a blue pixel for detecting blue light. In the above pixel arrangement, the two green pixels are arranged diagonally.

[0053] Figure 2(B) is a cross-sectional view of the pixel group 400 in the I-I' section of Figure 2(A), as shown above. Each pixel consists of a light guide layer 414 including a microlens 413 and a light receiving layer 415.

[0054] The light guide layer 414 consists of a microlens 413 for efficiently guiding light incident on the pixels to the light receiving layer 415, a color filter that transmits light in the wavelength band detected by each pixel, and a light guide member having wiring for pixel readout and pixel driving. The light receiving layer 415 converts the light incident on the light guide layer 414 into an electrical signal. The light receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0055] Figures 3(A) to 3(D) illustrate the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 3(A) is a schematic diagram showing the exit pupil 500 of the imaging optical system 101 and the light incident on the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 of the infrared pixel IR of the photoelectric conversion element 102. The photoelectric conversion element 102 has multiple pixels, but for simplicity, we will describe a single infrared pixel IR.

[0056] The microlens 413 of the infrared pixel IR is arranged so as to be optically conjugate to the exit pupil 500 and the light-receiving layer 415. As a result, light passing through the first pupil region 510, which is a partial pupil region contained within the exit pupil 500, is incident on the first photoelectric conversion unit 411, and light passing through the second pupil region 520 is incident on the second photoelectric conversion unit 412. do.

[0057] Each pixel's first photoelectric conversion unit 411 converts the received light into photoelectric signals and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the photoelectric conversion element 102. The first image signal shows the intensity distribution of the first image formed on the photoelectric conversion element 102 by light that has mainly passed through the first pupil region 510.

[0058] Each pixel's second photoelectric conversion unit 412 converts the received light into photoelectric signals and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 contained in the photoelectric conversion element 102. The second image signal shows the intensity distribution of the second image formed on the photoelectric conversion element 102 by light that has mainly passed through the second pupil region 520.

[0059] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 3(B) to (D).

[0060] Figures 3(B), (C), and (D) are schematic diagrams showing the photoelectric conversion element 102 and the imaging optical system 101. In the figures, 511 represents the first light passing through the first pupil region 510, and 521 represents the second light passing through the second pupil region 520.

[0061] Figure 3(B) shows the state when the image is in focus, with the first light 511 and the second light 521 converging on the photoelectric conversion element 102. At this time, the parallax between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is 0.

[0062] Figure 3(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a negative value.

[0063] Figure 3(D) shows the state where the image is defocused in the positive w-axis direction. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a positive value.

[0064] A comparison of Figures 3(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax is proportional to the amount of defocus.

[0065] Therefore, the amount of disparity between the first image signal and the second image signal can be detected using a region-based matching method, such as the block matching method, and converted into a defocus amount. The block matching method is a method in which a region with a high degree of similarity (hereinafter referred to as the disparity matching region) from the other image is matched with a selected region from one image, and the positional difference between that region and the other is defined as the disparity.

[0066] Furthermore, by using the imaging formula of the imaging optical system 101, the amount of defocus on the image side can be converted into the distance to the object. The imaging formula of the imaging optical system 101 is given by the following equation 1, where f is the focal length of the photoelectric conversion element 102, Ipp is the distance from the principal point on the image side to the photoelectric conversion element 102, ΔL is the amount of defocus, and D is the distance to the object.

[0067]

number

[0068] Next, Figure 4 shows an example of the relationship between the propagation of reflected light and exposure timing in range gate control, illustrating the relationship between the propagation of synchrotron radiation and its reflected light from the light emitter 200 and the exposure timing of the camera 100. In Figure 4, the horizontal axis represents distance and the vertical axis represents time.

[0069] Figure 4 illustrates a method for acquiring an image (range-gated image) of the target distance by synchronizing the light emission timing and exposure timing according to the target distance (range-gate control). A camera that acquires a target distance image using this range-gate control is called a range-gate camera.

[0070] First, let's explain the horizontal axis. Fog 610 exists between distance x1 and distance x2, and a vehicle 620 is located at distance x3. Also, in Figure 4, range gate control is used, with the position of camera 100 as the starting point, and range gate images are acquired from there within a predetermined distance range.

[0071] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 620 is located within the target distance range R. Furthermore, the subject image within the target distance range R is photoelectrically converted by the photoelectric conversion element 102 and held as an electric charge in each pixel.

[0072] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission at the light emitter 200, and time tf is defined as the end time of light emission. In this case, the light emission period is tf. Also, when acquiring a range gate image within the target distance range R, with the position of camera 100 as the starting point, the exposure start time (charge accumulation start time) is defined as time t1, and the exposure end time (charge accumulation end time) is defined as time t2.

[0073] Time t1 is the moment when the light emitted from the light emitter 200 at time 0 returns to the camera 100 as reflected light from the vehicle 620. Also, t2 is the moment when the light emitted from the light emitter 200 returns to the camera 100 as reflected light from a point that has traveled a distance R from distance D to the target distance range.

[0074] Furthermore, time t3 is defined as the timing when the first reflected light returns to camera 100, and time t4 is defined as the timing when the last reflected light from fog 610 returns to camera 100.

[0075] In range gate control, exposure (charge accumulation) is not performed during the period from time t3 to time t4 when the reflected light from the fog 610 reaches the camera 100, and exposure (charge accumulation) is performed only during the period from time t1 to time t2 when the reflected light from distance D reaches the camera within the target distance range R. This makes it possible to remove the fog 610 while clearly acquiring an image of the vehicle 620.

[0076] Here, we will explain the time it takes for reflected light from an object at distance x to return to camera 100. Let time tr be the timing when the emitted light, which began to be emitted at time 0, strikes an object at distance x and returns to the imaging unit as reflected light.

[0077] At this time, the relationship between the timing time tr of the reflected light returning and the distance x from the object being imaged is given by the following equation 2. Time tr=2x / speed of light c (approximately 3×10^8m / s) (Formula 2)

[0078] As shown in Figure 4, when the imaging range is defined as the target distance range R from distance D, the exposure (charge accumulation) timing time t1 at the starting point of the target distance range R can be obtained by substituting distance D into distance x in equation (2) above, as shown in equation 3 below. Time t1=2D / speed of light c...(Formula 3)

[0079] Furthermore, the exposure (charge accumulation) timing time t2 at the endpoint of the target distance range R can be obtained by substituting the distance D+R into the distance x in equation 2 above and adding the time tf, as shown in equation 4 below. t2=tf+2(D+R) / speed of light c (Equation 4)

[0080] By controlling the time tr from light emission to exposure (charge accumulation) according to the target distance x (the distance at which imaging is desired), range gate control is achieved, enabling clear imaging of subjects at the target distance even if there is fog or other obstructions between the camera and the target distance.

[0081] Figure 5 shows an example of the light emission and exposure (charge accumulation) control operation for one frame in range gate control, illustrating the control operation for obtaining a range gate image per frame.

[0082] In Figure 5, the vertical synchronization signal indicates the frame period of the image, with the period between one low pulse and the next low pulse being one frame time. The light emission control indicates the light emission timing of the light emission unit 201, and the exposure (charge accumulation) control of the photoelectric conversion element indicates the length of the exposure (charge accumulation) period for each frame of the photoelectric conversion element 102 output by the camera control unit 105, as well as the timing of the control signal.

[0083] In range gate control, the light emission period is controlled in a pulsed manner by the camera control unit 105, and exposure (charge accumulation) of the photoelectric conversion element 102 is performed only for reflected light from a specific target distance range R.

[0084] Here, let tf be the emission period from the start to the end of emission, t1 be the time from the start of emission to the start of exposure (charge accumulation), and t2 be the time from the start of emission to the end of exposure (charge accumulation). In this case, t1 represents the period from the start of emission until the light reaches a specific target distance range R and the reflected light returns to camera 100, and the time from t1 to t2 is the period during which the reflected light from the specific target distance range R is exposed.

[0085] For range gate control to function correctly, it is necessary to synchronize the timing of light emission start and exposure start according to a predetermined target distance range. In this embodiment, the camera control unit 105 synchronizes the timing by transmitting the same reference signal to the photoelectric conversion element 102 and the light emission control unit 202.

[0086] As shown in the timing chart of Figure 5, the period from the start of one emission to the start of the next emission constitutes a range gate operation cycle. During one range gate operation cycle, the light received by the photoelectric conversion element 102 is converted into an electric charge and held within the photoelectric conversion element 102.

[0087] In this state, the next range gate operation cycle is performed, and the light newly received by the photoelectric conversion element 102 is converted into an electric charge and added to the charge held within the photoelectric conversion element 102. The interval between one emission and the next emission is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 100.

[0088] As shown in Figure 5, within one frame time, the range gate operation cycle is performed a predetermined number of times. The charge that was last added and held within one frame time is sent to the memory in the photoelectric conversion element 102, and the charge that was subsequently added and held is reset.

[0089] In this embodiment, since the exposure (charge accumulation) period is synchronized with the light emission from the light emitter 200, it is possible to obtain a clear image for a targeted distance range even under adverse weather conditions such as fog.

[0090] Next, the method for calculating the image range, which is performed by the image processing unit 103 and the distance measuring unit 104 based on the image signal captured by range gate control in this embodiment, will be explained using Figures 6, 7, and 8.

[0091] Figure 6 shows an example of the relationship between the range range and the field of view during imaging in the range gate control according to Embodiment 1. The horizontal axis represents the horizontal distance from the camera 100, and the vertical axis represents the vertical height from the ground surface. Figures 7(A) and 7(B) show the image range obtained during imaging with the range gate control according to Embodiment 1.

[0092] The orientation and position of the camera 100 are such that the optical axis of the imaging optical system 101 is parallel to the horizontal direction and its vertical height from the ground surface is Hc. φc is the vertical field of view of the camera 100, and it has a field of view that is symmetrical with respect to the optical axis of the imaging optical system 101.

[0093] φ1 is the field of view that can be captured when imaging with distance D1 and range range R1 using range gate control, and the range of φ1 is given by the following equation 5. φ1=atan(Hc / D1)+atan(φc / 2)...(Formula 5)

[0094] As shown in Figure 7(A), the image captured at this time is obtained in a range corresponding to φ1, which is narrower vertically downwards than the vertical range of the image corresponding to φc. Note that in Figures 7(A) and (B), the upward direction represents the upward direction of the subject, i.e., the vertical height direction, and represents the downward direction on the light-receiving surface of the photoelectric conversion element 102. This is because the imaging optical system 101 forms an inverted image of the subject on the light-receiving surface of the photoelectric conversion element 102.

[0095] φ2 is the field of view that can be captured when imaging with distance D2 and range range R2 using range gate control, and the range of φ2 is given by the following equation 6. φ2=atan(Hc / D2)+atan(φc / 2)...(Formula 6)

[0096] As shown in Figure 7(B), the image captured at this time is obtained as an image of the range corresponding to φ2, which is narrower vertically at the bottom compared to the vertical range of the image corresponding to φc.

[0097] Since D2 > D1, equations 5 and 6 show that φ2 < φ1. Therefore, the vertical range of the captured image is narrower at distance D2 than at distance D1.

[0098] Therefore, in this embodiment, the calculation processing range is determined based on the distance of the subject closer to the imaging unit. That is, within a predetermined distance range, the larger the distance of the subject closer to the imaging unit, the smaller the calculation processing range becomes. Also, within a predetermined distance range, the larger the distance of the subject closer to the imaging unit, the smaller the calculation processing range becomes in the vertical direction of the image.

[0099] Furthermore, the greater the distance of the subject closer to the imaging unit within the predetermined distance range, the more the calculation processing range will be set to the upper part of the subject in the image. Here, the upper part of the subject in the image refers to the area above the subject in the image, as shown in Figure 7(B).

[0100] Figure 8 is a flowchart illustrating an example of the imaging method according to Embodiment 1. The following description focuses on the case where distance D1 and range R1 are captured, but the same processing flow applies when capturing distance D2 and range R2.

[0101] First, in step S11, the camera control unit 105 determines the vertical height Hc of the camera 100 from the ground surface. Here, Hc is the vertical height of the camera 100 from the ground surface when the camera 100 is mounted on a stationary mobile body 300 on the ground surface. This value is read from a predetermined value that has been recorded in the storage unit 106 beforehand.

[0102] Next, in step S12, the camera control unit 105 determines the emission time of the light emitter 200, the exposure (charge accumulation) time of the camera 100, and the exposure (charge accumulation) timing for capturing distance D1 and range range R1 using range gate control.

[0103] Next, in step S13, the camera control unit 105 performs imaging using range gate control to acquire an image signal. That is, it operates the light emitter 200 and the photoelectric converter 102 to perform imaging using range gate control at the light emission time, exposure time, and exposure timing determined in step S12. As a result, an image signal with distance D1 and range R1 is acquired by the photoelectric converter 102 and input to the image processing unit 103.

[0104] Next, in step S14, the camera control unit 105 determines the calculation processing range of the image signal to be processed in a later stage. That is, the camera control unit 105 determines the calculation processing range of the acquired image signal that will be processed by the image processing unit 103 and the distance measuring unit 104, which are the calculation units in the later stage. In this way, the camera control unit 105, as a control means, determines the calculation processing range in step S14 (control step) based on a predetermined distance range.

[0105] Here, the range range R1 of distance D1, the range φ1 that can be photographed using range gate control, is calculated using equation 5 described above, and the image range corresponding to φ1 is determined as the calculation processing range. The calculation processing range determined in this way is input from the camera control unit 105 to the image processing unit 103 and the distance measuring unit 104.

[0106] Next, in step S15, the image processing unit 103 and the distance measuring unit 104 perform predetermined calculations (image processing, image recognition, distance measurement, etc.) on the calculation range determined in step S14. Here, step S15 functions as a calculation processing step that performs predetermined calculations on the image obtained from the image signal.

[0107] In this embodiment, the calculation processing range for image signal processing in the subsequent calculation unit is changed according to the distance D of range gate imaging, thereby reducing the amount of calculation processing required for image signal processing.

[0108] <Embodiment 2> Embodiment 2 of the present invention will now be described. In this embodiment, in calculating the range of the image processing area for image signal processing, the height of the subject is also taken into consideration in addition to the contents described in Embodiment 1. The same parts as in Embodiment 1 will not be explained.

[0109] In Embodiment 2, a method for calculating the calculation range, which is performed by the subsequent calculation units, the image processing unit 103 and the distance measuring unit 104, based on the image signal captured by range gate control, will be explained with reference to Figures 9, 10, and 11.

[0110] Figure 9 shows an example of the relationship between the range range and the field of view during imaging in the range gate control according to Embodiment 2, where the horizontal axis represents the horizontal distance from the camera 100 and the vertical axis represents the vertical height from the ground surface.

[0111] The orientation and position of camera 100 are such that the optical axis of the imaging optical system 101 is parallel to the horizontal direction and at a vertical height Hc from the ground surface. φc is the vertical field of view of camera 100, and the field of view is symmetrical with respect to the optical axis of the imaging optical system 101. Ho is the subject height range, which is the height of the subject that camera 100 will photograph and measure distance from.

[0112] φ1' is the field of view range that can be captured when imaging with distance D1 and range range R1 using range gate control, and the range of φ1' is expressed, for example, by the following equation 7. φ1´=atan(Hc / D1)+atan((Ho-Hc) / D1) (Formula 7)

[0113] Figures 10(A) and (B) show examples of image ranges obtained during imaging using range gate control according to Embodiment 2. In the above case, the image captured will be a range corresponding to φ1', which is narrower at the top and bottom of the vertical direction compared to the vertical range of the image corresponding to φc, as shown in Figure 10(A).

[0114] φ2' is the field of view range that can be captured when imaging with distance D2 and range range R2 using range gate control, and the range of φ2' is expressed, for example, by the following equation 8. φ2′=atan(Hc / D2)+atan((Ho-Hc) / D2) (Formula 8)

[0115] The image captured at this time, as shown in Figure 10(B), will be in the range corresponding to φ2', which is narrower at the top and bottom of the vertical direction compared to the vertical range of the image corresponding to φc.

[0116] Since D2 > D1, from equations 7 and 8, we get φ2' < φ1'. Therefore, the vertical range of the captured image is narrower at the farther distance D2 than at the farther distance D1.

[0117] Figure 11 is a flowchart illustrating an example of the imaging method according to Embodiment 2. The following description will focus on the case where distance D1 and range R1 are captured, but the same processing flow applies when capturing distance D2 and range R2.

[0118] First, in step S21, the camera control unit 105 determines the vertical height Hc of the camera 100 from the ground surface. Here, Hc is the vertical height of the camera 100 from the ground surface when the camera 100 is mounted on a stationary mobile body 300 on the ground surface. This value is read from a predetermined value that has been recorded in the storage unit 106 beforehand.

[0119] Furthermore, in step S21, the camera control unit 105 obtains the height Ho of the subject. This can be done by estimating the subject's height Ho based on the image recognition of the previous field, or, if the subject's height Ho is already known, by reading it from memory, for example.

[0120] Next, in step S22, the camera control unit 105 determines the emission time of the light emitter 200, the exposure (charge accumulation) time of the camera 100, and the exposure (charge accumulation) timing for capturing distance D1 and range range R1 using range gate control.

[0121] Next, in step S23, the camera control unit 105 operates the light emitter 200 and the photoelectric converter 102 to perform imaging using range gate control at the light emission time, exposure time, and exposure timing determined in step S22. As a result, the image signal is acquired by the photoelectric converter 102 and input to the image processing unit 103.

[0122] Next, in step S24, the camera control unit 105 determines the range of the image signal that will be processed in subsequent stages. Specifically, the camera control unit 105 determines the range of the acquired image signal that will be processed by the image processing unit 103 and the distance measuring unit 104, which are the subsequent processing units.

[0123] Here, the range φ1' that can be captured using range gate control is calculated using equation 7 above, with distance D1 and range range R1. The image range corresponding to φ1' is defined as the calculation processing range. The determined calculation processing range is input from the camera control unit 105 to the image processing unit 103 and the distance measuring unit 104.

[0124] Next, in step S25, the image processing unit 103 and the distance measuring unit 104 perform calculation processing on the image signal for the calculation processing range determined in step S24.

[0125] In this embodiment 2, the amount of computational processing associated with image signal processing can be reduced by changing the computational processing range in the subsequent calculation unit according to the distance D of range gate imaging and the subject height acquired in advance.

[0126] Furthermore, for example, the orientation of camera 100 may not be parallel to the horizontal direction but may have an angle, and the vertical height Hc may be measured in real time when the mobile body 300 is moving, rather than when it is stationary.

[0127] In other words, the attitude changes of the mobile body 300 may be measured in real time using an attitude sensor or the like, and the attitude and height of the camera 100 may be corrected based on that. Also, the vertical field of view of the camera 100 may be asymmetrical with respect to the optical axis.

[0128] Furthermore, the range captured by range gate control may be multiple distances or multiple ranges. Even in that case, the same approach as described above should be used to calculate the shooting angle of view, taking into account the attitude of camera 100, the vertical angle of view, the imaging range by range gate control, etc.

[0129] Furthermore, the calculation range for calculations performed in the subsequent calculation unit is not limited to the determination method described above. For example, it may be selected from a plurality of predetermined calculation ranges according to the imaging distance of range gate control, etc.

[0130] In the above embodiment, the size of a portion of the calculation processing range within the image for one frame, which is generated from all pixels read from the photoelectric conversion element 102, is determined based on a predetermined distance range.

[0131] However, it is also possible to decide to change the readout range of the signals read from the photoelectric conversion element 102 in order to perform calculations based on a predetermined distance range. That is, if the calculation range is small, the readout range of the pixel signals may be reduced by reading out only the signals of a pixel range corresponding to the size of a portion of the calculation range, instead of reading out the signals of all pixels from the photoelectric conversion element 102.

[0132] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0133] Furthermore, the present invention includes, for example, a system that implements the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0134] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the distance measuring camera system, etc., via a network or various storage media.

[0135] The computer (or CPU, MPU, etc.) in the distance measuring camera system, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0136] (Configuration 1) An imaging device comprising: a light-emitting unit that emits pulsed light; an imaging unit that generates an image signal by exposing the reflected light from a subject when the pulsed light strikes the subject; a control unit that controls the light emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit; and an arithmetic processing unit that performs predetermined arithmetic processing on an image obtained from the image signal, wherein the control unit determines the arithmetic processing range for performing the arithmetic processing based on the predetermined distance range.

[0137] (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the control unit determines the calculation processing range based on the distance of the subject on the side closer to the imaging unit within the predetermined distance range.

[0138] (Configuration 3) The imaging apparatus according to Configuration 1 or 2, wherein the control unit makes the calculation processing range smaller as the distance of the subject closer to the imaging unit within the predetermined distance range increases.

[0139] (Configuration 4) The imaging apparatus according to Configuration 3, wherein the control unit makes the calculation processing range smaller in the vertical direction of the image as the distance of the subject closer to the imaging unit within the predetermined distance range increases.

[0140] (Configuration 5) The imaging apparatus according to any one of Configurations 1 to 4, wherein the control unit sets the calculation processing range to a range that is closer to the subject in the image as the distance of the subject on the side closer to the imaging unit within the predetermined distance range increases.

[0141] (Method) An imaging method for controlling a light-emitting unit that emits pulsed light and an imaging unit that exposes the reflected light from a subject that has been reflected by the pulsed light and generates an image signal, comprising: a control step of controlling the light emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the reflected light from a subject within a predetermined distance range from the imaging unit; and a calculation processing step of performing a predetermined calculation on an image obtained from the image signal, wherein the control step determines a calculation processing range for performing the calculation based on the predetermined distance range.

[0142] (Program) A computer program for controlling each part of the imaging device described in any one of configurations 1 to 5 by computer. [Explanation of Symbols]

[0143] 100: Camera 101: Imaging Optical System 102: Photoelectric conversion element 103: Image Processing Unit 104: Distance measurement section 105: Camera Control Unit 106: Storage section 107: Communications Department 200: Light emitter 201: Light-emitting part 202: Light emission control unit 203: Communications Department 300: Mobile 301: ECU 302: Vehicle Control Unit 303: Display section 400: Pixel group 411: First photoelectric conversion unit 412: Second photoelectric conversion unit 413: Microlens 414: Light guide layer 415: Light receiving layer 500: Exit pupil 510: First pupil area 520: Second Eye Region 511: The First Light 521: The Second Light 610: Fog 620: Vehicle

Claims

1. A light-emitting unit that emits pulsed light, The imaging unit generates an image signal by exposing the reflected light that strikes the subject with the pulsed light and reflecting it, A control unit that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the imaging unit to reflected light from a subject within a predetermined distance range from the imaging unit, A calculation processing unit that performs predetermined calculations on an image obtained from the aforementioned image signal, Equipped with, The control unit determines the calculation processing range for performing the calculation based on the predetermined distance range. An imaging device characterized by the following.

2. The control unit determines the calculation processing range based on the distance of the subject closer to the imaging unit within the predetermined distance range. The imaging apparatus according to claim 1, characterized by the following:

3. The control unit makes the calculation processing range smaller as the distance of the subject closer to the imaging unit within the predetermined distance range increases. The imaging apparatus according to claim 1, characterized by the following:

4. The control unit makes the calculation processing range smaller in the vertical direction of the image as the distance of the subject closer to the imaging unit within the predetermined distance range increases. The imaging device according to claim 3, characterized by the following:

5. The control unit sets the calculation processing range to be a range closer to the upper part of the subject in the image, as the distance of the subject closer to the imaging unit within the predetermined distance range increases. The imaging apparatus according to claim 1, characterized by the following:

6. An imaging method for controlling a light-emitting unit that emits pulsed light and an imaging unit that exposes the reflected light, which is reflected when the pulsed light strikes a subject, and generates an image signal, A control step that controls the emission timing of the light-emitting unit and the exposure timing of the imaging unit so that the imaging unit exposes the imaging unit to reflected light from a subject within a predetermined distance range from the imaging unit. A calculation processing step which involves performing a predetermined calculation on an image obtained from the aforementioned image signal, Equipped with, The control step involves determining the calculation processing range for performing the calculation based on the predetermined distance range. An imaging method characterized by the following.

7. A computer program for controlling each part of the imaging apparatus described in any one of claims 1 to 5 by computer.

Images