Image synthesis apparatus, image synthesis method, and computer program

Abstract

To provide an image synthesis device capable of generating a clear composite image using images within multiple target distance ranges. [Solution] The image synthesis apparatus is characterized by comprising: a photoelectric conversion unit that acquires images corresponding to multiple target distance ranges; a distance value acquisition unit that generates distance values ​​for a subject for each pixel of the images corresponding to the target distance ranges; and an image synthesis unit that compares the distance values ​​with the target distance ranges and performs image synthesis using the brightness values ​​of the pixels whose distance values ​​are included in the target distance ranges, and without using the brightness values ​​of the pixels whose distance values ​​are not included in the target distance ranges.

Description

Technical Field

[0001] The present invention relates to an image synthesizing apparatus, an image synthesizing method, a computer program, and the like.

Background Art

[0002] There is an imaging method using a camera called a range gate camera. This is an imaging method that takes an image of only the subject within a target distance range, which is the distance range in front of the camera that the user wants to image.

[0003] As a method, pulsed light is emitted in front of the camera at a predetermined period, and the image sensor inside the camera is exposed at a timing when the reflected light from the target distance range can be imaged. By limiting the imaging range to the target distance range, it is possible to clearly image only the subject within the target distance range (hereinafter, this technology is called range gate control).

[0004] With this range gate control, for example, even in bad weather, a subject within a predetermined distance range can be clearly imaged. However, in the range gate image, which is an image obtained by range gate control, even within the same range gate image, the area outside the target distance range is an unclear image.

[0005] Therefore, when acquiring a clear image from a short distance to a long distance in front of the camera, a plurality of range gate images in a plurality of target distance ranges are required. When acquiring as one image, these range gate images need to be synthesized, but to obtain a clear synthesized image, only the clear areas of each range gate image need to be synthesized.

[0006] As a technology for synthesizing images, Patent Document 1 discloses a technology for synthesizing images based on distance information to the subject.

Prior Art Documents

Patent Documents

[0007] [Patent Document 1] Japanese Patent Publication No. 2016-076785 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, the prior art disclosed in the aforementioned patent documents recognizes the subject based on distance information and corrects the amount of movement of the same subject in multiple consecutively acquired images. Therefore, it is difficult to select and combine clear image regions of different subjects from multiple range-gated images.

[0009] Therefore, one of the objectives of the present invention is to provide an image synthesis device capable of generating a clear composite image using images within multiple target distance ranges. [Means for solving the problem]

[0010] To achieve the above objective, the one-sided image synthesis apparatus of the present invention is A photoelectric conversion unit that acquires images corresponding to multiple target distance ranges, A distance value acquisition unit that generates a distance value of the subject for each pixel of the image corresponding to the target distance range, An image synthesis unit compares the distance value with the target distance range, uses the brightness values ​​of the pixels whose distance value is included in the target distance range, and performs image synthesis without using the brightness values ​​of the pixels whose distance value is not included in the target distance range. It is characterized by having the following features. [Effects of the Invention]

[0011] According to the present invention, an image synthesis device capable of generating a clear composite image using images within multiple target distance ranges can be provided. [Brief explanation of the drawing]

[0012] [Figure 1]This figure shows an example of the configuration of an image sensor according to an embodiment of the present invention. [Figure 2] This is a schematic diagram showing an example configuration of the sensor board 11. [Figure 3] This figure shows an example of the configuration of the circuit board 21. [Figure 4] This figure shows an example of an equivalent circuit of the photoelectric conversion unit 102 of the pixel unit 101 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. [Figure 5] This diagram schematically illustrates the relationship between the operation of the APD201 and its output signals. [Figure 6] (A) is a schematic diagram showing an example of the configuration of the photoelectric conversion unit 102, and (B) is a diagram showing an example of the AA cross-section of Figure 6(A). [Figure 7] (A) is a diagram showing the light incident on the photoelectric conversion unit 102, and (B) to (D) are diagrams showing the relationship between the amount of parallax and the amount of defocus. [Figure 8] This is a functional block diagram showing an example configuration of the IR emitter 500, camera 600, and mobile unit 700. [Figure 9] This diagram illustrates an example of the relationship between the propagation of synchrotron radiation from the IR emitter 500 and its reflected light, and the exposure timing of the camera 600. [Figure 10] This is a timing chart illustrating the control operations required to obtain a range gate luminance information image per frame time. [Figure 11] (A) is a figure showing an example of an image obtained by an imaging method that does not rely on range gate control, and (B) is a figure showing an example of a range gate luminance information image with a target distance range of short distance. (C) is a figure showing an example of a range gate luminance information image with a target distance range of medium distance, and (D) is a figure showing an example of a range gate luminance information image with a target distance range of long distance. (E) is a figure showing an example of an image obtained by combining the range gate luminance information images from Figures 11(B) to (D) in this embodiment. [Figure 12] This figure illustrates an example of an image synthesis method that acquires three range-gate luminance information images and generates a combined range-gate luminance information image. [Figure 13A]This is a diagram for explaining an example of a synthesis method in a case where pixels (u, v) for which distance information cannot be obtained are included. [Figure 13B] This is a diagram for explaining an example of a synthesis method in a case where distance information cannot be obtained using the luminance value PDn(u, v) in all range gate luminance information images at pixel (u, v). [Figure 14A] This is a flowchart for explaining a processing example of a method for generating a range gate luminance information composite image. [Figure 14B] This is a flowchart for explaining a subsequent processing example of FIG. 14A. [Figure 15] This is a diagram for explaining an example of a synthesis method in a case where there are a plurality of distance values HDn(u, v) that satisfy the obtained distance value HDn(u, v) ⊂ target distance range Dn in step S106.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are given the same reference numerals, and duplicate explanations are omitted or simplified.

[0014] FIG. 1 is a diagram showing a configuration example of an image pickup device according to an embodiment of the present invention. In the present embodiment, an example will be described in which the image pickup device unit 100 has a so-called stacked structure in which two substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected.

[0015] However, a so-called non-stacked structure in which the configurations included in the sensor substrate and the configurations included in the circuit substrate are arranged in a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0016] FIG. 2 is a schematic diagram showing a configuration example of the sensor substrate 11, and shows a top view of the sensor substrate 11. In the pixel region 12 of the sensor substrate 11, a plurality of pixel portions 101 are two-dimensionally arranged in a plurality of rows and columns.

[0017] The pixel unit 101 includes a photoelectric conversion unit 102 containing an avalanche photodiode (hereinafter referred to as APD), which converts incident light into electrical signals. The number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0018] Figure 3 shows an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 for processing the charge photoelectrically converted by each photoelectric conversion unit 102 in Figure 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0019] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies control pulses to multiple pixels arranged in the row direction, row by row. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0020] The signals output from the photoelectric conversion unit 102 of each pixel are processed by the respective signal processing circuit 103. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0021] Signals are output to the vertical signal line 113 from the signal processing circuit 103 of the pixels in the row selected by the vertical scanning circuit 110. The signals output to the vertical signal line 113 are output to the outside of the image sensor unit 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has multiple buffers built in that are connected to the vertical signal line 113.

[0022] As shown in Figures 2 and 3, multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12 in a plan view.

[0023] Specifically, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0024] Note that the arrangement of the vertical signal line 113, the read circuit 112, and the output circuit 114 is not limited to the example shown in Figure 3. For example, the vertical signal line 113 may be arranged extending in the row direction, and the read circuit 112 may be located at the end of the vertical signal line 113.

[0025] Furthermore, the signal processing circuit 103 does not necessarily need to be provided for each photoelectric conversion unit 102; a single signal processing circuit may be shared among multiple photoelectric conversion units 102, and sequential signal processing may be performed.

[0026] Figure 4 shows an example of an equivalent circuit of the photoelectric conversion unit 102 of the pixel unit 101 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102.

[0027] The APD201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other node of the APD201 can be connected via the quench element 202 to a power line to which a drive voltage VH (second voltage), which is higher than voltage VL, is supplied.

[0028] In Figure 4, one node of the APD201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0029] Furthermore, when a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the device operates when the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the device operates when the voltage difference between the anode and cathode is near or below the breakdown voltage.

[0030] An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0031] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. The quench element 202 is connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD 201.

[0032] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD201 and thereby suppressing avalanche multiplication (quench operation). In addition, the quench element 202 also works to restore the voltage supplied to the APD201 to the drive voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation).

[0033] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained when a photon is detected and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. Figure 4 shows an example in which one inverter is used as the waveform shaping unit 210, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

[0034] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210, holds the count value, and supplies the count value to the selection circuit 212. Furthermore, when the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0035] The selection circuit 212 receives a control pulse SEL from the vertical scanning circuit 110 in Figure 3 via the drive line 214 (not shown in Figure 3) in Figure 4, which switches the electrical connection between the counter circuit 211 and the vertical signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal.

[0036] Figure 5 schematically shows the relationship between the operation of APD201 and the output signal. The input side of the waveform shaping unit 210 is node A, and the output side is node B. Between time ta and time tb, a potential difference of VH-VL is applied to APD201.

[0037] When a photon is incident on APD201 at time ta, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0038] As the voltage drop increases further and the potential difference applied to APD201 decreases, the avalanche multiplication of APD201 stops, as shown at time tb, and the voltage level at nodeA no longer drops below a certain value.

[0039] Subsequently, between time tb and time tc, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time tc, nodeA settles back to its original potential level. At this time, any portion of the output waveform at nodeA that falls below a certain threshold is reshaped by the waveform shaping unit 210 and output as a pulse signal at nodeB.

[0040] In this embodiment, the camera 600 is capable of acquiring distance information for each pixel 101 using an image plane phase-detection method. The acquisition of distance information for each pixel 101 using the image plane phase-detection method will be described below.

[0041] In this embodiment, distance information for each pixel 101 is obtained using the image plane phase-difference method, but the method is not limited to the image plane phase-difference method. As long as distance information for each pixel can be obtained, distance measurement may be performed using, for example, multiple cameras (stereo cameras) that are spaced far apart by baseline length.

[0042] Figure 6(A) is a schematic diagram showing an example of the configuration of the photoelectric conversion unit 102, and Figure 6(B) is a diagram showing an example of the AA cross-section of Figure 6(A). As shown in Figure 6(A), in this embodiment, the photoelectric conversion unit 102 in each pixel unit 101 has two photoelectric conversion units: a first photoelectric conversion unit 102a and a second photoelectric conversion unit 102b.

[0043] Furthermore, the first photoelectric conversion unit 102a and the second photoelectric conversion unit 102b each have separate APD201. In addition, each APD201 has a separate signal processing circuit 103.

[0044] As shown in Figure 6(B), the pixel section 101 has a light guide section 1001 positioned on the upper surface (light-receiving side) of the photoelectric conversion section 102. The light guide section 1001 is a light guide member having a microlens section 1001a for efficiently guiding the light beam incident on the pixel section 101 to the photoelectric conversion section 102, and a filter (not shown) that allows light in a predetermined wavelength band to pass through.

[0045] The photoelectric conversion unit 102 converts the light incident via the light guide unit 1001 into an electrical signal. In this embodiment, the aforementioned filter is configured to transmit only IR (infrared) light.

[0046] Figure 7(A) shows the light incident on the photoelectric conversion unit 102. The microlens unit 1001a is positioned so that the exit pupil 300 of the imaging optical system 601 (described later) and the photoelectric conversion unit 102 are optically conjugate.

[0047] As a result, the light beam that passes through the first pupil region 300a, which is a partial pupil region contained within the exit pupil 300, is incident on the first photoelectric conversion unit 102a. Similarly, the light beam that passes through the second pupil region 300b, which is a partial pupil region, is incident on the second photoelectric conversion unit 102b.

[0048] The first photoelectric conversion unit 102a in each pixel unit 101 converts the received light beam into electricity and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 102a included in the image sensor unit 100. The first image signal shows the intensity distribution of the image formed on the image sensor unit 100 by the light beam that mainly passed through the first pupil region 300a.

[0049] The second photoelectric conversion unit 102b in each pixel unit 101 converts the received light beam into photoelectric signals and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 102b included in the image sensor unit 100. The second image signal shows the intensity distribution of the image formed on the image sensor unit 100 by the light beam that mainly passed through the second pupil region 300b.

[0050] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 7(B) to (D).

[0051] Figures 7(B) to 7(D) illustrate the relationship between parallax and defocus. In these figures, the first luminous beam 310a represents the luminous beam that has passed through the first pupil region 300a, and the second luminous beam 310b represents the luminous beam that has passed through the second pupil region 300b.

[0052] Figure 7(B) shows the state when the image is in focus, with the first luminous beam 310a and the second luminous beam 310b converged in the photoelectric conversion unit 102. At this time, the parallax between the first image signal formed by the first luminous beam 310a and the second image signal formed by the second luminous beam 310b is 0.

[0053] Figure 7(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light beam 310a and the second image signal formed by the second light beam 310b is not zero, but has a negative value.

[0054] Figure 7(D) shows the state where the image is defocused in the positive w-axis direction. At this time, the parallax between the first image signal formed by the first light beam 310a and the second image signal formed by the second light beam 310b is not zero, but has a positive value.

[0055] A comparison of Figure 7(C) and Figure 7(D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax is proportional to the amount of defocus.

[0056] Therefore, the amount of disparity between the first image signal and the second image signal can be detected by a region-based matching method, and the amount of disparity can be converted into a defocus amount via a predetermined conversion coefficient.

[0057] Specifically, a first luminance image corresponding to the first image signal is generated, a point of interest is set within the generated first luminance image, and a matching region is set centered on the point of interest. Next, a second luminance image corresponding to the second image signal is generated, a reference point is set within the generated second luminance image, and a reference region is set centered on the reference point.

[0058] The correlation between the first luminance image and the second luminance image within the reference region is calculated while sequentially moving the reference point, and the reference point with the highest correlation is designated as the corresponding point. The amount of relative positional shift between the point of interest and the corresponding point is defined as the amount of disparity at the point of interest.

[0059] By calculating the amount of disparity while sequentially moving the point of focus, the amount of disparity at multiple pixel positions can be calculated. A value indicating the disparity is identified for each pixel, and data showing the disparity distribution is generated.

[0060] Furthermore, by using the imaging formula (Equation 1) of the imaging optical system 601, which will be described later, the amount of defocus on the image side can be converted into the distance to the subject. Here, a specific pixel section 101 in Figure 2 is denoted as pixel (u,v) using coordinates, the focal length of the imaging optical system 601 is f, and the distance from the principal point on the image side to the image sensor section 100 is Ipp.

[0061] In this case, the defocus amount ΔL(u,v), which is the amount of defocus at pixel (u,v), can be converted into a distance value H(u,v), which is the distance information to the subject at pixel (u,v), using the imaging formula in Equation 1 below.

number

[0062] In the explanation so far, the focal length f and the distance Ipp from the principal point on the image side to the image sensor 100 have been assumed to be constant values ​​regardless of the angle of view, but this is not the only option. If the imaging magnification of the imaging optical system 601 changes significantly with each angle of view, at least one of the focal length f or the distance Ipp from the principal point on the image side to the image sensor 100 may be a value that changes with each angle of view.

[0063] As described above, in this embodiment, it is possible to calculate the distance value H(u,v), which is distance information to the subject, for each pixel (u,v).

[0064] Note that while the image-plane phase-difference method described here has two photoelectric conversion units, it is not limited to this; any method with at least two photoelectric conversion units will suffice. For example, the image-plane phase-difference method can be implemented using a quad-pixel structure.

[0065] Next, the system comprising the IR emitter 500, camera 600, and mobile unit 700 in this embodiment will be described. In this embodiment, the camera 600 and other components function as an image synthesis device.

[0066] Figure 8 is a functional block diagram showing an example configuration of the IR emitter 500, camera 600, and mobile body 700 according to this embodiment. The imaging device (camera 600, IR emitter 500) of this embodiment is mounted on the mobile body 700, and the camera unit, which consists of an imaging optical system 601 and an image sensor unit 100, is arranged to capture images in at least one direction, for example, the front, rear, or side of the mobile body.

[0067] Furthermore, multiple camera units may be placed on the mobile body 700. In addition, the same number of IR emitters 500 may be placed on the mobile body 700 as the number of camera units, so that each camera unit can illuminate a subject in the direction of its optical axis with IR light.

[0068] In this embodiment, the mobile body 700 is described using the example of an automobile, but the mobile body can be any mobile object such as an aircraft, train, ship, drone, AGV, or robot.

[0069] Furthermore, some of the functional blocks shown in Figure 8 are realized by having the computers (not shown) included in the IR emitter 500, camera 600, and mobile unit 700, respectively, execute computer programs stored in memory (not shown) which is a storage medium.

[0070] However, some or all of these can be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0071] Furthermore, the functional blocks shown in Figure 8 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0072] The camera 600 includes the image sensor unit 100, imaging optical system 601, image processing unit 603, camera control unit 605, storage unit 606, communication unit 607, etc., as described in Figures 1 to 7. The image processing unit 603 includes a brightness information image generation unit 650, a distance information generation unit 651, and a brightness information image synthesis unit 652.

[0073] The imaging optical system 601 can form an image (optical image) of the subject on the image sensor unit 100 and has an exit pupil at a predetermined distance from the image sensor unit 100. The image sensor unit 100 is composed of avalanche photodiodes, as described in Figures 1 to 5, for photoelectric conversion of the optical image. The image sensor unit 100 outputs a first image signal and a second image signal to the luminance information image generation unit 650 and the distance information generation unit 651 in the image processing unit 603.

[0074] The luminance information image generation unit 650 sums the first image signal and the second image signal for each pixel (u,v) output from the image sensor unit 100 to calculate the luminance value P(u,v), which is the luminance information for the pixel (u,v).

[0075] Then, based on the luminance values ​​P(u,v) of all pixels (u,v) in the pixel region 12, a luminance information image, which is an image of the subject, is created. The luminance information image is sent to the luminance information image synthesis unit 652 and the ECU (Electric Control Unit) 701 of the mobile body 700.

[0076] Meanwhile, the distance information generation unit 651 calculates the distance value H(u,v), which is the distance information at pixel (u,v), based on the first image signal and the second image signal output from the image sensor unit 100, using the aforementioned equation 1. The distance information generation unit 651 functions as a distance value acquisition unit that generates the distance value of the subject for each pixel of the image corresponding to the target distance range.

[0077] Since the luminance value P(u,v) and distance value H(u,v) at a pixel (u,v) are generated from the same image signal, the luminance value P(u,v) and distance value H(u,v) are temporally synchronized. In other words, each pixel (u,v) in the generated luminance information image will simultaneously possess both a luminance value P(u,v) and a distance value H(u,v).

[0078] The distance information generated by the distance information generation unit 651 is sent to the luminance information image synthesis unit 652 and the ECU 701. The luminance information image synthesis unit 652 functions as an image synthesis unit, generating a luminance information synthesis image based on the input luminance information image and distance information, and sending it to the ECU 701. The generation of this luminance information synthesis image will be described later.

[0079] Furthermore, in the luminance information images and luminance information composite images generated by these image processing units 603, various image processing steps are performed on the image signal acquired by the image sensor unit 100 to generate the final luminance information images and luminance information composite images. The above image processing steps include, for example, at least one of the following: black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression.

[0080] The camera control unit 605 has a built-in CPU and memory that stores computer programs, and the CPU controls various parts of the camera 600 by executing the computer programs stored in the memory.

[0081] Furthermore, the camera control unit 605 functions as a control means, for example, by outputting a reference signal to the image sensor unit 100, and controlling the exposure period and exposure timing of the image sensor unit 100 based on that reference signal. These controls are based on the target distance range of the range gate image captured by the range gate control described later.

[0082] In the following description, exposure refers to the operation in the photoelectric conversion unit 102 from the start to the end of charge accumulation, photoelectric conversion, or imaging. In this embodiment, charge accumulation, photoelectric conversion, and imaging are used interchangeably, and the operations of charge accumulation, photoelectric conversion, and imaging include, for example, the operation of counting the signal photoelectrically converted by the APD201 in the counter 211 circuit.

[0083] Furthermore, the camera control unit 605 also transmits the same reference signal as described above to the IR emitter 500 via the communication unit 607. In this way, the same reference signal transmitted to the image sensor unit 100 is also transmitted to the IR emitter 500. Then, by having the IR emitter 500 control its light emission based on this reference signal, the exposure timing inside the image sensor unit 100 and the light emission timing of the IR emitter 500 can be synchronized.

[0084] Furthermore, the camera control unit 605 functions as a control unit that controls the light emission timing of the light emission unit and the exposure timing of the photoelectric conversion unit 102 so that the photoelectric conversion unit 102 exposes the subject reflected light within the target distance range.

[0085] The storage unit 606 includes, for example, a recording medium such as a memory card or a hard disk, and can store and read image signals.

[0086] The communication unit 607 is equipped with wireless and wired interfaces and outputs the generated image signal to the outside of the camera 600 and receives various signals from the outside. In this embodiment, the communication unit 607 is also connected to the communication unit 503 of the IR emitter 500 and has the role of transmitting the aforementioned reference signal and control commands from the camera control unit 605 to the IR emitter 500.

[0087] The IR emitter 500 includes an IR light-emitting unit 501, a light-emitting control unit 502, and a communication unit 503. The IR light-emitting unit 501 is positioned in front of the mobile body 700, for example, together with the camera unit, and consists of a lens and a light-emitting unit using a near-infrared LED. The light-emitting unit outputs pulsed light for a predetermined emission time in response to a pulse signal output from the light-emitting control unit 502.

[0088] The light emission control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal at a predetermined timing using the reference signal as a reference, and outputs it to the IR light emission unit 501.

[0089] Here, the light emission control unit 502 can set the period until a pulse is output, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next, based on the reference signal.

[0090] The camera control unit 605 sets a predetermined value to the light emission control unit 502 via the communication unit 607 and the communication unit 503, so that a pulse signal is output to the IR light emission unit 501 at a predetermined timing based on the reference signal, and the light emission period of the IR light emitter 500 is controlled. In this way, the light emission control unit 502 controls the light emission based on the same signal as the reference signal input to the image sensor unit 100.

[0091] The communication unit 503 communicates with the communication unit 607 of the camera 600, receives setting information and reference signals from the camera control unit 605 to the light emission control unit 502, and transmits them to the light emission control unit 502.

[0092] The ECU701 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control various parts of the mobile unit 700.

[0093] The output of the ECU 701 is supplied to the vehicle control unit 702 and the display unit 703. The vehicle control unit 702 functions as a motion control means that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 701. The display unit 703 functions as a display means and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700.

[0094] Based on the output of the ECU 701, the display unit 703 displays various information to the driver of the mobile unit 700, such as images acquired by the image sensor unit 100, recognition results from the image recognition unit 604, and the vehicle's driving status, using a GUI, for example.

[0095] Furthermore, the image processing unit 603 and other components shown in Figure 8 do not necessarily have to be mounted on the mobile unit 700. For example, they may be installed on an external terminal separate from the mobile unit 700, for remotely controlling the mobile unit 700 or for monitoring the movement of the mobile unit.

[0096] Figure 9 shows an example of the relationship between the propagation of the irradiated light from the IR emitter 500 and its reflected light, and the exposure timing of the camera 600 in this embodiment. Based on Figure 9, a method for acquiring a range gate luminance information image, which is an image of the target distance range, by performing range gate control (control that synchronizes the light emission timing and exposure timing according to the aforementioned target distance range, will be explained.

[0097] A camera that acquires range-gate luminance information images using range-gate control in this manner is called a range-gate camera. Figure 9 shows distance on the horizontal axis and time on the vertical axis.

[0098] First, let's explain the horizontal axis. The position of the moving object 700 is set to 0, fog 810 exists between distance x1 and distance x2, and vehicle 820 is located at distance x3. In Figure 9, range gate control is used, starting from position D, and range gate brightness information images are acquired within the range width R. In this case, the range width R is the target distance range to be imaged. At this time, vehicle 820 is located within the range width R.

[0099] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission in the IR emitter 500, and time tf is defined as the end time of light emission. In this case, the light emission period is tf. Also, when acquiring range gate luminance information images within the range width R from a starting point at distance D, the exposure start time in the photoelectric conversion unit 102 is defined as time t1, and the exposure end time is defined as time t2.

[0100] Time t1 is the moment when the synchrotron radiation emitted from IR emitter 500 at time 0 returns to camera 600 as reflected light from a distance D. Time t2 is the moment when the synchrotron radiation emitted from IR emitter 500 at time tf returns to camera 600 as reflected light from a point that has traveled a range width R from distance D.

[0101] Furthermore, time t3 is defined as the timing when the first reflected light from fog 810 returns to camera 600, and time t4 is defined as the timing when the last reflected light from fog 810 returns to camera 600.

[0102] In range gate control, exposure is not performed in the photoelectric conversion unit 102 during the period from time t3 to time t4 when the reflected light from the fog 810 reaches the camera 600. Then, exposure is performed in the photoelectric conversion unit 102 only during the period from time t1 to time t2 when the reflected light of range width R from distance D arrives, thereby removing the fog 810 while clearly acquiring an image of the vehicle 820.

[0103] Here, we will explain the time it takes for reflected light from an object at distance x to return to the camera 600. Let time tr be the timing when the synchrotron radiation that started to be emitted at time 0 strikes an object at distance x and returns to the photoelectric conversion unit 102 as reflected light. At this time, the relationship between the timing time tr of the reflected light returning and the distance x to the object to be imaged is given by equation 2 below. Time tr=2x / speed of light c (approximately 3×10^8m / s) (Formula 2)

[0104] As shown in Figure 9, when the imaging range is defined as the range width R from the distance D, the exposure timing time t1 at the start of the range can be calculated using the following equation 3 by substituting the distance D into the distance x in equation 2 above. Time t1=2D / speed of light c...(Equation 3)

[0105] Furthermore, the exposure timing time t2 at the end of the range width R can be obtained by substituting the distance D + range width R into the distance x in equation 2 above and adding the time tf, as shown in equation 4 below. Time t2=tf+2(D+R) / speed of light c...(Equation 4)

[0106] By controlling the time tr from light emission to exposure according to the desired imaging distance x (target distance range), range gate control can be achieved, enabling clear imaging of subjects within the target distance range even if there is fog or other obstructions between the camera and the target distance range.

[0107] Figure 10 is a timing chart illustrating the control operation for obtaining a range gate luminance information image per frame time. In this embodiment, the range gate luminance information image is generated by performing an exposure operation in the photoelectric conversion unit 102 in synchronization with the emission from the IR emitter 500, and is acquired based on this IR image. In this embodiment, the photoelectric conversion unit 102 acquires images corresponding to multiple target distance ranges.

[0108] In Figure 10, the "vertical synchronization signal" indicates the frame period of the image, where the period between one low pulse and the next low pulse is one frame time.

[0109] "IR light emission control" indicates the timing of light emission in the IR emitter 500, and "exposure control" indicates the start and end timings of exposure in the camera 600. "Counter value" indicates the increase or decrease in the photon count of the counter circuit 211 for each pixel.

[0110] The "RES signal" is a control pulse supplied to the counter circuit 211 via the drive line 213, and the pulse resets the count value that was being held.

[0111] First, we will explain the range gate control for obtaining a range gate luminance information image. In this control, the emission period of IR light is controlled by the light emission control unit 502 to emit light in a pulsed manner, and the photon counting operation (exposure operation) is performed only on the reflected IR light from a specific range.

[0112] As mentioned above, the emission period from the start to the end of emission is denoted as tf, the time from the start of emission to the start of photon counting as t1, and the time from the start of emission to the end of photon counting as t2. Furthermore, t1 represents the period from the start of emission until the light reaches a specific range and the reflected light returns to camera 600.

[0113] The time from t1 to t2 is the period during which the number of photons of reflected light in a specific range is counted, and this period represents the exposure time from the start to the end of exposure. The counter value increases according to the number of photons during the exposure time.

[0114] For range gate control to function correctly, it is necessary to synchronize the timing of the start and end of light emission and the start and end of exposure according to a predetermined target distance range. Therefore, in this embodiment, the camera control unit 605 synchronizes the exposure control unit and the light emission control unit 502 by transmitting the same reference signal.

[0115] As shown in the IR light emission control on the timing chart, the period from the start of one emission to the start of the next emission constitutes the range gate operation cycle. The counter value counted in one range gate operation cycle is retained, and the counter value is incremented in the next range gate operation cycle.

[0116] Furthermore, the time between flashes is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 600.

[0117] As shown in Figure 10, the range gate operation cycle is repeated a set number of times within one frame time. Then, the information of the last incremented counter value within one frame time is sent from the counter circuit 211 to the selection circuit 212, and the counter value is then reset by the RES signal.

[0118] In this type of range gate control, the exposure period is synchronized with the emission from the IR emitter 500, making it possible to obtain a clear IR image for the target range even in adverse weather conditions such as fog.

[0119] As mentioned above, the range gate luminance information image obtained based on range gate control is controlled to expose reflected light from the target distance range, and therefore, due to this control, images of the forward and backward regions of the target distance range cannot be obtained clearly.

[0120] Figure 11(A) shows an example of an image obtained by an imaging method that does not rely on range gate control. In adverse weather conditions such as fog, the camera 600 captures a pedestrian 2001 at close range, a vehicle 2002 at medium range, and a building 2003 at far range, but the image is blurry due to the fog.

[0121] Figure 11(B) shows an example of a range gate luminance information image with the target distance range set to the short-range, and Figure 11(C) shows an example of a range gate luminance information image with the target distance range set to the medium-range. Furthermore, Figure 11(D) shows an example of a range gate luminance information image with the target distance range set to the long-range.

[0122] As shown in Figure 11(B), in the image with the short-range as the target distance, the pedestrian 2001 at close range is clear, but it is blurry in the medium-range and far-range areas. As shown in Figure 11(C), in the image with the medium-range as the target distance, the vehicle 2002 at medium range is clear, but it is blurry in the short-range and far-range areas.

[0123] As shown in Figure 11(D), in the image where the target distance range is set to the long-distance range, building 2003, which is located at a distance, is clearly visible, but it is unclear in the short-distance and medium-distance areas.

[0124] In range gate control, reflected light from the forward and backward regions of the target distance range is not exposed, so brightness values ​​cannot be obtained in those regions. In addition, unintended reflected light from ambient light and synchrotron radiation from the IR emitter 500, as well as diffuse reflection from fog, etc., result in an unclear image in the region outside the target distance range.

[0125] Therefore, in this embodiment, in order to obtain a single clear image from near to far distances in adverse weather conditions as shown in Figure 11(A), multiple range gate luminance information images are synthesized with a target distance range from near to far.

[0126] However, as mentioned above, range gate luminance information images are blurry outside the target distance range within the image. Therefore, if range gate luminance information images from the near-range to the far-range range are simply added together, the overall image will be blurry.

[0127] Figure 11(E) shows an example of an image obtained by combining the range gate luminance information images from Figures 11(B) to (D) in this embodiment. As shown in Figure 11(E), the combined image is clear across the entire image region.

[0128] Thus, in the synthesis method of this embodiment, a clear range gate luminance information synthesis image can be generated by generating a synthesis image based on the clear image regions of each target distance range in multiple range gate luminance information images.

[0129] The following describes the method for synthesizing range gate luminance information images by the luminance information image synthesis unit 652 in this embodiment.

[0130] Figure 12 illustrates an example of an image synthesis method that acquires three range-gate luminance information images and generates a combined range-gate luminance information image.

[0131] In Figure 12, the luminance information image generation unit 650 acquires three range gate luminance information images based on range gate control for three different target distance ranges: target distance range D1, target distance range D2, and target distance range D3.

[0132] Then, based on the three range-gate luminance information images described above, range-gate luminance information image PD1, range-gate luminance information image PD2, and range-gate luminance information image PD3 are generated, and a combined range-gate luminance information composite image PDt is generated, which is a composite image of these three images.

[0133] In Figure 12, for example, the target distance range D1 is set to 0-100 [m], the target distance range D2 to 100-200 [m], and the target distance range D3 to 200-300 [m].

[0134] Here, the luminance values ​​P(u,v) of pixels (u,v) in range-gate luminance information images PD1, PD2, and PD3 are denoted as luminance value PD1(u,v), luminance value PD2(u,v), and luminance value PD3(u,v), respectively.

[0135] The respective luminance values ​​PD1(u,v), PD2(u,v), and PD3(u,v) are luminance information output by the luminance information image generation unit 650. These luminance values ​​PD1(u,v), PD2(u,v), and PD3(u,v) correspond to distance values ​​HD1(u,v), HD2(u,v), and HD3(u,v), respectively.

[0136] Distance values ​​HD1(u,v), HD2(u,v), and HD3(u,v) are distance information output by the distance information generation unit 651. In other words, for pixels (u,v) of the range gate luminance information image PD1 obtained within the target distance range D1, the luminance value PD1(u,v) corresponds to the distance value HD1(u,v).

[0137] Similarly, in the range gate luminance information image PD2 obtained in the target distance range D2, the luminance value PD2(u,v) corresponds to the distance value HD2(u,v) at each pixel (u,v). Similarly, in the range gate luminance information image PD3 obtained in the target distance range D3, the luminance value PD3(u,v) corresponds to the distance value HD3(u,v) at each pixel (u,v).

[0138] In Figure 12, the distance value HD1(u,v) obtained by the distance information generation unit 651 is 250[m], the distance value HD2(u,v) obtained similarly is 250[m], and the distance value HD3(u,v) obtained similarly is 250[m].

[0139] At this time, it is determined whether the distance value HDn(u,v), which is the distance information obtained for each target distance range Dn for each pixel (u,v), is included in the corresponding target distance range Dn.

[0140] This reveals that only the distance value HD3(u,v) (=250[m]) within the target distance range D3 (=200-300[m]) is included in the corresponding target distance range D3 (=200-300[m]).

[0141] Therefore, in this embodiment, the luminance information image synthesis unit 652 sets the value of the luminance value PD3(u,v) as the synthesized luminance value PDt(u,v) of the pixel (u,v) in the range gate luminance information synthesis image PDt.

[0142] Specifically, the luminance information image synthesis unit 652 compares each target distance range Dn with the distance value HDn(u,v) obtained within that target distance range Dn. Then, the distance value HDn(u,v) included within the range of the target distance range Dn and the luminance value PDn corresponding to the distance value HDn(u,v) are set as the synthesized luminance value PDt(u,v) for each coordinate in the range gate luminance information synthesis image PDt.

[0143] A range-gate luminance information composite image PDt is generated based on the set composite luminance value PDt(u,v). Thus, in this embodiment, when generating the range-gate luminance information composite image PDt, the luminance value P(u,v) of the pixel containing the distance value HDn(u,v) in the target distance range Dn is used as the set value.

[0144] In other words, the distance value is compared with the target distance range, and image synthesis is performed using the brightness values ​​of pixels whose distance value falls within the target distance range, while excluding the brightness values ​​of pixels whose distance value does not fall within the target distance range. This makes it possible to eliminate unintended reflected light from outside the target distance range.

[0145] Here, for a pixel (u,v) in the target distance range Dn, if the obtained distance information, the distance value HDn(u,v), is included in the target distance range Dn, we express this as distance value HDn(u,v) ⊂ target distance range Dn. Conversely, if it is not included, we express this as distance value HDn(u,v) NOT ⊂ target distance range Dn, and will explain this further below.

[0146] As mentioned above, the distance information generation unit 651 calculates the correlation between the reference point and the point of interest set in the first luminance image and the second luminance image, respectively, and further calculates the amount of disparity.

[0147] In range-gate luminance information images, exposure of reflected light outside the target distance range is restricted by range-gate control. As a result, there are areas outside the target distance range where luminance values ​​cannot be obtained, or areas where the luminance values ​​are difficult to calculate the correlation between. Therefore, calculating the amount of disparity becomes difficult, and pixels (u,v) from which distance information cannot be obtained also occur.

[0148] Similarly, there are subjects for which it is difficult to obtain clear reference points or points of interest, making it difficult to calculate correlation and even parallax (for example, a clear sky or roads of the same color), and in these cases, pixels (u,v) for which distance information cannot be obtained will occur.

[0149] The following describes a synthesis method for pixels (u,v) where such distance information cannot be obtained. Figures 13A and 13B show a method for generating a range-gate luminance information synthesis image PDt from range-gate luminance information images PD1, PD2, and PD3 obtained based on range-gate control in target distance ranges D1, D2, and D3.

[0150] Furthermore, the target distance range settings are the same as in Figure 12: target distance range D1 is 0-100 [m], target distance range D2 is 100-200 [m], and target distance range D3 is 200-300 [m]. Note that the symbols in Figure 13 are the same as in Figure 12, so their explanation is omitted.

[0151] Figure 13A illustrates an example of a synthesis method when there are pixels (u,v) for which distance information cannot be obtained. In Figure 13(A), the distance value HD1(u,v), which is the distance information obtained by range gate control, is 50 [m], and distance information cannot be obtained for distance values ​​HD2(u,v) and HD3(u,v).

[0152] In other words, the distance value HD1(u,v) is equal to the target distance range D1, and distance information is not available for distance values ​​HD2(u,v) and HD3(u,v). In such cases, in this embodiment, the composite luminance value PDt(u,v) in the range gate luminance information composite image PDt is set to the luminance value PD1(u,v).

[0153] In other words, the brightness value PDn(u,v) corresponding to pixels (u,v) for which distance information is not available is excluded, and the brightness value PDn(u,v) corresponding to distance value HDn(u,v) that satisfies the distance value HDn(u,v) ⊂ target distance range Dn is set as the value.

[0154] Therefore, in pixels (u,v), the brightness values ​​of areas where appropriate brightness values ​​and distance information cannot be obtained due to exposure limitations (charge storage time or photoelectric conversion time) by range gate control are excluded. Then, it becomes possible to set the brightness values ​​of the area set as the target distance range as the composite brightness value PDt(u,v).

[0155] Figure 13(B) illustrates an example of a synthesis method when distance information cannot be obtained for the luminance value PDn(u,v) in all range-gated luminance information images at pixel (u,v). In Figure 13(B), the only difference from Figure 13(A) is that distance information cannot be obtained for the distance value HD1(u,v); all other settings and configurations are the same as in Figure 13(A).

[0156] In Figure 13(B), distance information is not obtained for all distance values ​​HD1(u,v), HD2(u,v), and HD3(u,v). In such cases, in this embodiment, the composite luminance value PDt(u,v) in the range gate luminance information composite image PDt is set to the average value of luminance values ​​PD1(u,v), PD2(u,v), and PD3(u,v).

[0157] In other words, if the distance value HDn(u,v) for all range-gate luminance information images cannot be obtained for a pixel (u,v), the average value of the luminance values ​​PDn(u,v) of each range-gate luminance information image PDn is set in the range-gate luminance information composite image PDt.

[0158] Therefore, even for subjects where the aforementioned parallax amount is difficult to calculate, the obtained luminance value can be used as a set value in the composite luminance value PDt(u,v).

[0159] Furthermore, if the distance value HDn(u,v) for all range-gated luminance information images cannot be obtained for a pixel (u,v), instead of using the average value of the luminance values ​​PDn(u,v), the highest luminance value PDn(u,v) may be set as the composite luminance value PDt(u,v).

[0160] In other words, for pixels in multiple images captured within the target distance range from which no distance values ​​can be obtained, the highest brightness value among the brightness values ​​of the pixels in the multiple images may be used for image synthesis.

[0161] In each range-gate luminance information image, for example, the luminance value PD1(u,v) is the luminance value of a region where appropriate distance information cannot be obtained due to the limitations of exposure (charge storage time or photoelectric conversion time) imposed by range-gate control. Therefore, distance information and luminance values ​​cannot be obtained, and in the case of luminance value PD2(u,v), distance information of the subject cannot be obtained because it is difficult to calculate the amount of disparity, but the luminance value can be obtained.

[0162] In such cases, by using the obtained luminance value as a set value, it becomes possible to generate an appropriate range-gated luminance information composite image PDt.

[0163] Figure 14A is a flowchart illustrating an example of a method for generating a range-gate luminance information composite image in this embodiment, and Figure 14B is a flowchart illustrating a continuation of the processing example shown in Figure 14A. Note that the processing flows in Figures 14A to 14B are executed periodically, for example, in units of frame periods.

[0164] Furthermore, the CPU and other components within the camera control unit 605 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowcharts shown in Figures 14A and 14B.

[0165] In Figure 14A, when the generation of a range gate luminance information composite image is started, in step S101, the camera control unit 605 sets K target distance ranges Dn (where n is an integer from 1 to K) and sets n=1.

[0166] Next, in step S102, range gate control is performed in the target distance range Dn. That is, exposure (charge accumulation, photoelectric conversion, or imaging) is performed by range gate control, and the brightness value PDn(u,v) and distance value HDn(u,v) of each pixel (u,v) in the target distance range Dn are obtained. Note that in step S102, n = n + 1.

[0167] Here, step S102 functions as an imaging step that acquires images corresponding to multiple target distance ranges, and also as a distance value acquisition step that acquires the distance value of the subject for each pixel of the image corresponding to the target distance range.

[0168] In step S103, it is determined whether n=K or not. If it is determined to be No in step S103, the process returns to step S102 and the operation of step S102 is repeated a predetermined number of times (K times). If it is determined to be Yes in step S103, the process proceeds to step S104, where u=1 and v=1, and the process proceeds to step S105 in Figure 14B.

[0169] In step S105, the luminance information image synthesis unit 652 determines whether it was able to acquire at least one distance value HDn(u,v) from among the K distance values. If the result in step S105 is Yes, that is, if at least one distance value was acquired, the process proceeds to step S106; otherwise, the process proceeds to step S108.

[0170] In step S106, the luminance information image synthesis unit 652 compares the target distance range Dn with the obtained distance value HDn(u,v). Then, the distance value HDn(u,v) that satisfies the distance value HDn(u,v) ⊂ target distance range Dn, and the corresponding luminance value PDn(u,v) are set as the setting value for the synthesized luminance value PDt(u,v) in the luminance information synthesis image PD, and the process proceeds to step S107.

[0171] On the other hand, in step S108, the average value of K luminance values ​​PDn(u,v) is set as the composite luminance value PDt(u,v) in the luminance information composite image PD, and the process proceeds to step S107. That is, for pixels in multiple images captured within multiple target distance ranges for which no distance value can be obtained, the average value of the luminance values ​​of the pixels in the multiple images is used to perform image synthesis as described later.

[0172] In step S107, the variable u = u + 1, and the process proceeds to step S109. In step S109, it is determined whether u = e at pixel (u, v). If the result in step S109 is Yes, the process proceeds to step S110. If the result in step S109 is No, i.e., if u = e, the process returns to step S105.

[0173] In step S110, set u=1 and v=v+1, and proceed to step S111. In step S111, determine whether v=e or not. If it is determined to be Yes, proceed to step S112; otherwise, return to step S104.

[0174] In step S112, a luminance information composite image PD is generated based on the composite luminance value PDt(u,v) set for each coordinate in steps S105 to S111.

[0175] Here, steps S106 to S112 function as an image synthesis step, comparing the distance value with the target distance range and performing image synthesis using the brightness values ​​of pixels whose distance value falls within the target distance range. In addition, the image synthesis step performs image synthesis without using the brightness values ​​of pixels whose distance value does not fall within the target distance range.

[0176] Figure 15 illustrates an example of a synthesis method when, in step S106, there are multiple distance values ​​HDn(u,v) that satisfy the obtained distance value HDn(u,v) ⊂ target distance range Dn.

[0177] Figure 15, like Figure 12, shows a method for generating a combined range gate luminance information image PDt from range gate luminance information images PD1 to PD3 obtained based on range gate control in the target distance range D1 to D3.

[0178] In Figure 15, the settings for each target distance range are the same as in Figure 12: target distance range D1 is 0-100 [m], target distance range D2 is 100-200 [m], and target distance range D3 is 200-300 [m]. The symbols in Figure 15 are the same as in Figure 12, so their explanation is omitted.

[0179] In Figure 15, the distance value HD1(u,v), obtained by range gate control, is 99 [m], the similarly obtained distance value HD2(u,v) is 101 [m], and the similarly obtained distance value HD3(u,v) is 101 [m].

[0180] In other words, distance value HD1(u,v) ⊂ target distance range D1, distance value HD2(u,v) ⊂ target distance range D2, and distance value HD3(u,v) NOT ⊂ target distance range D3. This is the case, for example, when the subject is at the boundary between target distance range D1 and target distance range D2, and both target distance range D1 and target distance range D2 satisfy distance value HDn(u,v) ⊂ target distance range Dn.

[0181] In such cases, the influence of distance measurement errors on the distance value obtained by the phase-detection imaging method of the camera 600 is a possible cause. Therefore, in this embodiment, a luminance information composite image including distance measurement errors is generated.

[0182] In other words, in this embodiment, the range of distance values ​​that can be expected after considering the distance measurement error is defined as the error-considered distance measurement value range. For example, the distance measurement accuracy of the imaging plane phase difference method by the camera 600 in this embodiment is ±10%.

[0183] In this case, if the distance value is 100[m], the error-considered distance measurement range will be 90-110[m]. That is, the distance value is the distance value range including the distance measurement error. At this time, the error-considered distance measurement ranges of the distance values ​​HDn(u,v) obtained by range gate control are as follows: for distance value HD1(u,v) 99[m], the range is 89.1-108.9[m]. Similarly, for distance value HD2(u,v) 101[m], the range is 90.9-111.1[m], and for distance value HD3(u,v) 101[m], the range is also 90.9-111.1[m].

[0184] When comparing the distance value HDn(u,v) with the target distance range Dn using the error-considered distance measurement range, the distance value HD1(u,v) (error-considered distance measurement range) ⊂ target distance range D1. Also, the distance value HD2(u,v) (error-considered distance measurement range) ⊂ target distance range D2, and the distance value HD3(u,v) (error-considered distance measurement range) NOT ⊂ target distance range D3.

[0185] Therefore, the distance values ​​HDn(u,v) in target distance range D1 and target distance range D2 satisfy the condition HDn(u,v) ⊂ target distance range Dn.

[0186] In this embodiment, in such cases, the luminance value P(u,v) in the range gate luminance information composite image PDt is the average value of the luminance values ​​corresponding to the distance values ​​that satisfy the distance value HDn(u,v) ⊂ target distance range Dn.

[0187] In other words, the average values ​​of the luminance values ​​PD1(u,v) and PD2(u,v) corresponding to the distance values ​​HD1(u,v) and HD2(u,v) are used. That is, for pixels with multiple distance values ​​that fall within the target distance range, the average value of the luminance values ​​of the pixels whose distance values ​​fall within the target distance range is used for image synthesis.

[0188] In this embodiment, the obtained distance value HDn(u,v) is compared with the target distance range Dn within the error-considered distance range, including the distance measurement error. Furthermore, if there is an area where the error-considered distance range overlaps with the target distance range Dn, the distance value HDn(u,v) ⊂ target distance range Dn, and the corresponding luminance value P(u,v) is taken as the composite luminance value PDt(u,v) in the range-gate luminance information composite image PDt.

[0189] Furthermore, if there are multiple distance values ​​HDn(u,v) that satisfy the condition HDn(u,v) ⊂ Target distance range Dn, the average value of the distance value HDn(u,v) that satisfies the condition HDn(u,v) ⊂ Target distance range Dn and the corresponding luminance value P(u,v) is used.

[0190] In this embodiment, if there are multiple distance values ​​HDn(u,v) that satisfy the condition HDn(u,v) ⊂ target distance range Dn, including the distance measurement error, the average value of the corresponding luminance value P(u,v) and the satisfying distance value HDn(u,v) is used. However, it is not limited to the average value.

[0191] For example, the weighted addition and averaging can be performed by changing the proportion of the luminance value P(u,v) added according to the proportion of the error-considered distance measurement value range that overlaps with the target distance range Dn. In other words, for pixels that have multiple distance value ranges included in the target distance range, the luminance value may be added according to the proportion of the distance value ranges included in the target distance range.

[0192] Alternatively, the brightness value PDn(u,v) corresponding to the smallest distance value HDn(u,v) among the obtained distance values ​​HDn(u,v) may be used as the set value. That is, for pixels with multiple distance values ​​included in the target distance range, the brightness value of the pixel with the smallest distance value may be used for image synthesis. This is because the error-considered distance measurement range becomes smaller as the distance to the subject decreases.

[0193] Furthermore, in the above embodiment, the image sensor unit 100 is configured to convert only IR light into photoelectric data for imaging, but this is not the only configuration. For example, even when using an image sensor unit capable of imaging visible light and performing range gate control using visible light, the same type of synthesis as in this embodiment is possible.

[0194] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0195] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0196] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the distance measuring camera system, etc., via a network or various storage media.

[0197] The computer (or CPU, MPU, etc.) in the distance measuring camera system, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0198] (Configuration 1) An image synthesis apparatus comprising: a photoelectric conversion unit that acquires images corresponding to multiple target distance ranges; a distance value acquisition unit that generates distance values ​​for a subject for each pixel of the images corresponding to the target distance ranges; and an image synthesis unit that compares the distance values ​​with the target distance ranges and performs image synthesis using the brightness values ​​of the pixels in which the distance values ​​are included in the target distance ranges, and without using the brightness values ​​of the pixels in which the distance values ​​are not included in the target distance ranges.

[0199] (Configuration 2) The image synthesis apparatus according to Configuration 1, characterized in that, in a plurality of images captured within a plurality of target distance ranges, for pixels from which no distance value can be obtained, the average value of the brightness value of the pixels in the plurality of images is used to perform the image synthesis.

[0200] (Configuration 3) The image synthesis apparatus according to Configuration 1, characterized in that, in a plurality of images captured within a plurality of target distance ranges, for pixels in which no distance value can be obtained, the highest brightness value among the brightness values ​​of the pixels in the plurality of images is used to perform the image synthesis.

[0201] (Configuration 4) For pixels that have multiple distance values ​​included in the target distance range, the image synthesis is performed using the average value of the brightness values ​​of the pixels whose distance values ​​are included in the target distance range, as described in any one of Configurations 1 to 3.

[0202] (Configuration 5) The image synthesis apparatus according to any one of Configurations 1 to 4, characterized in that the distance value is a distance value range including the distance measurement error.

[0203] (Configuration 6) The image synthesis apparatus according to Configuration 5, characterized in that for pixels that have multiple distance value ranges included in the target distance range, the brightness value is added in proportion to the ratio of the distance value ranges included in the target distance range.

[0204] (Configuration 7) An image synthesis apparatus according to any one of Configurations 1 to 6, characterized in that, for pixels that have multiple distance values ​​included in the target distance range, the image synthesis is performed using the brightness value of the pixel with the smallest distance value.

[0205] (Configuration 8) An image synthesis apparatus according to any one of Configurations 1 to 7, comprising: a light-emitting unit that emits pulsed light; and a control unit that controls the light emission timing of the light-emitting unit and the exposure timing of the photoelectric conversion unit so that the photoelectric conversion unit exposes the reflected light from a subject within the target distance range.

[0206] (Method) An image synthesis method characterized by comprising: an imaging step of acquiring images corresponding to multiple target distance ranges; a distance value acquisition step of acquiring the distance value of a subject for each pixel of the image corresponding to the target distance range; and an image synthesis step of comparing the distance value with the target distance range, using the brightness value of the pixels in which the distance value is included in the target distance range, and not using the brightness value of the pixels in which the distance value is not included in the target distance range.

[0207] (Program) A computer program for controlling each part of the image synthesis apparatus described in any one of items 1 to 8 of the configuration. [Explanation of symbols]

[0208] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Image sensor section 101: Pixel section 102: Photoelectric conversion unit 102a: First photoelectric conversion unit 102b: Second photoelectric conversion unit 103: Signal Processing Circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output Circuit 115: Control pulse generation unit 201:APD 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Selection Circuit 213: Drive line 214: Drive line 300: Exit pupil 300a: First pupil region 300b: Second pupil area 310a: First luminous beam 310b: Second luminous beam 500: IR light emitter 501: IR light-emitting part 502: Light emission control unit 503: Communications Department 600: Camera 601: Imaging Optical System 603: Image Processing Unit 604: Image Recognition Unit 605: Camera Control Unit 606: Storage section 607: Communications Department 650: Brightness Information Image Generation Unit 651:Distance information generation unit 652: Brightness Information Image Synthesis Unit 700: Mobile 701: ECU 702: Vehicle Control Unit 703: Display section 810: Fog 820: Vehicle 830: Pedestrian 900: Visible light emitter 901: Visible light emitting part 1001: Light guide 1001a: Microlens section 1200: Split area 2001: Pedestrian 2002: Vehicles 2003: Building

Claims

1. A photoelectric conversion unit that acquires images corresponding to multiple target distance ranges, A distance value acquisition unit that generates a distance value of the subject for each pixel of the image corresponding to the target distance range, An image synthesis unit compares the distance value with the target distance range, uses the brightness values ​​of the pixels whose distance value is included in the target distance range, and performs image synthesis without using the brightness values ​​of the pixels whose distance value is not included in the target distance range. An image synthesis apparatus characterized by having the following features.

2. The image synthesis apparatus according to claim 1, characterized in that, in a plurality of images captured within a plurality of target distance ranges, for pixels from which no distance value can be obtained, the average value of the brightness value of the pixels in the plurality of images is used to perform the image synthesis.

3. The image synthesis apparatus according to claim 1, characterized in that, in a plurality of images captured within a plurality of target distance ranges, for pixels from which no distance value can be obtained, the highest brightness value among the brightness values ​​of the pixels in the plurality of images is used to perform the image synthesis.

4. The image synthesis apparatus according to claim 1, characterized in that, for pixels where there are multiple distance values ​​included in the target distance range, the image synthesis is performed using the average value of the brightness values ​​of the pixels whose distance values ​​are included in the target distance range.

5. The image synthesis apparatus according to claim 1, characterized in that the distance value is a distance value range including the distance measurement error.

6. The image synthesis apparatus according to claim 5, characterized in that, for pixels that have multiple distance value ranges included in the target distance range, the brightness value is added in proportion to the ratio of the distance value ranges included in the target distance range.

7. The image synthesis apparatus according to claim 1, characterized in that, for pixels that have multiple distance values ​​included in the target distance range, the image synthesis is performed using the brightness value of the pixel with the smallest distance value.

8. A light-emitting unit that emits pulsed light, The image synthesis apparatus according to claim 1, further comprising a control unit that controls the light emission timing of the light emission unit and the exposure timing of the photoelectric conversion unit so that the photoelectric conversion unit exposes the reflected light from a subject within the target distance range.

9. The imaging step involves acquiring images corresponding to multiple target distance ranges, A distance value acquisition step, which acquires the distance value of the subject for each pixel of the image corresponding to the target distance range, Image synthesis step: Compare the distance value with the target distance range, use the brightness values ​​of the pixels whose distance value is included in the target distance range, and perform image synthesis without using the brightness values ​​of the pixels whose distance value is not included in the target distance range. An image synthesis method characterized by having the following features.

10. A computer program for controlling each part of an image synthesis apparatus according to any one of claims 1 to 8 by computer.

Images