Imaging device, imaging method, program

Abstract

In stereo cameras using SPAD sensor cameras, when ambient light is low, obstacle detection may not be possible in the early stages due to the exposure time, which could lead to a slower detection cycle. [Solution] When it is determined that there is a distance measurement image in which no object can be detected, and that the current situation in which the vehicle is driving is one in which sudden movements or cutting in are likely to occur, the exposure start timing of one of the cameras constituting the stereo camera is shifted.

Description

Technical Field

[0001] The present invention relates to an imaging device mounted on a vehicle that obtains parallax from a plurality of images and performs object detection.

Background Art

[0002] In recent years, a function has been mounted to detect obstacles on a vehicle's driving road and automatically decelerate or stop the vehicle. This is so-called ADAS (Advanced Driver Assistance System).

[0003] One technique for detecting obstacles in this ADAS is a stereo camera. This stereo camera can perform triangulation using the parallax of two cameras and measure the distance to an obstacle. In this case, this method will be referred to as a stereo ranging method. It can also be said to be a camera that measures distance from the parallax of two captured images obtained from each of two imaging means. The stereo ranging method has the property that the accuracy for distant objects is higher as the distance between the two cameras (= baseline length) is wider, and the accuracy for close objects is higher as it is narrower. Therefore, in the prior art, in order to ensure ranging accuracy from close range to far range, techniques have been disclosed in which a plurality of sets of cameras with different baseline lengths are arranged or provided (Patent Document 1). In Patent Document 1, it is necessary to prepare two sets of stereo cameras (four cameras as cameras), which has a cost disadvantage.

[0004] One solution to this problem is a system that uses a distance measurement method that utilizes image plane phase difference (Patent Document 2). This method is a camera that measures distance using a sensor that can obtain two images from different viewpoints from the image sensor by placing two pixels (usually one pixel) under each microlens arranged on the image sensor. It can also be described as a camera that can measure distance using image plane phase difference in a pupil-splitting method, consisting of one imaging optical system and one image sensor. Alternatively, it can be described as a compound eye camera equipped with a lens array and a sensor having multiple lenses arranged on the same plane, and a camera that can measure distance from the parallax of multiple individual eye images captured by the compound eye camera. Since this method was originally adopted for autofocus (AF) in single-lens reflex cameras, we will refer to this method as the DAF distance measurement (Dualpixel AF) method in this case. Patent Document 2 introduces a system in which two DAF distance measurement cameras are prepared, stereo distance measurement is performed using images output from each of the two cameras for long-distance measurement, and the DAF distance measurement of one camera is used for short-distance measurement.

[0005] On the other hand, a high frame rate is one of the important performance characteristics of ADAS cameras. In order for an ADAS camera to safely slow down or stop a vehicle when an obstacle suddenly appears, such as something jumping out or cutting in, it is necessary to detect these as quickly as possible. To achieve this, the frame rate at which the camera captures images needs to be increased, and the detection cycle of jumps and cuts needs to be shortened. One technology that achieves this high frame rate is the SPAD sensor camera using a Single Photon Avalanche Diode (=SPAD) sensor (Patent Document 3). This SPAD sensor is a sensor that uses a photon counter and, in addition to being highly sensitive, also has a high frame rate readout function. Due to the nature of the photo counter, the brightness value of the pixel is stored digitally and can be read out any number of times. Therefore, it is possible to acquire image data at an early stage before the predetermined exposure time has been reached. In other words, it is possible to read out the brightness value multiple times at any timing of less than one frame. This makes it possible to achieve a higher frame rate than a normal CMOS sensor. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-53950 [Patent Document 2] Japanese Patent Publication No. 2011-203238 [Patent Document 3] Japanese Patent Publication No. 2022-106660 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, in the conventional example described above, it may not be possible to speed up the detection cycle for sudden appearances or interruptions. This occurs when ambient light levels decrease. Similar to CMOS sensors, SPAD sensor cameras experience a decrease in signal-to-noise ratio (SNR) and distance measurement accuracy as exposure time decreases. Therefore, when ambient light levels decrease, distance measurement images acquired early in the exposure process have a low SNR, resulting in reduced distance measurement accuracy and decreased obstacle detection performance. Consequently, in low ambient light conditions, obstacle detection may not be possible early in the exposure process, potentially preventing a speedy detection cycle.

[0008] Therefore, the present invention aims to provide a device that reduces the decrease in the detection cycle of sudden outbursts and interruptions, depending on the situation, even when ambient illuminance decreases. [Means for solving the problem]

[0009] To achieve the above objective, the first invention of the imaging device according to this application is characterized by comprising: a plurality of imaging means capable of reading luminance values ​​multiple times at any timing of less than one frame, and capable of measuring distance using pupil-splitting image plane phase difference consisting of one imaging optical system and one image sensor; a first object detection means that detects an object based on the result of measuring distance using the image plane phase difference in one of the plurality of imaging means; a second object detection means that detects an object based on the result of stereo distance measurement in two of the plurality of imaging means; an object detection determination means that determines whether the second object detection means can detect an object; and an exposure start timing setting means that sets the exposure start timing of one of the imaging means and the exposure start timing of the other imaging means to be offset when the object detection determination means determines that it cannot detect an object. [Effects of the Invention]

[0010] Even when ambient light levels decrease, the device can be designed to mitigate the reduction in the detection cycle for sudden outbursts and interruptions, depending on the situation. [Brief explanation of the drawing]

[0011] [Figure 1] This is an overall diagram of the system. [Figure 2] This diagram illustrates the positional relationship between camera 100A, camera 100B, and camera control unit 110. [Figure 3] This diagram shows the pixel arrangement of the SPAD-DAF sensor. [Figure 4] This diagram illustrates the timing between the frame start signal, the subframe synchronization signal, and the output DAF ranging image. [Figure 5] This diagram illustrates the different cases for each state pattern. [Figure 6] This is a sequence diagram for determining the state pattern of DAF ranging. [Figure 7] This diagram shows the signal processing of the SPAD-DAF sensor. [Figure 8] This diagram illustrates the exposure time and SNR for each subframe. [Figure 9] It is a diagram for explaining countermeasures for Pattern 2. [Figure 10] It is a diagram for explaining countermeasures for Pattern 3. [Figure 11] It is a diagram for explaining countermeasures for Pattern 4. [Figure 12] It is a diagram for explaining a determination sequence for applying countermeasures for each pattern. [Figure 13] It is an overall system diagram in the second embodiment. [Figure 14] It is a diagram for explaining a determination process for implementing countermeasures in each pattern.

Mode for Carrying Out the Invention

[0012] <The First Embodiment> Hereinafter, Example 1 of the present invention will be described. FIG. 1 is an overall diagram of this system. This system is composed of a camera 100A, a camera 100B, and a camera control unit 110.

[0013] FIG. 2 is a diagram for explaining the positional relationship among camera 100A, camera 100B, and camera control unit 110. This system is installed in a moving body such as an automobile. 201 is an automobile, and the left side of the drawing is the front of the vehicle body. Cameras 100A and 100B are mounted on the front glass of the automobile 201. Cameras 100A and 100B are SPAD-DAF sensor cameras that are capable of DAF distance measurement and equipped with SPAD sensors. That is, it is possible to detect a nearby object from the result of DAF distance measurement using only camera 100A (object detection is possible). Similarly, it is possible to detect a nearby object from the result of DAF distance measurement using only camera 100B. Cameras 100A and 100B mounted on the automobile 201 are configured to be substantially parallel, and stereo distance measurement is possible with cameras 100A and 100B. That is, when using cameras 100A and 100B, it is possible to detect a distant object from the result of stereo distance measurement. Cameras 100A and 100B are connected to camera control unit 110 disposed inside the console of the driver's seat, and are configured such that camera control unit 110 can control cameras 100A and 100B. An ECU 130 is provided beside camera control unit 110, and is capable of transmitting a distance measurement image that is the output of camera control unit 110.

[0014] Returning to FIG. 1, the internal configurations of cameras 100A and 100B will be described. Lenses 101A and 101B are lenses that form images on sensors 102A and 102B. Sensors 102A and 102B are SPAD-DAF sensors. Since each of the SPAD technology and DAF technology is equivalent to the conventional example, detailed description thereof will be omitted. A sensor structure combining the SPAD structure and the DAF structure will be described.

[0015] Figure 3 is a schematic diagram showing the configuration of an image sensor having two light-receiving units within one pixel. Figure 3(A) is a top view of sensors 102A and 102B as seen from the direction of light incidence. Sensor 102 is composed of multiple 2x2 pixel groups 301 arranged in a matrix. Each pixel group 301 has green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 301, green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 302 and a second photoelectric conversion unit 303. The first photoelectric conversion unit 302 and the second photoelectric conversion unit 303 can be controlled differently. Figure 3(B) is a cross-sectional view of the pixel group 301 in Figure 3(A) along the I-I' cross section. Each pixel consists of a microlens 304, a light guide layer 305, and a light receiving layer 306. The light guide layer 305 is a light guide member having a microlens 304 for efficiently guiding light incident on the pixel to the light receiving layer 306, a color filter that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and wiring for image readout and pixel driving. The light receiving layer 306 is a photoelectric conversion unit that converts the light incident on the light guide layer 305 into electrical signals and outputs them as electrical signals. The light receiving layer 306 has a first photoelectric conversion unit 302 and a second photoelectric conversion unit 303. In the above description, the pixel group 301 is arranged as shown in (A) as green pixel G1, green pixel G2, red pixel R, and blue pixel B, but this arrangement is not limited to this, and infrared pixels IR that receive infrared light may be arranged, and the order may be different. Furthermore, in the following explanation, the image signal output from the first photoelectric conversion unit will be referred to as the first image signal, and the image signal output from the second photoelectric conversion unit will be referred to as the second image signal. Generally, distance measurement can be performed by acquiring both the first and second image signals and calculating the parallax amount (positional shift amount) between the first and second image signals. When acquiring an image for viewing purposes, the first and second image signals are combined and acquired.

[0016] Figure 7 shows the equivalent circuit of the signal processing circuit corresponding to one pixel in the pixel group 301 in Figure 3. The APD701A and APD701B included in the photoelectric conversion unit 302 and photoelectric conversion unit 303 generate charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of APD701A and APD701B is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other node of APD701A and APD701B is connected to a power line to which a drive voltage VH (second voltage), which is higher than voltage VL, is supplied.

[0017] In Figure 7, one node of APD701A and APD701B is the anode, and the other node of APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of APD701A and APD701B such that the 907 operates in avalanche multiplication mode. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0018] Furthermore, when a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the voltage difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0019] The signal processing circuit 703 includes switches 702A and 702B, a waveform shaping unit 703A and 703B, a counter circuit 704A, and a counter circuit 704B. Switches 702A and 702B are connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of APD701A or APD701B. Switches 702A and 702B switch the resistance value between APD701A or APD701B and the power line to which the drive voltage VH is supplied. Here, switching the resistance value preferably means changing the resistance value by a factor of 10 or more, and more preferably by a factor of 100 or more. Hereinafter, when the resistance value decreases, it will be referred to as turning on switches 702A and 702B, and when the resistance value increases, it will be referred to as turning off switches 702A and 702B. Switches 702A and 702B function as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to APD701A and APD701B and thereby suppressing avalanche multiplication (quench operation). Switches 702A and 702B also function to restore the voltage supplied to APD701A and APD701B to the drive voltage VH by supplying current to compensate for the voltage drop caused by the quench operation (recharge operation). Counter circuits 704A and 704B count the number of pulses output from waveform shaping units 703A and 703B and hold the count value. Furthermore, when the control pulse RES is supplied via drive lines 705A and 705B, the signals held in counter circuits 704A and 704B are reset. Here, counter circuits 704A and 704B generate signals based on the difference in count values ​​at the start and end of the storage period. Control pulses SEL are supplied to memory circuits 706A and 706B via drive lines 705A and 705B, which switch the electrical connection and disconnection between counter circuits 704A and 704B and vertical signal lines 707A and 707B.Memory circuits 706A and 706B function as memory for temporarily storing the counter's count value, and output the output signals from the pixel counter circuits 704A and 704B to the vertical signal lines 707A and 707B.

[0020] Furthermore, switches such as transistors may be placed between switches 702A and 702B and APD701A and APD701B, or between photoelectric conversion units 302 and 303 and signal processing circuit 703 to switch electrical connections. Similarly, the supply of voltage VH or voltage VL to photoelectric conversion units 302 and 303 may be electrically switched using switches such as transistors. By adopting such a sensor configuration, a SPAD-DAF sensor combining SPAD technology and DAF technology is realized.

[0021] Returning to the explanation of the overall system diagram in Figure 1, development units 105A and 105B are the development units. Sensors 102A and 102B are SPAD-DAF sensors, so they output two images with different parallax. These images are RAW signals in RGGB Bayer array. Since parallax is not required in development units 105A and 105B, the two images with different parallax are combined as RAW signals to generate a single image. Next, debayering and YUV conversion are performed to output a YUV422 color image. In some cases, sensor correction such as correction of defective pixels or noise filtering may be added before development units 105A and 105B to improve image quality. The development processing in this area can be the general type that generates a color signal from a RAW signal.

[0022] Distance measuring units 106A and 106B are distance measuring units. The distance measuring units generate a DAF distance measuring image from two RAW images (right-eye image and left-eye image) from different viewpoints of sensors 102A and 102B. A distance measuring image is an image in which distance values ​​are placed instead of brightness values ​​as in a typical developed image. Each pixel has a distance value, and a larger value indicates a greater distance. The distance calculation method using DAF distance measuring is the same as in conventional examples, so a detailed explanation is omitted here. Sensor control units 103A and 103B receive the frame start signal and subframe synchronization signal generated by the camera control unit 110 and control the imaging timing of sensors 102A and 102B. Details of the timing will be described later.

[0023] Next, the camera control unit 110 will be described. 111 is the frame start signal generation unit. This frame start signal generation unit 111 generates a frame start signal that identifies the beginning of a frame. This frame start signal can generate frame start signal A for camera 100A and frame start signal B for camera 100B separately, and transmits them to sensor control 103A and sensor control 103B. This signal allows the imaging timing of cameras 100A and 100B to be changed. Frame start signal A is sent to the subframe synchronization signal generation unit 112 so that it can be synchronized with the subframe synchronization signal generation unit 112.

[0024] 112 is a subframe synchronization signal generation unit. Based on the frame beginning signal from the frame beginning signal generation unit 111, this subframe synchronization signal generation unit 112 generates a subframe synchronization signal that synchronizes the subframes of the video signals from cameras 100A and 100B. Since the subframe synchronization signal is a common signal sent to cameras 100A and 100B, the developed images and DAF distance measurement images from cameras 100A and 100B are synchronized on a subframe basis. The relationship and timing of this frame beginning signal and subframe synchronization signal are important aspects of this matter, so a detailed explanation will be given later.

[0025] The stereo distance measuring unit 113 is a circuit that calculates a stereo distance measuring image. The inputs are the developed image from camera 100A and the developed image from camera 100B. These two developed images correspond to images with different parallax (right-eye image and left-eye image), similar to the DAF distance measuring process. The stereo distance measuring unit 113 calculates a stereo distance measuring image from these two images (right-eye image and left-eye image).

[0026] The vehicle speed measuring device 114 is a vehicle speed measuring device. It is capable of measuring the rotation speed of the vehicle's tires, and the vehicle's speed can be calculated from the tire radius and rotation speed. The illuminance meter 115 is an ambient illuminance meter that measures the illuminance from the subject entering the camera (ambient illuminance detection means). The values ​​from the speed measuring device 114 and the illuminance meter 115 are sent to the imaging timing control unit 118.

[0027] SW116 is a switching circuit for distance measurement images. The input images are the stereo distance measurement image output from the stereo distance measurement unit 113, the DAF distance measurement image output from distance measurement unit 106A, and the DAF distance measurement image output from distance measurement unit 106B. The distance measurement image to be output is determined under the control of the imaging timing control unit 118.

[0028] 118 is an imaging timing control unit that controls the frame leading signal generation unit 111, the subframe synchronization signal generation unit 112, and SW116. The velocity measuring device 114 and the illuminance meter 115 take measurements at regular intervals based on commands from the imaging timing control unit 118, and the imaging timing control unit 118 acquires the values. These two measurement values ​​are used for synchronization control and control of SW116. As this is an important part of the project, further details will be described later.

[0029] Unit 130 is the ECU (Electronic Control Unit), which receives the distance measurement images generated by this system and uses them as information for determining automatic deceleration and automatic stopping of the vehicle. Since the responsibility of this system is limited to generating the distance measurement images, explanations regarding obstacle recognition and vehicle deceleration decisions are omitted.

[0030] Next, we will explain the imaging timing control unit 118, which is one of the important parts of this case. Figure 4 is a diagram illustrating the timing between the frame start signal, the subframe synchronization signal, and the output DAF distance measurement image. The frame start signal can also be called the exposure start timing. 401 is frame start signal A, 402 is frame start signal B, and 403 is the subframe synchronization signal. 411A and 415A are frame distance measurement images of camera 100A, and 411B and 415B are frame distance measurement images of camera 100B. 412A, 413A, and 414A are subframe distance measurement images of camera 100A, and 412B, 413B, and 414B are subframe distance measurement images of camera 100B. The falling edge of the subframe synchronization signal indicates the start timing of the subframe. The frame start signal is High active, and if the frame start signal is High when the subframe synchronization signal falls, it means that the subframe is the beginning of the frame. At this time, the timing of frame-first signal A (401) and frame-first signal B (402) are synchronized (the exposure start timings are the same), so the image output timings of 411A~415A and 411B~415B are synchronized.

[0031] Next, we will explain the relationship between the output of the SPAD sensor and the SNR. Figure 8 shows the relationship between the exposure time and SNR of the frame distance measurement image and the subframe distance measurement image. The horizontal axis in Figure 8 is time. 911 indicates the exposure start timing, and the falling edge timing is the exposure start timing. Since the SPAD sensor can read the brightness value multiple times at any timing of less than one frame, the subframe distance measurement images can be read as follows. 902, 903, and 904 are subframe distance measurement images read before reaching one frame time, and 901 and 905 are frame distance measurement images generated by exposure for one frame time. 909 is the time of one frame, and is the exposure time of images 901 and 905. 906, 907, and 908 are the exposure times of images 902, 903, and 904, respectively. The exposure times of 906, 907, and 908 are 1 / 4, 2 / 4, and 3 / 4 of one frame time, respectively. Figure 910 shows the signal-to-noise ratio (SNR) in each subframe and frame ranging image, with the vertical axis representing SNR. In this explanation, the exposure time for a subframe is defined as the time taken to divide the frame into four parts, but the number of divisions and time intervals are not limited to this. Similar to CMOS sensors, SPAD-DAF sensor cameras also experience a decrease in SNR and ranging accuracy as the exposure time decreases. Therefore, when ambient illumination decreases, subframe ranging images with short exposure times, such as images 902 and 903, are more likely to fail to meet the required SNR for ranging, resulting in images with reduced ranging accuracy.

[0032] The imaging timing control unit 118 determines whether the DAF distance measurement image can be used based on the ambient illuminance. In other words, it determines whether object detection is possible.

[0033] Furthermore, the system determines whether two images used for stereo distance measurement can be used based on ambient illuminance. In other words, it determines whether or not an object can be detected (object detection determination means).

[0034] The following explains the state patterns of the DAF ranging image and the two images used for stereo ranging.

[0035] <State Patterns 1-5> Figure 5 shows the different cases for each state pattern. 501 is frame-first signal A, 502 is frame-first signal B, and 503 is the subframe synchronization signal. 561 to 565 are the timings for each subframe.

[0036] Pattern 1 consists of frame-based ranging images (511 and 515) and subframe ranging images (512-514). All frames from 511 to 515 have good signal-to-noise ratios (SNR), making it possible to detect obstacles on the road in all frames and subframes.

[0037] Pattern 2 consists of frame-based ranging images (521 and 525) and subframe ranging images (522-524). Only 522 has a poor signal-to-noise ratio (SNR) and is unable to detect obstacles on the road. The remaining images (521, 523-525) have good SNRs and are capable of detecting obstacles on the road in both frame and subframe images.

[0038] Pattern 3 consists of frame-based ranging images (531 and 535) and subframe ranging images (532-534). Images 532 and 533 have poor signal-to-noise ratio (SNR) and are unable to detect obstacles on the road. Images 531, 534, and 535 have good SNR and are capable of detecting obstacles on the road in both frame and subframe images.

[0039] Pattern 4 consists of frame-based ranging images (541 and 545) and subframe ranging images (542-544). Subframes 542-544 have a poor signal-to-noise ratio (SNR), making it impossible to detect obstacles on the road. The remaining frames (551 and 555) have a good SNR, making it possible to detect obstacles on the road.

[0040] Pattern 5 consists of frame-based ranging images (551 and 555) and subframe ranging images (552-554). All of images (551-555) have poor signal-to-noise ratios (SNR) and are unable to detect obstacles on the road surface.

[0041] Next, we will explain how to determine which of patterns 1 to 5 the DAF distance measurement images from cameras 100A and 100B are in. In this embodiment, the relationship between ambient illuminance and the SNR of the DAF distance measurement image has been measured in advance. The material and color of the surface of the obstacle that serves as the subject for cameras 100A and 100B are used as parameters to determine the measurement limit for distance measurement. Figure 6 shows the sequence for determining the pattern. In step 601, ambient illuminance is obtained from the illuminometer 115 in Figure 1. In step 602, it is determined whether the ambient illuminance is higher than the 200 lux obtained in step 601, and if it is higher, the process is moved to step 607. In step 607, it is determined that the pattern is 1 in Figure 5, and the determination is terminated. If step 602 determines that the ambient illuminance is low, the process is moved to step 603. In step 603, it is determined whether the ambient illuminance obtained in step 601 is higher than 100 lux, and if it is higher, the process is moved to step 608. In step 608, it is determined that the pattern is 2 in Figure 5, and the determination is terminated. If 603 determines that the ambient illuminance is low, the process moves to 604. 604 determines if the ambient illuminance obtained in 601 is higher than 66 lux, and if so, the process moves to 609. Process 609 determines that it is pattern 3 in Figure 5, and the process ends. If 604 determines that the ambient illuminance is low, the process moves to 605. 605 determines if the ambient illuminance obtained in 601 is higher than 55 lux, and if so, the process moves to 610. Process 610 determines that it is pattern 4 in Figure 5, and the process ends. If 605 determines that the ambient illuminance is low, it determines that it is pattern 5 in Figure 5, and the process ends. This sequence is used to determine the current situation.

[0042] Next, I will explain the countermeasures for each pattern.

[0043] Pattern 1 does not have a subframe that prevents the detection of obstacles on the road surface, so no countermeasures are necessary.

[0044] The countermeasure for Pattern 2 is shown in Figure 9. 1001 is frame-first signal A, 1002 is frame-first signal B, and 1003 is subframe synchronization signal. 1011A and 1015A are frame-based distance measurement images from camera 100A, and 1011B and 1015B are frame-based distance measurement images from camera 100B. 1012A, 1013A, and 1014A are subframe-based distance measurement images from camera 100A, and 1012B, 1013B, 1014B, and 1016B are subframe-based distance measurement images from camera 100B. Pattern 2 is marked with an "X" because obstacle detection on the road surface is not possible from the subframe-based distance measurement images of 1012A and 1012B.

[0045] As shown in Figure 5, if the frame start signals A501 and B502 are set to the same timing and cameras 100A and 100B are operated at the timing before the countermeasure was implemented, subframe distance measurement images that cannot detect obstacles on the road will be output simultaneously. As a result, as shown in 522 of Figure 5, one subframe period becomes a period during which sudden appearances or interruptions cannot be detected.

[0046] Therefore, by using this timing, only DAF distance measurement will be performed for the duration of two subframes, but it will be possible to properly detect pop-ups and interruptions at the subframe intervals. Specifically, 1001 and 1002 are set with a shift of one subframe. In other words, the exposure start timing setting is shifted by one subframe. Consequently, the imaging timing control unit 118 issues a command to SW116 to output a stereo distance measurement image for period 1020, DAF distance measurement image B for period 1021, DAF distance measurement image A for period 1022, and a stereo distance measurement image for period 1023. Whether to apply this countermeasure in the situation of Pattern 2 depends on the conditions, which will be explained later.

[0047] Figure 10 shows the countermeasures for Pattern 3. 1101 is frame-first signal A, 1102 is frame-first signal B, and 1103 is subframe synchronization signal. 1111A and 1115A are frame-based distance measurement images from camera 100A, and 1111B is frame-based distance measurement image from camera 100B. 1112A, 1113A, and 1114A are subframe-based distance measurement images from camera 100A, and 1112B, 1113B, 1116B, and 1117B are subframe-based distance measurement images from camera 100B. Pattern 3 is marked with an "X" because obstacle detection on the road surface is not possible from the subframe-based distance measurement images 1112A, 1113A, 1112B, 1113B, and 1117B.

[0048] As shown in Figure 5, if camera 100A and camera 100B are operated at the same timing as before the countermeasures were implemented, with frame start signals A501 and B502 being the same timing, subframe distance measurement images that cannot detect obstacles on the road will be output simultaneously. As a result, as shown in 532 and 533 of Figure 5, there will be a period of two subframes during which sudden appearances or interruptions cannot be detected.

[0049] Therefore, by using this timing, although only DAF distance measurement will be possible, it will be possible to properly detect pop-ups and interrupts at subframe intervals. Specifically, 1001 and 1002 are set with a shift of two subframes. In other words, the exposure start timing setting is shifted by two subframes. Consequently, the imaging timing control unit 118 issues a command to SW116 to output DAF distance measurement image A during period 1120, DAF distance measurement image B during period 1121, and DAF distance measurement image A during period 1122. Whether to apply this countermeasure in the situation of Pattern 3 depends on the conditions, which will be explained later.

[0050] Figure 11 shows the countermeasures for Pattern 4. 1201 is frame-first signal A, 1202 is frame-first signal B, and 1203 is subframe synchronization signal. 1211A and 1215A are frame-based distance measurement images from camera 100A, and 1211B is frame-based distance measurement image from camera 100B. 1212A, 1213A, and 1214A are subframe-based distance measurement images from camera 100A, and 1212B, 1213B, and 1217B are subframe-based distance measurement images from camera 100B. Pattern 4 is marked with an "X" because obstacle detection on the road surface is not possible from the subframe-based distance measurement images 1212A, 1213A, 1212B, 1213B, 1216B, and 1217B.

[0051] As shown in Figure 5, if cameras 100A and 100B are operated at the pre-correction timing with frame start signals A501 and B502 set at the same timing, subframe distance measurement images that cannot detect obstacles on the road will be output simultaneously. As a result, as shown in 542, 543, and 544 of Figure 5, there will be a period of three subframes during which sudden appearances or interruptions cannot be detected. By using this timing, it becomes possible to detect sudden appearances or interruptions at a timing of once every two subframes. Specifically, 1001 and 1002 are set with a shift of three subframes. In other words, the exposure start timing setting is shifted by three subframes. Therefore, the imaging timing control unit 118 issues a command to SW116 to output DAF distance measurement image A during period 1220, DAF distance measurement image B during period 1222, and DAF distance measurement image A during period 1224. Periods 1221 and 1223 are periods when distance measurement is not possible, so data with all zeros in the distance value field is sent to the ECU130. By incorporating into the specifications beforehand that distance measurement is impossible when the distance value is 0, the ECU130 can determine that distance measurement is impossible. Whether to apply this countermeasure in the situation of Pattern 4 depends on the conditions, which will be explained later.

[0052] Finally, we will explain the decision-making process for implementing countermeasures in each pattern. As explained earlier, implementing countermeasures minimizes the number of subframes that cannot be measured. However, countermeasures are not always applied; they are only applied when necessary. Figure 12 illustrates the decision-making sequence for applying countermeasures in each pattern.

[0053] Process 1301 acquires vehicle speed information from the vehicle (speed acquisition means). The purpose of acquiring vehicle speed information is to determine in process 1302 whether the situation is likely to cause a sudden darting out (situation determination means). In this case, there are three scenarios assumed to be situations in which a sudden darting out is likely to occur. residential area A street scene with shops lining both sides and cars parked on the side of the road. Traffic is congested on both highways and regular roads. 1) In residential areas, sudden appearances by people such as children, animals, and bicycles are expected. There are many intersections, so it is necessary to slow down and drive carefully. 2) In urban areas, there are many parked cars, and it is expected that people, animals, and bicycles will cross from between them, so it is necessary to slow down and drive carefully. 3) Even in traffic jams, it is expected that cars will cut in from the left or right, and motorcycles will weave through traffic, so it is necessary to drive carefully. All of 1) to 3) are situations where it is inevitable to drive at a reduced speed. Therefore, in process 1302, when the car is traveling at a speed of 40 km / h or less, it is determined that this is a situation where sudden appearances are likely to occur. In other words, when the car's speed is below the threshold, it is determined that object detection is necessary. Conversely, when traveling at a speed higher than that, it is assumed that the road is wide and not congested, allowing for higher speeds, or that it is an uncongested highway, so the possibility of sudden appearances is reduced. If process 1302 determines that it is not a situation where sudden appearances are likely to occur, then no countermeasures are needed in any pattern, so the process moves to 1303 and it is determined that no countermeasures are needed. If process 1302 determines that the situation is likely to cause a sudden departure, process 1304 retrieves the pattern of the current situation. Process 1305 checks if the current situation is pattern 1, and if so, process 1306 determines that no countermeasures are necessary and terminates the process.

[0054] If NO, the process moves to process 1307. Process 1307 checks if the current situation is pattern 2, and if YES, process 1308 determines that no action is needed and terminates the process. If NO, the process moves to process 1309. Process 1309 checks if the current situation is pattern 3, and if YES, process 1310 determines that no action is needed and terminates the process. If NO, the process moves to process 1311. Process 1311 checks if the current situation is pattern 4, and if YES, process 1312 determines that no action is needed and terminates the process. If NO, the process moves to process 1313. Since the current situation is pattern 5, the ECU is notified that this device is in a malfunction state and the process terminates.

[0055] <Second Embodiment> In the first embodiment, when determining whether to implement countermeasures for each pattern (as explained in Figure 12), the determination of whether a situation is likely to occur was made based solely on the vehicle's speed.

[0056] In the second embodiment, an example of acquiring driving environment information from the ECU and making a comprehensive judgment will be described. In the second embodiment, only the differences from the first embodiment will be described.

[0057] Figure 13 is an overall system diagram of the second embodiment. Components similar to those in Figure 1, which is an overall system diagram of the first embodiment, are given the same numbers. The difference from the first embodiment is the driving warning area information 1401 located within the ECU 130. The driving warning area information is a flag (1 bit) indicating a driving warning area, stored in a car navigation system (not shown) connected to the ECU 130. Since the car navigation system is connected to a GPS (not shown), the current location can be determined. When the current location enters a driving warning area stored in the car navigation system, it is sent to the imaging timing control unit 118 as driving warning area information. In the second embodiment, the driving warning area information is data stored in the car navigation system, but it may be updated via OTT or other means.

[0058] Figure 14 illustrates a method for determining whether to implement countermeasures in each pattern, using both the acquired driving warning area information and vehicle speed.

[0059] In process 1514, driving warning area information is obtained from ECU 130. In process 1515, the driving warning area information is checked, and if it is active, the process proceeds to process 1504 (external situation determination means). If it is not active, the process proceeds to process 1501. In process 1501, the vehicle speed is obtained. In process 1502, if the vehicle is traveling at a speed of 40 km / h or less, it is determined that this is a situation where sudden darts are likely to occur. If process 1502 determines that this is not a situation where sudden darts are likely to occur, then no countermeasures are needed in any pattern, so the process moves to 1503 and it is determined that no countermeasures are needed. If process 1502 determines that this is a situation where sudden darts are likely to occur, the pattern of the current situation is obtained in process 1504. In process 1505, it is checked whether the current situation is pattern 1, and if YES, the process ends in process 1506 where it is determined that no countermeasures are needed.

[0060] If NO, the process moves to process 1507. Process 1507 checks if the current situation is pattern 2, and if YES, process 1508 determines that no action is needed and terminates the process. If NO, the process moves to process 1509. Process 1509 checks if the current situation is pattern 3, and if YES, process 1510 determines that no action is needed and terminates the process. If NO, the process moves to process 1511. Process 1511 checks if the current situation is pattern 4, and if YES, process 1512 determines that no action is needed and terminates the process. If NO, the process moves to process 1513. Since the current situation is pattern 5, the ECU is notified that this device is in a malfunction state and the process terminates.

[0061] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0062] Furthermore, the present invention also includes cases in which a software program that realizes the functions of the above-described embodiment is supplied directly from a recording medium or via wired / wireless communication to a system or device having a computer capable of executing the program, and the program is executed.

[0063] Therefore, in order to realize the functional processing of the present invention on a computer or information processing device, the program code itself and the information processing method supplied to and installed on the computer also realize the present invention. In other words, the computer program itself and the information processing method for realizing the functional processing of the present invention are also included in the present invention.

[0064] In that case, the form of the program is irrelevant, as long as it possesses the functionality of a program, including object code, programs executed by an interpreter, and script data supplied to the OS.

[0065] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0066] Another possible method for supplying the program is to store the computer program forming the present invention on a server on a computer network, and then have connected client computers download and run the computer program. [Explanation of Symbols]

[0067] 201 Automobile 100A Camera 100B Camera 110 Camera Control Unit 101A Lens 101B Lens 102A Sensor 102B Sensor 103A Sensor Control 103B Sensor Control 105A Development 105B development 106A ranging 106B Ranging 111 Frame-first signal generation unit 112 Subframe synchronization signal generation unit 113 Stereo rangefinder 114 Speed ​​measuring device 115 Illuminance meter 116 SW 118 Imaging Timing Control Unit 130 ECU

Claims

1. It is possible to read out luminance values ​​multiple times at any timing of less than one frame, and it is equipped with multiple imaging means that are configured to measure distance using pupil-splitting image plane phase difference, consisting of one imaging optical system and one image sensor. Among the plurality of imaging means, one imaging means is a first object detection means that detects an object based on the result of distance measurement using the image plane phase difference, Among the plurality of imaging means, a second object detection means detects an object based on the results of stereo distance measurement using two of the imaging means, The object detection determination means for determining whether or not the second object detection means can detect an object, The system includes an exposure start timing setting means that, when the object detection determination means determines that it cannot detect an object, sets the exposure start timing of one of the imaging means to be offset from the exposure start timing of the other imaging means. An imaging device characterized by the following.

2. It is equipped with a situation determination means that determines whether or not the situation requires the detection of an object, The exposure start timing setting means sets the exposure start timing of one imaging means to be offset from the exposure start timing of the other imaging means when the object detection determination means determines that an object can be detected and the situation determination means determines that an object detection is necessary. The imaging apparatus according to claim 1, characterized by the following:

3. The exposure start timing setting means synchronizes the exposure start timing of the multiple imaging means when the object detection determination means determines that it cannot detect an object, or when the situation determination means determines that it is not a situation where object detection is necessary. The imaging apparatus according to claim 2, characterized by the following:

4. The first object detection means is a compound eye camera comprising a lens array having multiple lenses arranged on the same plane and a sensor, and detects an object by outputting a distance measurement image from the parallax of multiple individual eye images captured by the compound eye camera. The second object detection means detects an object by outputting a distance measurement image from the parallax of two captured images acquired from each of the two imaging means. The imaging apparatus according to claim 1, characterized by the following:

5. The exposure start timing setting means shifts the exposure start timing of the other imaging means so that when the first object detection means of one imaging means cannot detect an object, the first object detection means of the other imaging means can detect an object. The imaging apparatus according to feature 1.

6. Equipped with an ambient illuminance detection means for detecting ambient illuminance, The object detection determination means determines whether the second object detection means can detect an object based on the ambient illuminance. The imaging apparatus according to feature 1.

7. The system includes a speed acquisition means for acquiring the speed of a moving object on which the imaging device is installed, The situation determination means determines that an object detection is necessary when the speed of the moving object is below a threshold. The imaging device according to feature 2.

8. It is equipped with an external situation determination means that determines whether or not a situation is likely to occur where people, animals, or moving objects are likely to suddenly appear or cut in, The situation determination means determines that object detection is necessary when the external situation determination means determines that a situation is likely to occur in which a person, animal, or moving object is likely to suddenly appear or cut in. The imaging device according to feature 2.

9. It is possible to read out luminance values ​​multiple times at any timing of less than one frame, and it has multiple imaging processes that enable distance measurement using pupil-splitting image plane phase difference, consisting of one imaging optical system and one image sensor. Among the plurality of imaging steps, one imaging step includes a first object detection step in which an object is detected based on the result of distance measurement using the image plane phase difference, Of the plurality of imaging steps, a second object detection step is performed in which an object is detected based on the results of stereo distance measurement in two of the imaging steps, The object detection determination step determines whether or not an object can be detected in the second object detection step, The system includes, if the object detection determination step determines that an object cannot be detected, an exposure start timing setting step that sets the exposure start timing of one of the imaging steps to be offset from the exposure start timing of the other imaging step. An information processing method performed by an information processing device, characterized by the following:

10. A computer program for causing a computer to execute each step of the information processing method described in claim 9.

Images