Image capturing apparatus, information processing method, and storage medium

Abstract

The invention discloses an image capturing apparatus, an information processing method, and a storage medium. When it is determined that there is a ranging image in which an object will not be detected and it is determined that a situation in which the vehicle is currently traveling is a situation in which a jump-in object or a cut-in object is likely to occur, the exposure start timing of one of the cameras constituting the stereo camera is shifted.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image capturing device that performs object detection by acquiring a parallax based on a plurality of images and is mounted on a vehicle, an information processing method, and a storage medium. BACKGROUND

[0002] In recent years, automobiles have been equipped with a function of detecting an obstacle on a travel path of the automobile and automatically decelerating or stopping the automobile. This is a so-called advanced driver assistance system (ADAS).

[0003] As one of techniques for the ADAS for detecting an obstacle, a stereo camera has been proposed. The stereo camera can perform triangulation by using a parallax of two cameras to measure a distance to an obstacle. This scheme is referred to as a stereo ranging scheme in the present application. The stereo camera can also be said to be a camera configured to measure a distance based on a parallax of two captured images acquired from two image capturing units, respectively. The stereo ranging scheme has a characteristic that as a separation (=baseline length) between the two cameras is wider, the accuracy of a long distance is higher, and as the separation is narrower, the accuracy of a short distance is higher. In view of the above, in the related art, Japanese Patent Publication No. 2020-53950 describes a technique of arranging a plurality of pairs of cameras having different baseline lengths or providing a plurality of pairs of cameras in order to secure ranging accuracy from a short distance to a long distance. According to Japanese Patent Publication No. 2020-53950, two sets of stereo cameras (four cameras) need to be prepared, and there is a disadvantage in cost.

[0004] As one of solutions, in Japanese Patent Publication No. 2011-203238, a system that also uses a ranging scheme using an image plane phase difference has also been proposed. The scheme includes a camera configured to perform ranging by using a sensor in which two pixels (one pixel in a normal configuration) are arranged under each of microlenses arranged on an image capturing sensor and can acquire two images having different viewpoints through the image capturing sensor. The camera can also be said to be a camera capable of performing ranging using an image plane phase difference of a pupil division scheme constituted by one imaging optical system and one image capturing element.

[0005] Alternatively, the camera can also be said to be a compound eye camera provided with a lens array having a plurality of lenses arranged on the same plane and a sensor that is capable of performing ranging based on the parallax of a plurality of independent eye images captured by the compound eye camera. This scheme is referred to as a dual-pixel AF (DAF) ranging scheme in the present application due to the original use for autofocus (AF) in a single-lens reflex camera. Japanese Patent Publication No. 2011-203238 introduces a system in which two cameras of the DAF ranging scheme are prepared, stereo ranging is performed for long-distance ranging by using images respectively output from the two cameras, and DAF ranging based on a single camera is used for short-distance ranging.

[0006] One of the important performance indicators of an ADAS camera is a high frame rate.

[0007] When an obstacle such as a jumping-in or a cutting-in suddenly appears, it is necessary to detect these obstacles as quickly as possible in order for the ADAS camera to safely decelerate or stop the vehicle. For this purpose, it is necessary to increase the frame rate at which the camera performs image capture in order to shorten the detection period of the jumping-in and the cutting-in.

[0008] As one of the techniques for this high frame rate, a SPAD sensor camera using a single-photon avalanche diode (SPAD) sensor is proposed in Japanese Patent Publication No. 2022-106660. The SPAD sensor is a sensor using a photon counter, and in addition to high sensitivity, it also has a high frame rate readout function. Due to the characteristics of the photon counter, since the luminance value of the pixel is digitally stored, the luminance value can be repeatedly read out. For this reason, it is possible to acquire image data in an early stage before the default exposure time is reached. In other words, the luminance value can be read out multiple times at an arbitrary time point during a frame, that is, at an arbitrary time point within the time required to capture a frame while the frame is being captured. With this configuration, a higher frame rate than that of a normal CMOS sensor can be achieved.

[0009] However, in the related art example described above, it can be difficult to shorten the detection period of the jumping-in and the cutting-in. This case includes the case where the environmental illuminance is low. As in the CMOS sensor, in the SPAD sensor camera as well, the shorter the exposure time, the lower the signal-to-noise ratio (SNR) and the lower the ranging accuracy. Therefore, in the case where the environmental illuminance is low, since the ranging image acquired in the early stage from the exposure time has a low SNR, the ranging accuracy decreases, and the obstacle detection performance decreases. As a result, in the case where the environmental illuminance is low, since it is difficult to detect the obstacle in the early stage from the exposure time, there is a risk that shortening the detection period will cause the obstacle to be successfully detected. SUMMARY

[0010] In view of the above, the present disclosure aims to provide an apparatus that reduces the deceleration of the detection period of a jump-in object and a cut-in object according to the situation even in the case where the ambient illuminance decreases.

[0011] A first disclosure of an image capturing apparatus according to the present application is an image capturing apparatus including at least one processor or circuit configured to function as: a plurality of image capturing units each configured to be capable of reading out a luminance value multiple times at an arbitrary time point during one frame and to be capable of performing ranging using an image plane phase difference of a pupil division scheme constituted by one imaging optical system and one image capturing element; a first object detection unit configured to detect an object based on a result of ranging using the image plane phase difference by one of the plurality of image capturing units; a second object detection unit configured to detect an object based on a result of stereoscopic ranging by two of the plurality of image capturing units; an object detection determination unit configured to determine whether an object can be detected by the second object detection unit; and an exposure start timing setting unit configured to set an exposure start timing of one of the image capturing units and an exposure start timing of another of the image capturing units so as to offset the exposure start timings from each other in a case where the object detection determination unit determines that an object cannot be detected.

[0012] Features of the present disclosure will become apparent from the following description of embodiments with reference to the drawings. The following description of embodiments is given by way of example only. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a whole diagram of the present system.

[0014] Figure 2 is an explanatory diagram for describing a positional relationship regarding how to position a camera, another camera, and a camera control unit.

[0015] Figure 3A illustrates a pixel arrangement of a SPAD-DAF sensor.

[0016] Figure 3B illustrates a pixel arrangement of a SPAD-DAF sensor.

[0017] Figure 4 is an explanatory diagram for describing timings of a frame leading signal, a subframe synchronization signal, and an output DAF ranging image.

[0018] Figure 5 The diagram illustrates the classification of state patterns.

[0019] Figure 6 It is a sequence diagram used to determine the state mode of DAF ranging.

[0020] Figure 7 The diagram illustrates the signal processing of the SPAD-DAF sensor.

[0021] Figure 8 It is an explanatory diagram used to describe the exposure time and SNR of each subframe.

[0022] Figure 9 This is an illustrative diagram used to describe measures for Mode 2.

[0023] Figure 10 This is an illustrative diagram used to describe measures for Mode 3.

[0024] Figure 11 This is an illustrative diagram used to describe measures for Mode 4.

[0025] Figure 12 It is an illustrative diagram used to describe a defined sequence of actions for each pattern.

[0026] Figure 13 This is a diagram of the overall system in the second embodiment.

[0027] Figure 14 It is an illustrative diagram used to describe the determination process regarding whether a measure is implemented in each mode. Detailed Implementation

[0028] Hereinafter, embodiments of the invention will be described with reference to the accompanying drawings. These embodiments are not intended to limit the invention as described in the appended claims. Each of the embodiments of the invention described below can be implemented individually or as a combination of multiple embodiments or features thereof, if desired, or if a combination of embodiments or elements or features from the various embodiments is advantageous. Multiple features are described in the embodiments, but not all of these features are always essential to the invention.

[0029] First Embodiment

[0030] The first embodiment of this disclosure will be described below. Figure 1 This is an overall diagram of the system. The system consists of camera 100A, camera 100B, and camera control unit 110.

[0031] Figure 2This is an explanatory diagram illustrating the positional relationship between cameras 100A, 100B, and camera control unit 110. This system is installed in a moving object such as a car. Regarding car 201, the left side of the diagram represents the front of the vehicle. Cameras 100A and 100B are mounted on the windshield of car 201. Cameras 100A and 100B are SPAD-DAF sensor cameras capable of performing DAF ranging and equipped with SPAD sensors. In other words, using only camera 100A, objects at short distances can be detected based on the results of DAF ranging (object detection is possible). Similarly, using only camera 100B, objects at short distances can be detected based on the results of DAF ranging. Cameras 100A and 100B mounted on car 201 are configured to be substantially parallel to each other, and cameras 100A and 100B can perform stereo ranging. In other words, when using cameras 100A and 100B, objects at long distances can be detected based on the results of stereo ranging. The following configuration is adopted: Cameras 100A and 100B are connected to a camera control unit 110 located inside the driver's seat console, and cameras 100A and 100B can be controlled from the camera control unit 110. An electronic control unit (ECU) 130 is also mounted next to the camera control unit 110 and can send ranging images as the output of the camera control unit 110.

[0032] Refer again Figure 1 The internal configurations of cameras 100A and 100B will be described first. Lenses 101A and 101B are lenses that form images on sensors 102A and 102B. Sensors 102A and 102B are SPAD-DAF sensors. Since each of the SPAD and DAF technologies is equivalent to the technology in the related technical examples, their detailed descriptions will be omitted.

[0033] The sensor structure obtained by combining SPAD and DAF structures will be described.

[0034] Figure 3A and Figure 3B This is a schematic diagram illustrating the configuration of an image capture element having two light receiving units in a single pixel. Figure 3AThis is a top view of sensors 102A and 102B as viewed from the direction of light incidence. Each of sensors 102A and 102B is constructed by arranging multiple pixel groups 301 in a matrix of two rows × two columns. Each pixel group 301 has a green pixel G1 and a green pixel G2 for detecting green light, a red pixel R for detecting red light, and a blue pixel B for detecting blue light. In pixel group 301, green pixels G1 and G2 are arranged diagonally opposite each other. In addition, each pixel has a first photoelectric conversion unit 302 and a second photoelectric conversion unit 303. Regarding the control of the first photoelectric conversion unit 302 and the second photoelectric conversion unit 303, different types of control can be performed on the first photoelectric conversion unit 302 and the second photoelectric conversion unit 303 respectively. Figure 3B yes Figure 3A The image shows cross-sectional views of pixel group 301 in sections IIIB-IIIB. Each pixel is composed of a microlens 304, a light guiding layer 305, and a light receiving layer 306. The light guiding layer 305 is a light guiding member having a microlens 304 arranged to efficiently guide light incident on the pixel to the light receiving layer 306, a color filter arranged to allow light of a wavelength band corresponding to the color of the light detected by each pixel to pass through it, and wiring for image readout and pixel driving. The light receiving layer 306 is a photoelectric conversion unit configured to perform photoelectric conversion of light incident via the light guiding layer 305 into an electrical signal to be output. The light receiving layer 306 has a first photoelectric conversion unit 302 and a second photoelectric conversion unit 303. Furthermore, in the above description, as... Figure 3A As illustrated, green pixel G1, green pixel G2, red pixel R, and blue pixel B are arranged as pixel group 301, but this arrangement is not limited to the above. Infrared pixels IR, configured to receive infrared light, can be arranged, and the order can be different. Furthermore, in the following description, the image signal output from the first photoelectric conversion unit will be described as the first image signal, and the image signal output from the second photoelectric conversion unit will be described as the second image signal. Typically, ranging can be performed by acquiring each of the first and second image signals and calculating the disparity (position offset) between the first and second image signals. Additionally, when acquiring an image for observation purposes, an image acquired by combining the first and second image signals is acquired.

[0035] Figure 7 The diagram illustrates the relationship between... Figure 3A and Figure 3BThe equivalent circuit of the signal processing circuit corresponding to one pixel in pixel group 301. The APD 701A and APD 701B included in the first photoelectric conversion unit 302 and the second photoelectric conversion unit 303 generate charge pairs according to the incident light through photoelectric conversion. One of the two nodes of APD 701A and APD 701B is connected to a power supply line supplying a driving voltage VL (first voltage). The other node of APD 701A and APD 701B is connected to a power supply line supplying a driving voltage VH (second voltage) that is higher than voltage VL.

[0036] exist Figure 7 In this design, one node of APD 701A and APD 701B is the anode, and the other node of APD 701A and APD 701B is the cathode. The anode and cathode of APD 701A and APD 701B are supplied with a reverse bias voltage to perform avalanche multiplication operation during exposure time 907. By establishing the state of supplying the aforementioned voltage, the charge generated by the incident light causes avalanche multiplication, and an avalanche current is generated.

[0037] It is important to note that, when a reverse bias voltage is supplied, there exists a Geiger mode in which the APD operates with a voltage difference between the anode and cathode greater than the breakdown voltage, and a linear mode in which the APD operates with a voltage difference between the anode and cathode close to, or less than or equal to, the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V, and the voltage VH (second voltage) is 1V.

[0038] The signal processing circuit 703 includes switches 702A and 702B, waveform shaping units 703A and 703B, and counter circuits 704A and 704B. Switches 702A and 702B are connected to a power supply line supplying the drive voltage VH and to one of the nodes at the anode and cathode of APDs 701A and 701B. Switches 702A and 702B then switch the resistance value between APDs 701A and 701B and the power supply line supplying the drive voltage VH. Here, to switch the resistance value, it is preferable to change the resistance value by a factor of 10 or more, and more preferably by a factor of 100 or more. Hereinafter, a decrease in the resistance value is also referred to as turning on switches 702A and 702B, and an increase in the resistance value is also referred to as turning off switches 702A and 702B. Switches 702A and 702B function as load circuits (quenching circuits) during signal amplification via avalanche multiplication and have the ability to suppress the voltage supplied to APDs 701A and 701B to suppress avalanche multiplication (quenching operation). Additionally, switches 702A and 702B have the ability to restore the voltage supplied to APDs 701A and 701B to the drive voltage VH by allowing a current equivalent to the voltage drop during the quenching operation to flow (recharge operation). Counter circuits 704A and 704B count the number of pulses output from waveform shaping units 703A and 703B and maintain the count value.

[0039] When a control pulse RES is supplied via drive lines 705A and 705B, the signals held in counter circuits 704A and 704B are reset. In this document, counter circuits 704A and 704B generate signals based on the difference between the count value at the start of the accumulation period and the count value at the end of the accumulation period. A control pulse SEL is supplied via drive lines 705A and 705B to memory circuits 706A and 706B to switch the electrical connection and disconnection between counter circuit 704A and vertical signal line 707A, and between counter circuit 704B and vertical signal line 707B. Memory circuits 706A and 706B serve as memories configured to temporarily store the count values ​​of the counters, and output the output signals from counter circuits 704A and 704B from the pixels to vertical signal lines 707A and 707B.

[0040] Note that electrical connections can be switched by arranging switches such as transistors between switches 702A and APD 701A, between switches 702B and APD 701B, or between the first photoelectric conversion unit 302 and the signal processing circuit 703, and between the second photoelectric conversion unit 303 and the signal processing circuit 703. Similarly, the supply of voltage VH or voltage VL to be supplied to the first photoelectric conversion unit 302 and the second photoelectric conversion unit 303 can be electrically switched by using switches such as transistors. When such a sensor configuration is adopted, a SPAD-DAF sensor obtained by combining SPAD technology and DAF technology is realized.

[0041] Describe again Figure 1 The diagram shows the overall system configuration. Developing units 105A and 105B constitute the developing unit. Since sensors 102A and 102B are SPAD-DAF sensors, they output two images with different parallaxes. Furthermore, these images are RAW (raw) signals from the Bayer array of the RGGB.

[0042] Parallax is not required in development 105A and development 105B; two images with different parallax are added as RAW signals to produce a single image. Next, Bayer removal and YUV conversion are performed to output a YUV422 color image. Depending on the situation, sensor corrections such as sensor defect pixel correction and noise filtering can be included before development 105A and development 105B to improve image quality. This type of development process can be a general process for generating color signals based on RAW signals.

[0043] Ranging units 106A and 106B constitute a ranging unit. Within this unit, a DAF ranging image is generated based on two RAW images (right-eye and left-eye images) from sensors 102A and 102B with different viewpoints. The ranging image has distance values ​​instead of brightness values ​​in a typical developed image. Distance values ​​are provided pixel-by-pixel, with larger values ​​indicating greater distance. Since the distance calculation method based on DAF ranging is similar to related art examples, a detailed description will be omitted herein. A frame preamble signal and a subframe synchronization signal generated by the camera control unit 110 are input to sensor controls 103A and 103B to control the image capture timing of sensors 102A and 102B. Details of the timing will be described below.

[0044] Next, the camera control unit 110 will be described. The frame preamble signal generation unit 111 generates a frame preamble signal for identifying the preamble of a frame. As this frame preamble signal, frame preamble signal A for camera 100A and frame preamble signal B for camera 100B can be generated and sent to sensor control 103A and sensor control 103B respectively. Based on this signal, the image capture timing of cameras 100A and 100B can be changed. Frame preamble signal A is sent to the subframe synchronization signal generation unit 112 for synchronization.

[0045] The subframe synchronization signal generation unit 112 generates a subframe synchronization signal based on the frame preamble signal from the frame preamble signal generation unit 111, for synchronizing the video signals of cameras 100A and 100B across subframes. Since the subframe synchronization signal sends a shared signal to cameras 100A and 100B, the developed images from cameras 100A and 100B are synchronized with the DAF ranging images on a subframe basis. Because the relationship and timing between the frame preamble signal and the subframe synchronization signal are important aspects of this application, they will be described in detail below.

[0046] The stereo ranging unit 113 is a circuit configured to calculate a stereo ranging image. The inputs include the developed image from camera 100A and the developed image from camera 100B. Similar to DAF ranging, these two developed images are equivalent to images with different parallax (right-eye image and left-eye image). The stereo ranging unit 113 calculates the stereo ranging image based on these two images (right-eye image and left-eye image).

[0047] Vehicle speed measuring device 114 is a speed measuring device for automobiles. Vehicle speed measuring device 114 can measure the number of rotations of the automobile tires and can calculate the automobile speed from the tire radius and the number of rotations. Illuminance meter 115 is an ambient illuminance meter and measures the illuminance from the subject incident on the camera (ambient illuminance detection unit). The values ​​from the speed measuring device and illuminance meter 115 are sent to image capture timing control unit 118.

[0048] SW 116 is a switching circuit for the ranging image. The input images include the stereo ranging image output by stereo ranging unit 113, the DAF ranging image output by ranging unit 106A, and the DAF ranging image output by ranging unit 106B. The ranging image to be output is determined under the control of image capture timing control unit 118.

[0049] The image capture timing control unit 118 controls the frame preamble signal generation unit 111, the subframe synchronization signal generation unit 112, and the SW 116. The velocity measuring device 114 and the illuminance meter 115, in response to commands from the image capture timing control unit 118, perform measurements at regular intervals to acquire values ​​via the image capture timing control unit 118. Synchronization control and the control of the SW 116 are performed using these two measurements. Since this is an important aspect of this application, details will be described below.

[0050] The electronic control unit (ECU) 130 is configured to receive a ranging image generated by this system, and the ranging image is used as determining information for automatic deceleration or automatic stopping of the vehicle. This system involves steps performed up to the generation of the ranging image; therefore, descriptions regarding obstacle recognition, vehicle deceleration determination, etc., will be omitted.

[0051] Next, the image capture timing control unit 118, which is one of the important aspects of this application, will be described. Figure 4 This is an illustrative diagram used to describe the timing of the frame preamble, subframe synchronization signal, and the DAF ranging image to be output. The frame preamble can also be referred to as the exposure start timing. Figure 4 The diagram illustrates signal 401 as frame preamble signal A, signal 402 as frame preamble signal B, and signal 403 as subframe synchronization signal. Images 411A and 415A are frame ranging images of camera 100A, and images 411B and 415B are frame ranging images of camera 100B. Images 412A, 413A, and 414A are subframe ranging images of camera 100A, and images 412B, 413B, and 414B are subframe ranging images of camera 100B. A drop in the subframe synchronization signal indicates the start of subframe timing. The frame preamble signal is active high and indicates that the subframe is the preamble of a frame when the frame preamble signal is high while the subframe synchronization signal is falling. At this time, since the timing of signal 401, which serves as frame preamble signal A, is aligned with the timing of signal 402, which serves as frame preamble signal B (exposure start timing match), the image output timing of images 411A to 415A is aligned with the image output timing of images 411B to 415B.

[0052] Next, the relationship between the output of the SPAD sensor and SNR will be described. Figure 8 The diagram illustrates the relationship between exposure time and SNR for frame ranging images and subframe ranging images. Figure 8 The horizontal axis represents time. Figure 8The illustration shows the exposure start timing 911, and the fall timing is the exposure start timing. Since the SPAD sensor can read out brightness values ​​multiple times within any timing interval less than one frame, the subframe ranging images can be read out as follows. Images 902, 903, and 904 are subframe ranging images read out before reaching one frame time, and images 901 and 905 are frame ranging images generated after the exposure reaches one frame time. Figure 8 The illustration also shows a frame time 909, which serves as the exposure time for images 901 and 905. Exposure times 906, 907, and 908 correspond to images 902, 903, and 904, respectively. Exposure times 906, 907, and 908 represent 1 / 4, 2 / 4, and 3 / 4 of the duration of a frame, respectively. SNR 910 indicates the SNR of each of the subframe ranging images and the frame ranging images, where the vertical axis represents the signal-to-noise ratio (hereinafter, SNR). In this description, the exposure time of a subframe corresponds to the time obtained by dividing the frame into four parts, but the number of segments and the time interval are not limited to the above. Similar to CMOS sensors, as the exposure time decreases, the SNR further decreases in SPAD-DAF sensor cameras, and the ranging accuracy also further decreases. Therefore, in situations of decreased ambient light, subframe ranging images with short exposure times, such as images 902 or 903, do not meet the required SNR for ranging, and there is a high probability of acquiring images with reduced ranging accuracy.

[0053] The image capture timing control unit 118 determines whether the DAF ranging image can be used based on the ambient illumination. In other words, the image capture timing control unit 118 determines whether an object can be detected.

[0054] The image capture timing control unit 118 also determines whether two images intended for stereo ranging can be used based on ambient illumination. In other words, it determines whether the DAF ranging image can be used based on ambient illumination. In other words, the image capture timing control unit 118 determines whether an object can be detected (this can be referred to as an object detection determination unit). The image capture timing control unit 118 may determine only whether the DAF ranging image can be used, and / or whether two images used for stereo ranging can be used.

[0055] The following text will describe the state of the DAF ranging image and the two images used for stereo ranging based on state modes.

[0056] State Modes 1 to 5

[0057] Figure 5 The diagram illustrates the classification of state patterns. Figure 5 The diagram illustrates the frame preamble signal A501, the frame preamble signal B502, the subframe synchronization signal 503, and the timing of each subframe from 561 to 565.

[0058] Mode 1 represents images 511 and 515 as frame ranging images and images 512 to 514 as subframe ranging images. In this mode, all images 511 to 515 have satisfactory SNR, and obstacles on the driving path can be detected in all frames and subframes.

[0059] Mode 2 represents images 521 and 525 as frame ranging images and images 522 to 524 as subframe ranging images. In this mode, only image 522 has an unsatisfactory SNR and is established as a state where obstacles on the driving path will not be detected. On the other hand, the remaining images 521 and 523 to 525 have satisfactory SNR and obstacles on the driving path can be detected in both frames and subframes.

[0060] Mode 3 represents images 531 and 535 as frame ranging images and images 532 to 534 as subframe ranging images. In this mode, images 532 and 533 have unsatisfactory SNR and are established as not detecting obstacles on the driving path, but on the other hand, the remaining images 531, 534, and 535 have satisfactory SNR and can detect obstacles on the driving path in both frames and subframes.

[0061] Mode 4 represents images 541 and 545 as frame ranging images and images 542 to 544 as subframe ranging images. In this mode, images 542 to 544, as subframes, have unsatisfactory SNR and are established as not detecting obstacles on the driving path. On the other hand, the remaining images 541 and 545 have satisfactory SNR and can detect obstacles on the driving path.

[0062] Mode 5 represents images 551 and 555 as frame ranging images and images 552 to 554 as subframe ranging images. In this mode, all images 551 to 555 have unsatisfactory SNR and establish a state where obstacles on the driving path will not be detected.

[0063] Next, a method for determining the state of the DAF ranging images of cameras 100A and 100B in the states corresponding to modes 1 to 5 will be described. In this embodiment, the relationship between ambient illumination and the SNR of the DAF ranging image is measured in advance. The measurement limit of the ranging is determined by classifying the material, color, etc. of the surface of the obstacle to be the subject of cameras 100A and 100B as parameters. Figure 6 The diagram illustrates the sequence used to determine the pattern. In 601, from... Figure 1The illuminance meter 115 acquires the ambient illuminance. In step 602, it is determined whether the ambient illuminance acquired in step 601 is higher than 200 lux, and if the ambient illuminance is higher than 200 lux, the process proceeds to step 607. In step 607, it is determined that the state is... Figure 5 In Mode 1, and the process ends. When it is determined in 602 that the ambient illuminance is below 200 lux, the process proceeds to 603. In 603, it is determined whether the ambient illuminance obtained in 601 is above 100 lux, and if the ambient illuminance is above 100 lux, the process proceeds to 608. In process 608, it is determined that the state is... Figure 5 In Mode 2, and the process ends. When it is determined in 603 that the ambient illuminance is below 100 lux, the process moves to 604. In 604, it is determined whether the ambient illuminance obtained in 601 is above 66 lux, and if the ambient illuminance is above 66 lux, the process moves to 609. In process 609, it is determined that the state is... Figure 5 In Mode 3, and the process ends. When it is determined in 604 that the ambient illuminance is below 66 lux, the process proceeds to 605. In 605, it is determined whether the ambient illuminance obtained in 601 is above 55 lux, and if the ambient illuminance is above 55 lux, the process proceeds to 610. In process 610, it is determined that the state is... Figure 5 In Mode 4, and the determination ends. When the ambient illuminance is determined to be below 55 lux in 605, the determination state is in... Figure 5 In pattern 5, and the end is determined. The current situation is determined by using this sequence.

[0064] Next, we will describe the measures for each mode.

[0065] Since there are no subframes in Mode 1 that will not detect obstacles on the driving path, no measures are needed.

[0066] Figure 9 The diagram illustrates the measures for Mode 2. Figure 9 The diagram illustrates signal 1001 as frame preamble signal A, signal 1002 as frame preamble signal B, and signal 1003 as subframe synchronization signal. Images 1011A and 1015A are frame ranging images of camera 100A, and images 1011B and 1015B are frame ranging images of camera 100B. Images 1012A, 1013A, and 1014A are subframe ranging images of camera 100A, and images 1012B, 1013B, 1014B, and 1016B are subframe ranging images of camera 100B. In Mode 2, since obstacles on the driving path will not be detected based on images 1012A and 1012B, which are subframe ranging images, these images are crossed out.

[0067] likeFigure 5 In this process, when frame preamble signal A 501 and frame preamble signal B 502 are set to the same timing, and cameras 100A and 100B are timed before taking action, a subframe ranging image that will not detect obstacles on the driving path is simultaneously output. For this reason, with... Figure 5 Similar to image 522, the time period for a subframe of an image becomes the time period for which jumps and cuts will not be detected.

[0068] In view of the above, by adopting this timing, although only DAF ranging is performed during the time period of the subframes used for two images, it becomes possible to appropriately detect jump-ins and cut-ins at subframe intervals.

[0069] Specifically, signals 1001 and 1002 are set by offsetting by one subframe. In other words, the exposure start timing setting is offset by one subframe. Therefore, commands are issued from the image capture timing control unit 118 such that SW 116 outputs a stereo ranging image during time period 1020, SW 116 outputs DAF ranging image B during time period 1021, SW 116 outputs DAF ranging image A during time period 1022, and SW 116 outputs a stereo ranging image during time period 1023. In the case of Mode 2, whether action is taken depends on the conditions, which will be described later.

[0070] Figure 10 The diagram illustrates the measures for Mode 3. Figure 10 The diagram illustrates signal 1101 as frame preamble signal A, signal 1102 as frame preamble signal B, and signal 1103 as subframe synchronization signal. Images 1111A and 1115A are frame ranging images of camera 100A, and image 1111B is a frame ranging image of camera 100B. Images 1112A, 1113A, and 1114A are subframe ranging images of camera 100A, and images 1112B, 1113B, 1116B, and 1117B are subframe ranging images of camera 100B. In Mode 3, since obstacles on the driving path will not be detected based on images 1112A, 1113A, 1112B, 1113B, and 1117B as subframe ranging images, these images are crossed out.

[0071] like Figure 5 In this process, when frame preamble signal A 501 and frame preamble signal B 502 are set to the same timing, and cameras 100A and 100B are timed before taking action, a subframe ranging image that will not detect obstacles on the driving path is simultaneously output. For this reason, with... Figure 5 Similar to images 532 and 533, the time period for the subframes of the two images becomes the time period for not detecting jumps and cuts.

[0072] In view of the above, by adopting this timing, although only DAF ranging is performed, it becomes possible to appropriately detect jumps and cuts at subframe intervals. Specifically, signals 1001 and 1002 are set by offsetting by two subframes. In other words, the exposure start timing setting is offset by two subframes. Therefore, commands are issued from the image capture timing control unit 118 in such a manner that SW 116 outputs DAF ranging image A during time period 1120, SW 116 outputs DAF ranging image B during time period 1121, and SW 116 outputs DAF ranging image A again during time period 1122. In the case of mode 3, whether to take action depends on the conditions, which will be described later.

[0073] Figure 11 The diagram illustrates the measures for Mode 4. Figure 11 The diagram illustrates signal 1201 as frame preamble signal A, signal 1202 as frame preamble signal B, and signal 1203 as subframe synchronization signal. Images 1211A and 1215A are frame ranging images of camera 100A, and image 1211B is a frame ranging image of camera 100B. Images 1212A, 1213A, and 1214A are subframe ranging images of camera 100A, and images 1212B, 1213B, and 1217B are subframe ranging images of camera 100B. In mode 4, since obstacles on the driving path will not be detected based on images 1212A, 1213A, 1212B, 1213B, 1216B, and 1217B as subframe ranging images, these images are crossed out.

[0074] like Figure 5 In this process, when frame preamble signal A 501 and frame preamble signal B 502 are set to the same timing, and cameras 100A and 100B are timed before taking action, a subframe ranging image that will not detect obstacles on the driving path is simultaneously output. For this reason, with... Figure 5Similar to images 542, 543, and 544, the time periods for the subframes of the three images become periods during which intrusions and cut-ins will not be detected. In view of the above, by adopting this timing, intrusions and cut-ins can be detected during the timing of one of the two subframe time periods. Specifically, signals 1001 and 1002 are set by offsetting by three subframes. In other words, the exposure start timing setting is offset by three subframes. Therefore, commands are issued from the image capture timing control unit 118 in such a manner that SW116 outputs DAF ranging image A during time period 1220, SW116 outputs DAF ranging image B during time period 1222, and SW116 outputs DAF ranging image A during time period 1224. Since time periods 1221 and 1223 are periods during which ranging will not be performed, data with ranging values ​​filled with 0 is supplied to ECU 130. By agreeing with ECU 130, a rule is included in the specifications beforehand that distance measurement cannot be performed when the distance measurement value is 0, making it possible to determine that distance measurement cannot be performed in ECU 130. In the case of Mode 4, whether to take action depends on the conditions, which will be described later.

[0075] Finally, the determination of whether to take action in each mode will be described. As explained so far, when actions are implemented, it becomes possible to minimize the subframes where ranging will not be performed. However, it should be noted that instead of always taking action, actions should only be taken when necessary. Figure 12 It is an illustrative diagram used to describe a defined sequence of actions for each pattern.

[0076] In process 1301, vehicle speed information is obtained from the vehicle (speed acquisition unit). The reason for obtaining the vehicle speed information is to determine in process 1302 whether a jumping object is likely to occur in the scenario (scenario determination unit). In this application, three scenarios are envisioned as scenarios where a jumping object may occur.

[0077] Residential area

[0078] The city has shops on both sides of the road and roadside parking on the shoulder.

[0079] Traffic congestion, regardless of highways / roads

[0080] 1) In residential areas, the possibility of people, animals, and bicycles jumping into the road is anticipated. There are many intersections, requiring drivers to slow down and drive carefully. 2) In urban areas, there is a large amount of roadside parking, and the possibility of people, animals, and bicycles crossing the parking spaces necessitates slowing down and driving carefully. 3) During traffic jams, similarly, the possibility of cars cutting in from the sides and motorcycles skidding by necessitates slowing down and driving carefully. Any of 1) to 3) is a scenario where slowing down and driving carefully is unavoidable. Therefore, in Case 1302, situations where a car is traveling at 40 km / h or lower are identified as potential scenarios for jumping objects.

[0081] In other words, when the car's speed is below or equal to a threshold, the situation requiring object detection is determined.

[0082] On the other hand, when the car is traveling at this threshold or higher, the likelihood of a jump-in is expected to decrease, such as on a wide, uncongested road or an unobstructed highway where the speed can be high. When it is determined in process 1302 that the state is not a situation where a jump-in is likely, since no action is needed in any mode, the process moves to 1303 to determine that no action is needed. If it is determined in process 1302 that the state is a situation where a jump-in is likely, the mode of the current situation is obtained in process 1304. In process 1305, it is checked whether the current situation is mode 1. If "yes", it is determined in process 1306 that no action is needed, and the process ends.

[0083] If "No", the process proceeds to process 1307. In process 1307, it is checked whether the current situation is mode 2. If "Yes", it is determined in process 1308 that no action is needed, and the process ends. If "No", the process proceeds to process 1309. In process 1309, it is checked whether the current situation is mode 3. If "Yes", it is determined in process 1310 that no action is needed, and the process ends. If "No", the process proceeds to process 1311. In process 1311, it is checked whether the current situation is mode 4. If "Yes", it is determined in process 1312 that no action is needed, and the process ends. If "No", the process proceeds to process 1313. Since the current situation is mode 5, the ECU is notified that the device is in a malfunctioning state, and the process ends.

[0084] Second Embodiment

[0085] In the first embodiment, when performing the determination of whether a measure is implemented in each mode (regarding...) Figure 12 (Description), which uses only the determination of the vehicle's speed to determine the possible occurrence of an intrusion.

[0086] In the second embodiment, an example of obtaining driving environment information from the ECU to perform determination in a comprehensive manner will be described. In the second embodiment, only the differences from the first embodiment will be described.

[0087] Figure 13 This is an overall system diagram in the second embodiment. It is different from the overall system diagram in the first embodiment. Figure 1 Components similar to those in the first embodiment are assigned the same reference numerals. The difference from the first embodiment is the driving warning area information 1401 arranged in the ECU 130. The driving warning area information refers to a flag (1 bit) indicating a driving warning area, which is stored in the vehicle navigation system (not shown in the figure) and connected to the ECU 130. Since a Global Positioning System (GPS) (not shown) is connected to the vehicle navigation system, the current location can be determined. A configuration is established whereby when the current location enters a driving warning area stored in the vehicle navigation system, the flag is sent as driving warning area information to the image capture timing control unit 118. In the second embodiment, the driving warning area information is set as data stored in the vehicle navigation system but can be updated via OTT (over-the-top) or the like.

[0088] Figure 14 This is an illustrative diagram describing a method for determining whether a measure is implemented in each mode by using both acquired driving warning area information and vehicle speed.

[0089] In process 1514, driving warning area information is obtained from ECU 130. In process 1515, the driving warning area information is checked, and if valid, the process proceeds to 1504 (external situation determination unit). If invalid, the process proceeds to 1501. In process 1501, the vehicle speed is obtained. In process 1502, if the vehicle is traveling at 40 km / h or lower, a situation where an obstacle is likely to appear is determined. If it is determined in process 1502 that the situation is not a situation where an obstacle is likely to appear, since no action is required in any mode, the process proceeds to 1503, and it is determined that no action is required. If it is determined in process 1502 that the situation is a situation where an obstacle is likely to appear, in process 1504, the mode of the current situation is obtained. In process 1505, it is confirmed whether the current situation is mode 1. If "yes", it is determined in process 1506 that no action is required, and the process ends.

[0090] If "No", the process proceeds to process 1507. In process 1507, it is checked whether the current situation is mode 2. If "Yes", it is determined in process 1508 that no action is needed, and the process ends. If "No", the process proceeds to process 1509. In process 1509, it is checked whether the current situation is mode 3. If "Yes", it is determined in process 1510 that no action is needed, and the process ends. If "No", the process proceeds to process 1511. In process 1511, it is checked whether the current situation is mode 4. If "Yes", it is determined in process 1512 that no action is needed, and the process ends. If "No", the process proceeds to process 1513. Since the current situation is mode 5, the ECU is notified that the device is in a functional fault state, and the process ends.

[0091] The present disclosure has been described above by way of embodiments, but the present disclosure is not limited to these specific embodiments, and various modes are included in the present disclosure without departing from the spirit of the present disclosure. Some of the above embodiments may also be appropriately combined.

[0092] Additionally, this disclosure includes situations where a program for software that implements the functions of the above embodiments is supplied directly from a recording medium or via wired / wireless communication to a system or apparatus having a computer capable of executing the program, and the program is executed.

[0093] Therefore, in order to implement the functional processing of this disclosure in a computer or information processing apparatus, the program code itself supplied and installed in the computer and the information processing method also implement this disclosure. In other words, the computer program itself and the information processing method used to implement the functional processing of this disclosure are also included in this disclosure.

[0094] In the above case, as long as the functionality of the program is provided, it does not matter what form the program takes (such as object code, a program to be executed by an interpreter, or script data to be supplied to the OS).

[0095] Examples of recording media used in the supply process may include, for example, hard disk drives, magnetic recording media such as magnetic tape, optical / magneto-optical storage media, and non-volatile semiconductor memories.

[0096] Alternatively, the following method can be conceived as a method of supplying the program: a server on a computer network stores the computer program constituting this disclosure, and a connected client computer downloads the computer program to execute the computer program.

[0097] A device can be provided to reduce the detection cycle of jumping and cutting objects even when the ambient light level is low.

[0098] Other embodiments

[0099] One or more embodiments of the present invention can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transient computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method executed by a computer of a system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the memory of a distributed computing system, an optical disk (such as an optical disc (CD), a digital versatile disc (DVD), or Blu-ray disc (BD)). TM One or more of the following: flash memory devices and memory cards.

[0100] While this disclosure has been described with reference to embodiments, it is to be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims is to be given the broadest interpretation to include all such modifications and equivalent structures and functions.

Claims

1. An image capturing apparatus comprising at least one processor or circuitry configured to function as: a plurality of image capturing units each of which is configured to be capable of reading out a luminance value multiple times at any time point during one frame and to be capable of performing ranging using an image surface phase difference of a pupil division scheme constituted by one imaging optical system and one image capturing element; a first object detection unit configured to detect an object by one of the plurality of image capturing units based on a result of ranging using the image surface phase difference; a second object detection unit configured to detect an object by two of the plurality of image capturing units based on a result of stereoscopic ranging; an object detection determination unit configured to determine whether an object can be detected by the second object detection unit; and an exposure start timing setting unit configured to set an exposure start timing of one of the image capturing units and an exposure start timing of another of the image capturing units so that the exposure start timings are offset from each other in a case where the object detection determination unit determines that an object cannot be detected.

2. The image capturing apparatus according to claim 1, further comprising: a situation determination unit configured to determine whether a situation requires object detection, wherein in a case where the object detection determination unit determines that an object cannot be detected and the situation determination unit determines that the situation requires object detection, the exposure start timing setting unit sets the exposure start timing of one of the image capturing units to be offset from the exposure start timing of another of the image capturing units.

3. The image capturing apparatus according to claim 2, wherein in a case where the object detection determination unit determines that an object cannot be detected or the situation determination unit determines that the situation does not require object detection, the exposure start timing setting unit sets the exposure start timings of a plurality of the image capturing units to be matched.

4. The image capturing apparatus according to claim 1, wherein the first object detection unit is a compound eye camera provided with a sensor and a lens array having a plurality of lenses of which the image capturing units are arranged on the same plane, and the first object detection unit detects an object by outputting a ranging image based on a parallax of a plurality of individual eye images captured by the compound eye camera, and the second object detection unit detects an object by outputting a ranging image based on a parallax of two image capturing images respectively acquired from the two image capturing units.

5. The image capturing apparatus according to claim 1, wherein ​ The exposure start timing setting unit shifts the exposure start timing of the other one of the image capturing units in such a manner that the timing at which the first object detection unit of one of the image capturing units is unable to detect an object, the timing at which the first object detection unit of the other one of the image capturing units is able to detect an object is established.

6. The image capturing apparatus according to claim 1, further comprising: an ambient illuminance detection unit configured to detect an ambient illuminance, wherein the object detection determination unit determines whether an object can be detected by the second object detection unit based on the ambient illuminance.

7. The image capturing apparatus according to claim 2, further comprising: a speed acquisition unit configured to acquire a speed of a moving object on which the image capturing apparatus is mounted, wherein in a case where the speed of the moving object is lower than or equal to a threshold value, the situation determination unit determines that the situation requires object detection.

8. The image capturing apparatus according to claim 2, further comprising: an external situation determination unit configured to determine whether a jump-in or cut-in of a person, an animal, or a moving object is likely to occur in the situation, wherein in a case where the external situation determination unit determines that a jump-in or cut-in of a person, an animal, or a moving object is likely to occur in the situation, the situation determination unit determines that the situation requires object detection.

9. An information processing method executed by an information processing apparatus, the information processing method comprising: a plurality of image capturing processes, each of the plurality of image capturing processes being configured to be able to read out a luminance value multiple times at an arbitrary time point during one frame and to be able to perform ranging using an image plane phase difference of a pupil division scheme constituted by one imaging optical system and one image capturing element; a first object detection process of detecting an object in one of the plurality of image capturing processes based on a result of ranging using the image plane phase difference; a second object detection process of detecting an object in two of the plurality of image capturing processes based on a result of stereoscopic ranging; an object detection determination process of determining whether an object can be detected in the second object detection process; and an exposure start timing setting process of setting an exposure start timing of one of the image capturing processes to be shifted with respect to an exposure start timing of the other one of the image capturing processes in a case where the object detection determination process determines that an object cannot be detected.

10. A storage medium storing a computer program for causing a computer to execute each process of the information processing method according to claim 9. ​

Images