Imaging apparatus

Abstract

To properly set imaging conditions of an imaging apparatus.SOLUTION: The imaging apparatus includes an imaging unit mounted on a vehicle and imaging the exterior of the vehicle, and an imaging control unit for controlling imaging conditions of the imaging unit based on the position of a steering wheel of the vehicle.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Technologies have been developed that detect the vehicle's driving environment based on images acquired by a camera mounted on the vehicle, and perform automatic driving control, such as following the vehicle ahead, and driving assistance, such as warnings, braking, and steering assistance, based on the detected driving environment data (see Patent Document 1). In conventional technology, solid-state imaging elements such as CCDs are used in vehicle-mounted cameras. Vehicle-mounted cameras that continuously capture images of roads and other objects play an important role in automatic driving control and driving assistance, but there have been few proposals for cameras that are intended to be mounted on vehicles, and the usability of the cameras has not been sufficient. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-79424 A Summary of the Invention

[0004] The imaging device according to the present invention is an imaging device mounted on a vehicle, and comprises an imaging section having a plurality of imaging areas for imaging a subject and capable of setting imaging conditions for each of the imaging areas, an input section related to the distance to surrounding objects, and an imaging control section that controls the imaging section, and when information related to the distance to surrounding objects changes, the imaging control section changes the imaging conditions of some of the plurality of imaging areas of the imaging section. [Brief description of the drawings]

[0005] [Figure 1] 1 is a schematic configuration diagram of a vehicle driving assistance device. [Diagram 2] FIG. 2 is a block diagram illustrating a configuration of a control device. [Diagram 3] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 4] 2 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Diagram 5] FIG. 2 is a diagram illustrating a circuit of a unit area. [Figure 6] FIG. 2 is a block diagram showing a functional configuration of an imaging element. [Figure 7] 2 is a diagram illustrating an example of positions of focus detection pixels on an imaging surface of an image sensor; [Figure 8] FIG. 2 is an enlarged view of a region including a portion of a focus detection pixel line. [Figure 9] FIG. 1 is a block diagram illustrating a configuration of a camera having an image sensor. [Figure 10] 2 is a diagram illustrating an imaging surface of an imaging chip, an imaging region, a region of interest, and a pause region. FIG. [Figure 11] 10 is a flowchart illustrating the flow of camera control processing executed by a control unit. [Figure 12] 10 is a flowchart illustrating details of an initial setting process. [Figure 13] FIG. 11 is a diagram illustrating a table of initial setting values. [Figure 14] 2 is a diagram illustrating an imaging surface of an imaging chip, an imaging region, a region of interest, and a pause region. FIG. [Figure 15] 2 is a diagram illustrating an imaging surface of an imaging chip, an imaging region, a region of interest, and a pause region. FIG. [Figure 16] 2 is a diagram illustrating an imaging surface of an imaging chip, an imaging region, a region of interest, and a pause region. FIG. [Figure 17] 10 is a flowchart illustrating details of a driving assistance setting process. [Figure 18] FIG. 13 is a diagram for explaining a flag Em. [Figure 19] FIG. 13 is a diagram for explaining a distance Z. [Figure 20] Fig. 20(a) is a diagram illustrating an example of the movement of the position and the change in size of the attention area when turning right at an intersection on a general road, and Fig. 20(b) is a diagram illustrating an example of the movement of the position and the change in size of the attention area when changing lanes while accelerating on a highway. [Figure 21] 13A and 13B are diagrams illustrating turn signal directions and changes in the size of an attention area. [Figure 22] 10 is a flowchart illustrating a process performed when a turn signal switch is operated according to the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <Camera usage scenarios> Fig. 1 is a schematic configuration diagram of a driving assistance device 2 of a vehicle 1 equipped with a camera 3 according to an embodiment of the present invention. In Fig. 1, the driving assistance device 2 is mounted on a vehicle 1 such as an automobile. The driving assistance device 2 is composed of a camera 3, a control device 4, a first driving control unit 5, a second driving control unit 6, etc. Note that, although an example in which an internal combustion engine is used as the driving source will be described here, a driving source such as a motor may also be used.

[0007] The camera 3 includes an imaging optical system having a plurality of lenses and an imaging element (a stacked imaging element in this embodiment (see FIG. 3)), and is attached, for example, to the front of the ceiling inside the vehicle. The camera 3 is directed toward the front of the vehicle 1, and its mounting height (the distance from the ground to the camera 3) is adjusted to, for example, 1.4 (m). The camera 3 acquires an image in the traveling direction of the vehicle 1, and performs distance measurement (ranging) to each subject (object) at a plurality of positions in the shooting screen based on the acquired image. The distance measurement is calculated by a ranging calculation using an image signal from a focus detection pixel provided in the stacked imaging element. The focus detection pixel and ranging will be described later. The image data and ranging data acquired by the camera 3 are sent to the control device 4. The camera 3 may be provided outside the vehicle, or the cameras 3 inside and outside the vehicle may work together, and the number of cameras 3 may be set appropriately. As an example, the white line detection described later may be performed using the camera 3 outside the vehicle, and the recognition of objects and obstacles may be performed using the cameras 3 inside and outside the vehicle.

[0008] The control device 4 includes a CPU 4a and a storage unit 4b, as exemplified in Fig. 2. The CPU 4a performs various calculations based on various programs stored in the storage unit 4b, using control parameters stored in the storage unit 4b and detection signals from various sensors (described later).

[0009] The first cruise control unit 5 performs constant speed cruise control and follow-up cruise control based on instructions from the control device 4. The constant speed cruise control is a control for making the vehicle 1 travel at a constant speed based on a predetermined control program. The follow-up cruise control is a control for making the vehicle 1 travel while maintaining a constant distance from the preceding vehicle if the speed of the preceding vehicle recognized by the control device 4 is equal to or lower than a target speed set for the vehicle 1 during constant speed cruise control.

[0010] The second driving control unit 6 performs driving assistance control based on instructions from the control device 4. The driving assistance control is control that outputs a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road and outputs a brake control signal to the brake control device 8 so that the vehicle 1 avoids colliding with an object based on a predetermined control program.

[0011] FIG. 1 further illustrates a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, a display device 14, a GPS device 15, a shift lever position detection device 16, and a microphone 17.

[0012] The throttle control device 7 controls the opening of a throttle valve (not shown) in accordance with the depression amount of an accelerator pedal 7a. The throttle control device 7 also controls the opening amount of the throttle valve in accordance with a throttle control signal sent from the first driving control unit 5. The throttle control device 7 further sends a signal indicating the depression amount of the accelerator pedal 7a to the control device 4.

[0013] The brake control device 8 controls the opening of a brake valve (not shown) according to the depression amount of a brake pedal 8a. The brake control device 8 also controls the opening amount of the brake valve according to a brake control signal from the second driving control unit 6. The brake control device 8 further sends a signal indicating the depression amount of the brake pedal 8a to the control device 4.

[0014] The steering control device 9 controls the steering angle of a steering device (not shown) in accordance with the rotation angle of the steering wheel 10. The steering control device 9 also controls the steering angle of the steering device in accordance with a steering control signal from the second driving control unit 6. The steering control device 9 further sends a signal indicating the rotation angle of the steering wheel 10 to each of the first driving control unit 5 and the control device 4.

[0015] The turn signal switch 11 is a switch for activating a turn signal (blinker) device (not shown). The turn signal device is a flashing light-emitting device that indicates a change in lane of the vehicle 1. When the turn signal switch 11 is operated by an occupant of the vehicle 1, an operation signal from the turn signal switch 11 is sent to the turn signal device, the second driving control unit 6, and the control device 4, respectively. The vehicle speed sensor 12 detects the vehicle speed V of the vehicle 1, and sends a detection signal to the first driving control unit 5, the second driving control unit 6, and the control device 4, respectively.

[0016] The yaw rate sensor 13 detects the yaw rate of the vehicle 1 and sends a detection signal to the second driving control unit 6 and the control device 4, respectively. The yaw rate is the rate of change of the rotation angle in the turning direction of the vehicle 1. The display device 14 displays information indicating the control state by the first driving control unit 5 and the second driving control unit 6, and the like. The display device 14 is configured, for example, by a HUD (Head Up Display) that projects information onto the windshield. Note that the display device 14 may be a display unit of a navigation device (not shown).

[0017] The GPS device 15 receives radio waves from GPS satellites and performs predetermined calculations using the information carried in the radio waves to calculate the position (latitude, longitude, etc.) of the vehicle 1. The position information calculated by the GPS device 15 is sent to a navigation device (not shown) and the control device 4. The shift lever position detection device 16 detects the position (e.g., parking (P), reverse (R), drive (D), etc.) of a shift lever (not shown) operated by an occupant of the vehicle 1. The shift lever position information detected by the shift lever position detection device 16 is sent to the control device 4.

[0018] The microphone 17 is composed of, for example, a front microphone, a right side microphone, and a left side microphone. The front microphone has directivity that mainly collects sounds in front of the vehicle 1. The right side microphone has directivity that mainly collects sounds on the right side of the vehicle 1. The left side microphone has directivity that mainly collects sounds on the left side of the vehicle 1. Each piece of sound information (forward, right side, left side) collected by the microphone 17 is sent to the control device 4.

[0019] <Detection of target object> In order to detect the roadway along which the vehicle 1 is traveling and objects, the control device 4 performs image processing on the image from the camera 3 as follows. First, the control device 4 generates a distance image (depth distribution image) based on distance measurement data at multiple positions within the shooting screen. The control device 4 performs a well-known grouping process based on the distance image data, and compares it with a frame (window) of three-dimensional road shape data, sidewall data, object data, etc. stored in advance in the storage unit 4b, extracts white line data (including white line data along the road and white lines (stop lines: intersection information) data crossing the road), sidewall data of guardrails and curbs existing along the road, and classifies and extracts objects / obstacles into other objects such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, etc. In this description, the term "white line" refers to a white or yellow line drawn on the roadway. The term "white line" also includes solid and broken lines.

[0020] <Driving assistance> The control device 4 recognizes the driving path and objects / obstacles that may become obstacles based on each piece of information extracted as described above, i.e., the white line data, the guardrail side wall data, and the object data, and causes the second driving control unit 6 to perform the driving assistance control based on the recognition result. That is, the control device 4 causes the vehicle 1 to travel along the road and prevents the vehicle 1 from colliding with objects.

[0021] <Drive control> The control device 4 estimates the vehicle's traveling path, for example, in the following four ways. (1) Vehicle path estimation based on white lines The image captured by the camera 3 provides data on white lines on both the left and right sides of the roadway, or on either the left or right side. If the shape of the lane in which the vehicle 1 is traveling can be estimated from this white line data, the control device 4 estimates that the vehicle's path is parallel to the white lines, taking into account the width of the vehicle 1 and its current position within the lane.

[0022] (2) Vehicle path estimation based on sidewall data such as guardrails and curbs When sidewall data for both the left and right sides of the roadway or either the left or right side is obtained from the image captured by the camera 3, and the shape of the lane in which the vehicle 1 is traveling can be estimated from this sidewall data, the control device 4 estimates that the vehicle's travel path is parallel to the sidewall, taking into account the width of the vehicle 1 and the vehicle 1's current position within the lane.

[0023] (3) Estimating vehicle path based on preceding vehicle trajectory The control device 4 estimates the vehicle travel path based on the past travel path of the preceding vehicle stored in the memory unit 4b. The preceding vehicle refers to the vehicle that is closest to the vehicle 1 among objects traveling in the same direction as the vehicle 1.

[0024] (4) Estimation of vehicle path based on vehicle 1’s driving trajectory The control device 4 estimates the vehicle's traveling path based on the driving state of the vehicle 1. For example, the control device 4 estimates the vehicle's traveling path using a turning curvature based on a detection signal from the yaw rate sensor 13 and a detection signal from the vehicle speed sensor 12. The turning curvature Cua is calculated by Cua = dψ / dt / V, where dψ / dt is the yaw rate (the rate of change of the rotation angle in the turning direction), and V is the vehicle speed of the vehicle 1.

[0025] The control device 4 estimates, for each of the above objects, the driving area of ​​the vehicle 1 at the position where the object exists based on the vehicle travel path in accordance with a predetermined driving control program stored in the memory unit 4b, and compares this driving area with the object position to determine whether each object is within the driving area. The control device 4 further recognizes the preceding vehicle based on the image pickup result of the camera 3. That is, the control device 4 determines the vehicle closest to the vehicle 1 as the preceding vehicle among the objects that exist within the driving area and travel in the forward direction (the same direction as the vehicle 1).

[0026] The control device 4 outputs information on the distance between the preceding vehicle and the vehicle 1, and information on the speed of the preceding vehicle, as outside-vehicle information, to the first cruise control unit 5. Here, the information on the speed of the preceding vehicle is calculated based on the vehicle speed V of the vehicle 1 acquired at predetermined time intervals and the change in the distance (inter-vehicle distance) to the preceding vehicle in the captured image, which is measured based on images acquired by the camera 3 at the above-mentioned predetermined time intervals in synchronization with the timing of acquiring the vehicle speed V.

[0027] The first driving control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a predetermined vehicle speed (target speed) that has been set in advance. As a result, the throttle control device 7 feedback controls the opening of a throttle valve (not shown) to automatically drive the vehicle 1 at a constant speed.

[0028] Furthermore, when the vehicle speed information of the preceding vehicle input from the control device 4 during constant speed driving control is equal to or lower than the target speed set for the vehicle 1, the first driving control unit 5 sends a throttle control signal to the throttle control device 7 based on the vehicle distance information input from the control device 4. Specifically, the first driving control unit 5 sets an appropriate target value for the vehicle distance based on the vehicle distance from the vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of the vehicle 1, and sends a throttle control signal to the throttle control device 7 so that the vehicle distance measured based on the image acquired by the camera 3 converges to the target value for the vehicle distance. As a result, the throttle control device 7 feedback controls the opening of a throttle valve (not shown) to make the vehicle 1 follow the preceding vehicle.

[0029] <Explanation of stacked image sensor> The stacked imaging element 100 provided in the above-mentioned camera 3 will be described. This stacked imaging element 100 is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. FIG. 3 is a cross-sectional view of the stacked imaging element 100. The imaging element 100 includes a back-illuminated imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked, and are electrically connected to each other by conductive bumps 109 such as Cu.

[0030] As shown in the figure, incident light is mainly incident in the Z-axis positive direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface (imaging surface). As shown in the coordinate axes, the left direction on the paper surface perpendicular to the Z-axis is the X-axis positive direction, and the front direction on the paper surface perpendicular to the Z-axis and the X-axis is the Y-axis positive direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 3.

[0031] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 includes a plurality of PDs (photodiodes) 104 that are two-dimensionally disposed and accumulate electric charges according to incident light, and transistors 105 that correspond to the PDs 104.

[0032] A color filter 102 is provided on the incident side of the PD layer 106 where the incident light is incident, via a passivation film 103. There are a plurality of types of color filters 102 that transmit different wavelength regions, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

[0033] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0034] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may include passive and active elements.

[0035] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed, etc., so that the aligned bumps 109 are joined and electrically connected to each other.

[0036] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, etc., so that the aligned bumps 109 are bonded to each other and electrically connected.

[0037] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may be adopted. Also, it is sufficient to provide about one bump 109 for each block described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in a peripheral region other than the pixel region where the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided.

[0038] The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0039] FIG. 4 is a diagram for explaining the pixel array of the imaging chip 113 and the unit area 131. In particular, the imaging chip 113 is shown as observed from the back side (imaging surface) side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In the example of FIG. 4, adjacent 4 pixels x 4 pixels, or 16 pixels, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be around 1000, for example 32 pixels x 64 pixels, or it may be more or less than that.

[0040] As shown in the partially enlarged view of the pixel region, the unit region 131 in Fig. 4 contains four so-called Bayer arrays, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R, arranged vertically and horizontally. The green pixels Gb and Gr are pixels having a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel having a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter 102 and receives light in the red wavelength band.

[0041] In this embodiment, a plurality of blocks are defined so that each block includes at least one unit region 131, and each block can control the pixels included in each block with different control parameters. In other words, imaging signals with different imaging conditions can be acquired for a pixel group included in one block and a pixel group included in another block. Examples of the control parameters are frame rate, gain, thinning rate, number of summation rows or number of summation columns for adding pixel signals, charge accumulation time or number of accumulations, number of digitization bits (word length), etc. The imaging element 100 can freely perform thinning not only in the row direction (X-axis direction of the imaging chip 113) but also in the column direction (Y-axis direction of the imaging chip 113). Furthermore, the control parameters may be parameters for image processing after image signals are acquired from pixels.

[0042] Fig. 5 is a diagram for explaining a circuit in a unit area 131. In the example of Fig. 5, one unit area 131 is formed by 9 pixels, 3 pixels x 3 pixels adjacent to each other. As described above, the number of pixels included in the unit area 131 is not limited to this, and may be more or less than this. The two-dimensional positions of the unit area 131 are indicated by symbols A to I.

[0043] The reset transistors of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 5, a reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 310 to turn on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 to turn on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. Dedicated reset wiring is also provided for the other pixels D to I to turn on and off their respective reset transistors.

[0044] The transfer transistors of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 5, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. Dedicated transfer wiring is also provided for the other pixels D to I to turn on and off the respective transfer transistors.

[0045] Furthermore, the selection transistors of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 5, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. Dedicated selection wiring is also provided for the other pixels D to I to turn on and off their respective selection transistors.

[0046] The power supply wiring 304 is commonly connected from pixel A to pixel I included in the unit area 131. Similarly, the output wiring 308 is commonly connected from pixel A to pixel I included in the unit area 131. The power supply wiring 304 is commonly connected between the multiple unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0047] By individually turning on and off the reset transistors and transfer transistors of the unit area 131, it is possible to control charge accumulation including the accumulation start time, accumulation end time, and transfer timing independently for pixels A to I included in the unit area 131. In addition, by individually turning on and off the selection transistors of the unit area 131, it is possible to output pixel signals of each of pixels A to I via a common output wiring 308.

[0048] Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order for rows and columns for pixels A to I included in the unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, pixel signals are output in the order of "ABCDEFGHI" in the example of Fig. 5.

[0049] In this way, by configuring the circuit based on the unit area 131, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output pixel signals at different frame rates between the unit areas 131. Also, by resting the unit areas 131 included in a part of the imaging chip 113 while causing the unit areas 131 included in other areas to accumulate charge (imaging), it is possible to cause imaging to be performed only in a predetermined area of ​​the imaging chip 113 and output the pixel signals. Furthermore, it is also possible to switch the area (target area for accumulation control) in which charge accumulation (imaging) is performed between frames, and cause imaging to be performed sequentially in different areas of the imaging chip 113 and output pixel signals.

[0050] Fig. 6 is a block diagram showing a functional configuration of the image sensor 100 corresponding to the circuit illustrated in Fig. 5. An analog multiplexer 411 sequentially selects nine PDs 104 forming a unit area 131 and outputs the respective pixel signals to output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed in the image sensor chip 113 together with the PDs 104.

[0051] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 that performs CDS and A / D conversion and is formed in the signal processing chip 111. The A / D converted pixel signals are passed to a demultiplexer 413 and stored in pixel memories 414 corresponding to the respective pixels. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 112.

[0052] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that although Fig. 6 shows the connections for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 415 for each unit area 131. For example, one arithmetic circuit 415 may process sequentially while referring to values ​​of the pixel memories 414 corresponding to each unit area 131 in order.

[0053] As described above, output wiring 308 is provided corresponding to each unit area 131. Since image pickup element 100 has imaging chip 113, signal processing chip 111, and memory chip 112 stacked on top of each other, by using bumps 109 for electrical connection between the chips for output wiring 308, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0054] <Explanation of distance measurement> FIG. 7 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 100. In this embodiment, the focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of FIG. 7, 15 focus detection pixel lines 60 are arranged at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 60 output image signals for distance measurement. In the imaging chip 113, normal imaging pixels are provided at pixel positions other than the focus detection pixel lines 60. The imaging pixels output image signals for monitoring the outside of the vehicle.

[0055] Fig. 8 is an enlarged view of a region including a portion of one of the focus detection pixel lines 60. Fig. 8 illustrates a red pixel R, a green pixel G (Gb, Gr), a blue pixel B, a focus detection pixel S1, and a focus detection pixel S2. The red pixel R, the green pixel G (Gb, Gr), and the blue pixel B are arranged according to the Bayer array rules described above.

[0056] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B indicate the light receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system 31 (FIG. 9). That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and light that passes through these mask openings reaches the light receiving section of the imaging pixel.

[0057] The shape of the light receiving regions (mask openings) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0058] The semicircular regions illustrated for the focus detection pixels S1 and S2 indicate the light receiving regions of the focus detection pixels. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in Fig. 8, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in Fig. 8, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S2. In this way, the focus detection pixels S1 and S2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system 31 (Fig. 9).

[0059] The positions of the focus detection pixel lines in the imaging chip 113 are not limited to the positions exemplified in Fig. 7. The number of focus detection pixel lines is also not limited to the example in Fig. 7. Furthermore, the shape of the mask openings in the focus detection pixels S1 and S2 is not limited to a semicircle, and may be, for example, a rectangular shape obtained by horizontally dividing the square-shaped light receiving regions (mask openings) in the imaging pixels R, G, and B.

[0060] Furthermore, the focus detection pixel line in the imaging chip 113 may be an arrangement of focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 113. An imaging element in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Fig. 8 is well known, and detailed illustration and description of these pixels will be omitted. In the example of Fig. 8, a configuration in which each of the focus detection pixels S1 and S2 receives one of a pair of light beams for focus detection, a so-called 1PD structure, has been described. Alternatively, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2007-282107, a configuration in which each of the focus detection pixels receives both of a pair of light beams for focus detection, a so-called 2PD structure, may be used. By using such a 2PD structure, it becomes possible to read image data from the focus detection pixels, and the focus detection pixels do not become defective pixels.

[0061] In this embodiment, the focus adjustment state (defocus amount) of the imaging optical system 31 is calculated by detecting the amount of image shift (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (Figure 9) based on the distance measurement image signals output from the focus detection pixel S1 and the focus detection pixel S2.

[0062] In general, the pair of images approach each other in a so-called front focus state in which the imaging optical system 31 forms a sharp image of an object (e.g., a preceding vehicle) in front of the intended focal plane, and move away from each other in a so-called back focus state in which the imaging optical system 31 forms a sharp image of the object behind the intended focal plane. In a focused state in which a sharp image of the object is formed on the intended focal plane, the pair of images coincide relatively. Therefore, the amount of relative positional deviation between the pair of images corresponds to the distance to the object (depth information).

[0063] The defocus amount calculation based on the phase difference is well known in the field of cameras, and detailed description will be omitted. Here, since the defocus amount and the distance to the object correspond one-to-one, the distance from the camera 3 to each object can be obtained by calculating the defocus amount for each object. In other words, the distance to each object can be measured (distance measurement) at multiple positions on the shooting screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a formula or a lookup table, and stored in the non-volatile memory 35b (FIG. 9).

[0064] <Camera description> Fig. 9 is a block diagram illustrating the configuration of a camera 3 having the above-mentioned image sensor 100. In Fig. 9, the camera 3 has an image capturing optical system 31, an image capturing unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36.

[0065] The imaging optical system 31 guides a light beam from the subject field to the imaging section 32. The imaging section 32 includes the imaging element 100 and a driving section 32a, and photoelectrically converts the image of the object formed on the imaging chip 113 by the imaging optical system 31. The driving section 32a generates a driving signal required for the imaging element 100 (imaging chip 113) to perform independent accumulation control in the above-mentioned block units. Instructions such as the position and shape of the above-mentioned blocks, their ranges, and accumulation times are transmitted from the control section 35 to the driving section 32a.

[0066] The image processing unit 33 cooperates with the work memory 34 to perform image processing on the image data captured by the imaging unit 32. The image processing unit 33 performs image processing such as contour enhancement and gamma correction, as well as color detection of objects included in the image.

[0067] The work memory 34 temporarily stores image data before and after image processing. The recording unit 36 ​​records image data in a storage medium such as a non-volatile memory. The control unit 35 is configured by, for example, a CPU, and controls the overall operation of the camera 3 in response to a control signal from the control device 4. For example, the control unit 35 performs a predetermined exposure calculation based on an image signal captured by the imaging unit 32, and instructs the driving unit 32a on the accumulation time of the imaging chip 113 required for proper exposure.

[0068] The control unit 35 includes a distance measurement calculation unit 35a and a non-volatile memory 35b. The distance measurement calculation unit 35a measures the distance (distance) to the object at each of the multiple positions on the shooting screen as described above. Image data acquired by the camera 3 and distance measurement data calculated by the camera 3 are sent to the control device 4 (FIG. 1). The non-volatile memory 35b stores the program executed by the control unit 35a and information required for distance measurement.

[0069] <Image sensor block control> The control device 4 controls the image pickup device 100 (imaging chip 113) of the camera 3 to perform the above-mentioned independent accumulation control in block units. For this purpose, the following signals are input to the control device 4 from each part of the vehicle 1 (FIG. 2). (1) Depression amount of accelerator pedal 7a A signal indicating the depression amount of an accelerator pedal 7a is input from the throttle control device 7 to the control device 4. (2) Depression amount of the brake pedal 8a A signal indicating the amount of depression of a brake pedal 8a is input from the brake control device 8 to the control device 4.

[0070] (3) Rotation angle of the steering wheel 10 A signal indicating the rotation angle of a steering wheel 10 is input from the steering control device 9 to the control device 4. The ratio between the rotation angle of the steering wheel 10 and the steering angle of the steering device depends on the steering gear ratio. (4) Vehicle 1's speed V A detection signal from the vehicle speed sensor 12 is input to the control device 4 .

[0071] (5) Operation signal of turn signal switch 11 An operation signal of the turn signal switch 11 is input to the control device 4. (6) Shift lever operating position A signal indicating the operational position of the shift lever detected by the shift lever position detector 16 is input to the controller 4.

[0072] (7) Vehicle 1 location information Position information measured by the GPS device 15 is input from the GPS device 15 to the control device 4 . (8) Sound information around vehicle 1 Sound information from the front, right side, and left side of the vehicle 1 collected by the microphone 17 is input to the control device 4.

[0073] 10 is a diagram illustrating the imaging surface of the imaging chip 113, regions (imaging region 81 and attention region 82) in which charge accumulation (imaging) is performed in the imaging chip 113, and regions (pause region 83) in which charge accumulation (imaging) is not performed in the row and column directions in the imaging chip 113. The attention region 82 is a region in which charge accumulation (imaging) is performed under conditions different from those of the imaging region 81. The sizes and positions of the imaging region 81 and attention region 82 in the imaging chip 113 are also one of the imaging conditions.

[0074] The control device 4 controls the unit areas 131 included in the imaging area 81 to be imaged by setting first conditions, and controls the unit areas 131 included in the attention area 82 to be imaged by setting second conditions. The control device 4 also pauses the imaging of the unit areas 131 included in the pause area 83. Note that a plurality of attention areas 82 may be provided, and the imaging conditions may be different between the plurality of attention areas. Also, the pause area 83 may not be provided.

[0075] <Explanation of the flow chart> Hereinafter, a method for determining the imaging area 81 and the attention area 82 will be described with reference to flowcharts (FIGS. 11, 12, and 17). FIG. 11 is a flowchart illustrating the flow of control processing of the camera 3 executed by the control device 4. A program for executing the processing according to the flowchart of FIG. 11 is stored in the storage unit 4b of the control device 4. For example, when power supply from the vehicle 1 is started or the engine starts, the control device 4 starts the program for performing the processing according to FIG. 11.

[0076] 11, the control device 4 judges whether or not flag a = 0. Flag a is a flag that is set to 1 when initial setting has been completed and is set to 0 when initial setting has not been completed. If flag a = 0, the control device 4 makes an affirmative judgment in step S10 and proceeds to step S20, and if flag a ≠ 0, the control device 4 makes a negative judgment in step S10 and proceeds to step S30.

[0077] In step S20, the control device 4 performs an initial setting process and proceeds to step S30. The details of the initial setting process will be described later. In step S30, the control device 4 performs a driving assist setting process and proceeds to step S40. In the driving assist setting process, an imaging area 81 and a focus area 82 are determined for the imaging element 100. The details of the driving assist setting process will be described later.

[0078] In step S40, the control device 4 sends an instruction to the camera 3 to drive the imaging area 81 and the attention area 82 in the image sensor 100 under predetermined conditions, respectively, to obtain an image. In this embodiment, for example, when the vehicle speed V increases from 0, the control device 4 sets a higher frame rate, a higher gain, a lower thinning rate, and a shorter accumulation time for the attention area 82 compared to the imaging area 81. As a result, the camera 3 captures an image, and distance measurements are performed at multiple positions on the shooting screen as described above. It is not necessary to make all of the frame rate, gain, thinning rate, accumulation time, etc. different between the imaging region 81 and the attention region 82, and it is sufficient to make at least one of them different. It is also possible for the control device 4 to set the attention region 82 not to perform thinning.

[0079] In step S45, the control device 4 acquires image data and distance measurement data from the camera 3, and proceeds to step S50. In step S50, the control device 4 determines whether or not a setting for displaying information has been made. If a display setting has been made, the control device 4 makes an affirmative decision in step 50 and proceeds to step S60. If a display setting has not been made, the control device 4 makes a negative decision in step 50 and proceeds to step S70.

[0080] In step S60, the control device 4 sends display information to the display device 14 (FIG. 1), and proceeds to step S70. The display information is information according to the state of the vehicle 1 determined in the driving assistance setting process (S30), and causes the display device 14 to display messages such as "Stopping," "Making an emergency stop," "Turning right," and "Turning left." Instead of or together with transmitting the display information, an audio signal for reproducing the above message may be transmitted to an audio reproduction device (not shown). In this case, an audio device of a navigation device (not shown) may be used as the audio reproduction device (not shown).

[0081] In step S70, the control device 4 judges whether or not an off operation has been performed. For example, when the control device 4 receives an off signal (e.g., an engine off signal) from the vehicle 1, the control device 4 judges step S70 to be positive, performs a predetermined off process, and ends the process in FIG. 11. For example, when the control device 4 does not receive an off signal from the vehicle 1, the control device 4 judges step S70 to be negative, and proceeds to step S80. In step S80, the control device 4 waits for a predetermined time (e.g., 0.1 seconds) and returns to step S30. When returning to step S30, the above-mentioned process is repeated.

[0082] <Initial setting process> FIG. 12 is a flowchart for explaining the details of the initial setting process in step S20 in the flowchart in FIG. 11. In step S21 in FIG. 12, the control device 4 inputs the position information of the vehicle 1 from the GPS device 15 (FIG. 1) and proceeds to step S22. In step S22, the control device 4 sets a flag indicating whether the lane in which the vehicle 1 runs is on the left or right side of the road, i.e., whether driving on the left side or the right side, based on the latitude and longitude included in the position information. Specifically, the name of the country in which the vehicle 1 is used is determined based on the latitude and longitude. Then, a flag indicating whether driving on the left side or the right side of the road in the country is set by referring to a database (not shown). The database indicating the relationship between the country name and the left and right of the lane is stored in advance in the storage unit 4b.

[0083] In step S23, the control device 4 sets a flag indicating the mounting position (right or left) of the handle (steering wheel 10) in the vehicle 1, and proceeds to step S24. Information indicating whether the handle is right-hand or left-hand is stored in advance in the storage unit 4b as specification information of the vehicle 1. In step S24, the control device 4 determines initial setting values ​​based on the table exemplified in Fig. 13, and proceeds to step S25. Note that the order of steps S21 and S23 may be reversed.

[0084] 13, four initial setting values ​​from "1" to "4" are prepared according to the combination of the mounting position (right or left) of the steering wheel 10 on the vehicle 1 and the position of the lane on the road (right or left). In the case of right-hand drive and left-hand traffic, the initial setting value is "4".

[0085] In step S25, the control device 4 sets the initial position of the attention area 82. The initial position of the attention area 82 is set to a position according to the initial setting value. Specifically, when the initial setting value is "1", the initial position of the attention area 82 is set to (Xq1, Yq), when the initial setting value is "2", the initial position of the attention area 82 is set to (Xq2, Yq), when the initial setting value is "3", the initial position of the attention area 82 is set to (Xq3, Yq), and when the initial setting value is "4", the initial position of the attention area 82 is set to (Xq4, Yq).

[0086] In this description, the position of the attention area 82 is represented by the coordinates (Xq, Yq) of the center of the attention area 82 in the coordinate system representing the imaging area 81. Fig. 10 illustrates the attention area 82 when the initial setting value is "4", and since the car has a right-hand drive and drives on the left side of the road, the initial position (Xq4, Yq) is determined so that the attention area 82 is set on the driver's seat side (to the right) in the left traffic lane.

[0087] FIG. 14 shows an example of the attention area 82 when the initial setting value is "1." Since the vehicle has a left-hand drive and drives on the right side of the road, the initial position (Xq1, Yq) is determined so that the attention area 82 is set on the driver's side (left side) in the right-hand traffic lane.

[0088] FIG. 15 shows an example of the attention area 82 when the initial setting value is "3." Since the car has a left-hand drive and drives on the left side of the road, the initial position (Xq3, Yq) is determined so that the attention area 82 is set on the driver's side (left side) in the left-hand traffic lane.

[0089] FIG. 16 shows an example of the attention area 82 when the initial setting value is "2." Since the car has a right-hand drive and drives on the right side of the road, the initial position (Xq2, Yq) is determined so that the attention area 82 is set on the driver's side (to the right) in the right-hand traffic lane.

[0090] In step S26 of FIG. 12, the control device 4 sets the initial size of the attention area 82. In this embodiment, the initial size (Px (X-axis direction) × Py (Y-axis direction)) of the attention area 82 is determined based on the size (dimension) of the target object (e.g., a preceding vehicle). When a preceding vehicle is included in the image acquired by the camera 3, the control device 4 estimates the size (dimension) of the preceding vehicle based on the image height of the preceding vehicle captured by the imaging chip 113, the known focal length of the imaging optical system 31, and the distance L from the vehicle 1 to the preceding vehicle obtained by distance measurement. Then, the number of pixels (Px (X-axis direction) × Py (Y-axis direction)) constituting the image obtained on the imaging chip 113 when a preceding vehicle of the estimated size (e.g., width 3 (m) × height 1.4 (m)) is captured from 1 (m) behind is set as the initial size.

[0091] Px and Py are calculated using the following formulas (1) and (2). Px = ox × L … (1) Py = oy × L … (2) Here, ox is the number of pixels in the X-axis direction constituting the image of the preceding vehicle captured by the imaging chip 113 at a distance of L (m), oy is the number of pixels in the Y-axis direction constituting the image of the preceding vehicle captured by the imaging chip 113 at a distance of L (m), and L is the distance from vehicle 1 to the preceding vehicle. In addition, in the coordinate system representing the imaging area 81, the Yq representing the initial position corresponds to the height center of the image obtained on the imaging chip 113 when imaging the preceding vehicle 1 (m) away (in this example, the part at a height of 0.7 (m) of the preceding vehicle).

[0092] In step S27, the control device 4 sends display information to the display device 14 (FIG. 1), sets flag a to 1, and ends the process in FIG. 12. The display information is information indicating that the initial setting process has ended, and causes the display device 14 to display a message such as "Initial setting is now complete."

[0093] <Driving assistance setting process> Fig. 17 is a flowchart for explaining the details of the driving assist setting process. In step S310 in Fig. 17, if the shift lever position information input from the shift lever position detection device 16 (Fig. 1) is "P" (parking), the control device 4 makes an affirmative judgment in step S310 and proceeds to step S320. If the shift lever position information input from the shift lever position detection device 16 (Fig. 1) is not "P", the control device 4 makes a negative judgment in step S310 and proceeds to step S420. The judgment in step S310 may also be applied when the shift lever is in "N" (neutral).

[0094] In step S320, the control device 4 inputs the vehicle speed V from the vehicle speed sensor 12, and proceeds to step S330. The control device 4 changes the frame rate of the attention area 82 according to, for example, the vehicle speed V. As described above, when setting the frame rate of the attention area 82 higher than the frame rate of the imaging area 81, the control device 4 sets the frame rate of the attention area 82 higher as the vehicle speed V increases, and sets the frame rate of the attention area 82 lower as the vehicle speed V decreases. In this case, the control device 4 may apply control such that the frame rate of the imaging area 81 other than the attention area 82 is also proportional to the vehicle speed V. In step S330, the control device 4 inputs the depression amount of the brake pedal 8a from the brake control device 8 (FIG. 1), and proceeds to step S340.

[0095] In step S340, the control device 4 judges whether or not the flag Em=0 based on the vehicle speed V and the depression amount (depression angle) of the brake pedal 8a. The flag Em is a flag that is set as illustrated in FIG. 18 based on the vehicle speed V and the amount of change in the depression amount (depression angle) of the brake pedal 8a. In this embodiment, the case of Em=1 is judged as emergency braking (sudden braking), and the case of Em=0 is judged as normal braking. If Em=0, the control device 4 makes an affirmative judgment in step S340 and proceeds to step S350. If Em=1, the control device 4 makes a negative judgment in step S340 and proceeds to step S430. Incidentally, Em=1 may be determined based on the amount of change in the opening of a brake valve (not shown) instead of the amount of change in depression (depression angle) of the brake pedal 8a. Also, Em=1 may be determined based on the amount of change in the vehicle speed V, or based on the amount of change in the reduction ratio of a transmission (not shown).

[0096] In step S350, the control device 4 inputs the depression amount of the accelerator pedal 7a from the throttle control device 7 (FIG. 1), and proceeds to step S360. In step S360, the control device 4 inputs the rotation angle θ of the steering wheel 10 from the steering control device 9, and proceeds to step S370. In step S370, the control device 4 determines whether or not a steering operation has been performed. If the rotation angle θ is greater than a predetermined value, the control device 4 makes an affirmative decision in step S370 and proceeds to step S380, and if the rotation angle θ is equal to or less than the predetermined value, the control device 4 makes a negative decision in step S370 and proceeds to step S440.

[0097] In step S380, the control device 4 calculates the amount of movement Xdist of the attention area 82 in the X-axis direction based on the rotation angle θ of the steering wheel 10 and the vehicle speed V, using the following equation (3). Xdist = θ × (V × 0.2) … (3) According to the above formula (3), the larger the steering angle of the steering device (that is, the rotation angle θ of the steering wheel 10) is, and the higher the vehicle speed V is, the larger the movement amount Xdist is.

[0098] In step S390, the control device 4 calculates the position (X coordinate) of the attention area 82 during traveling based on the initial position (XqN, Yq) of the attention area 82 set in the initial setting process, using the following equation (4). Xq = XqN + Xdist …(4) Here, N is one of the initial setting values ​​1 to 4 determined in the initial setting process. Xdist is the amount of movement of attention area 82 in the X-axis direction calculated in step S380, and corresponds to the number of pixels in the X-axis direction. By the process of step S390, the position of attention area 82 changes in response to the steering operation. The position of attention area 82 also changes depending on the magnitude of vehicle speed V.

[0099] In step S400, the control device 4 calculates the position (Y coordinate) of the attention area 82 during traveling based on the initial position (XqN, Yq) of the attention area 82 set in the initial setting process, using the following equation (5). Yq = Yq + P(Z) …(5) Here, P(Z) is the amount of movement of the attention area 82 in the Y-axis direction, and is the number of pixels in the Y-axis direction corresponding to the depth Z(m). For example, it represents how many pixels in the Y-axis direction an image of a road with a depth of 20(m) corresponds to. The relationship P(Z) between the depth Z and the number of pixels is stored in advance in the storage unit 4b (FIG. 2).

[0100] In general, when capturing an image of the traveling direction on a flat straight road, the number of pixels in the Y-axis direction corresponding to the image of the road on the imaging chip 113 increases as the depth Z (m) from the vehicle 1 increases. Therefore, the value of Yq, which corresponds to the height center of the image when capturing an image of a preceding vehicle 1 (m) away, is increased as the noteworthy preceding vehicle becomes farther away (i.e., the depth Z becomes deeper).

[0101] The control device 4 determines the depth Z of the preceding vehicle that should be noted by the following equation (6). Z =Za+Zb …(6) where Za is the braking distance (m) on a dry road, and Zb is the braking distance (m) on a wet road. Za and Zb are based on the values ​​exemplified in FIG. 19. In this embodiment, the position of the attention area 82 is determined so that the attention area 82 includes a preceding vehicle at a depth Z in front of the vehicle 1 (i.e., a position Z (m) away from the vehicle 1). This is based on the idea of ​​focusing on an area farther away than the distance required to stop the vehicle in the event of emergency braking. The value of the depth Z (Za+Zb) according to the vehicle speed V is stored in advance in the memory unit 4b (FIG. 2). According to the process of step S400, the position of the attention area 82 changes according to changes in the vehicle speed V. For the region of interest 82 whose position has been changed in this way, at least one of the frame rate, gain, thinning rate, accumulation time, etc. is made different between the imaging region 81 and the region of interest 82 .

[0102] In step S410, the control device 4 calculates the size (X_wid, Y_wid) of the attention area 82 during driving based on the initial size (Px × Py) of the attention area 82 set in the initial setting process, using the following equations (7) and (8), and terminates the processing in Figure 17. X_wid = Px / Z … (7) Y_wid = Py / Z … (8) where Px is the number of pixels in the X-axis direction set in step S26, and Py is the number of pixels in the Y-axis direction set in step S26. According to the above formulas (7) and (8), the size (X_wid, Y_wid) of attention area 82 while driving becomes smaller than the initial size (Px×Py) of attention area 82 as the attention-required preceding vehicle becomes farther away (depth Z becomes deeper). According to the process of step S410, the size of attention area 82 changes in response to changes in vehicle speed V. For the region of interest 82 whose size has been changed in this manner, at least one of the frame rate, gain, thinning rate, accumulation time, etc. is made different between the imaging region 81 and the region of interest 82 .

[0103] In step S420, which is reached after making a negative decision in step S310 described above, the control device 4 performs setting processing during a stop, and ends the processing in Fig. 17. The setting processing during a stop determines the position of the attention area 82 so that a preceding vehicle that is 1 (m) away is included in the attention area 82, for example. Also, the size of the attention area 82 in the X-axis direction is maximized so that objects located close to the sides of the vehicle 1 are included in the attention area 82 as much as possible.

[0104] In step S430, which is reached after a negative decision is made in step S340 described above, the control device 4 performs setting processing for sudden braking determination and ends the processing in FIG. 17. The setting processing for sudden braking determination, for example, stops thinning out in the attention area 82, increases the frame rate to the maximum, shortens the accumulation time, and sets a high gain. Note that the control device 4 may also increase the frame rate of the imaging area 81 other than the attention area 82. Furthermore, the control device 4 issues a recording instruction to the camera 3 so that the images captured by the camera 3 are stored in the recording unit 36 ​​for a predetermined time (for example, 5 to 15 seconds) after the negative decision is made in step S340.

[0105] After the sudden stop, the control device 4 further moves the attention area 82 to the initial position of the attention area 82 set in the initial setting process (step S25 in FIG. 12), and changes the size of the attention area 82 to the initial size (Px×Py) of the attention area 82 set in the initial setting process (step S26 in FIG. 12). As a result, the position and size of the attention area 82, which have changed due to the vehicle speed V while traveling, are returned to a position and size appropriate for when the vehicle is stopped.

[0106] In step S440, which is reached after making a negative decision in step S370 described above, the control device 4 performs settings so that the position (X coordinate) of the attention area 82 during driving is not moved. That is, when the rotation angle θ of the steering wheel 10 is equal to or less than a predetermined value, θ←0 is set, and the value of Xdist is also set to 0. In other words, when the operation angle of the steering wheel 10 does not reach a predetermined value, the position (X coordinate) of the attention area 82 is maintained. This helps reduce the processing load during small operations that are not turning operations.

[0107] 20(a) is a diagram illustrating the movement of the position of attention area 82 and the change in size of attention area 82 when turning right at an intersection on a general road. According to the driving assist setting process, when vehicle 1 is waiting to turn right behind a preceding vehicle, the position of attention area 82A is in the initial position, and the size of attention area 82A is approximately the same as the initial size (Px×Py). When the driver starts steering to the right while vehicle 1 is moving forward, the position of attention area 82B moves diagonally upward to the right. Because vehicle speed V is low, the size of attention area 82B is also approximately the same as the initial size (Px×Py).

[0108] FIG. 20(b) is a diagram illustrating the movement of the position of the attention area 82 and the change in the size of the attention area 82 when changing lanes while accelerating to the overtaking lane on the right side of the expressway. According to the driving assistance setting process, when the vehicle 1 is traveling at high speed, the position of the attention area 82A is higher than the initial position, and the size of the attention area 82A is smaller than the initial size (Px×Py). When the driver operates the steering wheel to the right while the vehicle 1 is accelerating, the position of the attention area 82B moves diagonally upward to the right. Since the vehicle speed V is fast, the size of the attention area 82B becomes even smaller. Note that FIG. 20 is an example of a case of left-hand traffic, and can also be used appropriately for turning left in a right-hand traffic or changing lanes in a right-hand traffic. In addition, the driver's line of sight may be detected by a gaze detection device (not shown) (for example, a gaze detection device is provided on the steering wheel) to set an area where the driver is not watching or an area that is a blind spot as the attention area 82. In addition, gaze detection can be performed using a corneal reflex method in which infrared light is reflected off the driver's cornea to detect the direction of the user's gaze, a limbus tracking method that utilizes the difference in light reflectance between the cornea and the sclera, and an image analysis method in which an image of the eye is captured by a camera and the gaze is detected through image processing. Any of these gaze detection methods may be used.

[0109] According to the above-described embodiment, the following advantageous effects can be obtained. (1) The camera is equipped with a control device 4 that recognizes at least one of the specifications of the vehicle 1 on which it is mounted and operations performed on the operating unit of the vehicle 1, an imaging unit 32 that has at least an attention area 82 and an imaging area 81 and captures images of the outside of the vehicle 1, and a control device 4 that sets different imaging conditions for the attention area 82 and the imaging conditions for the imaging area 81 based on the recognition results by the control device 4, so that the imaging conditions of the camera 3 can be set appropriately.

[0110] (2) The control device 4 recognizes the mounting position (right or left) of the handle (steering wheel 10) in the vehicle 1, and can therefore appropriately set the imaging conditions of the camera 3 according to the driver's riding position.

[0111] (3) The setting unit sets the frame rate of the attention area 82 and the frame rate of the imaging area 81 to be different depending on the position of the steering wheel 10, so that the imaging conditions of the camera 3 can be appropriately set, for example, by increasing the frame rate in the attention area 82 on the driver's side (right).

[0112] (4) The vehicle 1 is equipped with a control device 4 that detects information related to the vehicle speed V of the vehicle 1. The control device 4 sets the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81 to be different depending on the detection result of the information related to the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set depending on the vehicle speed V.

[0113] (5) The control device 4 changes at least one of the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81 when the information regarding the vehicle speed V increases and decreases, so that the imaging conditions of the camera 3 can be appropriately set, for example, by increasing the frame rate as the vehicle speed V becomes faster.

[0114] (6) When the rotation angle θ of the handle (steering wheel 10) exceeds a predetermined value, the control device 4 changes at least one of the frame rate for imaging the attention area 82 and the frame rate for imaging the imaging area 81 to a higher frame rate, so that the imaging conditions of the camera 3 can be changed in the case of a turning operation.

[0115] (7) The control device 4 transmits display information to the display device 14 of the vehicle 1 based on the imaging results of the imaging unit 32, so that necessary information can be provided to the occupants of the vehicle 1.

[0116] (8) When the rotation angle θ of the handle (steering wheel 10) does not reach a predetermined value, the control device 4 maintains the setting of at least one of the frame rate settings for imaging the attention area 82 and the frame rate setting for imaging the imaging area 81, thereby making it possible to avoid changing the imaging conditions during a small operation other than a turning operation. This prevents, for example, the frame rate of the attention area 82 from being changed more finely than necessary, which helps reduce the processing load.

[0117] (9) The control device 4 determines the imaging conditions that are different between the imaging conditions of the area of ​​interest 82 and the imaging conditions of the imaging area 81 by including at least one of the imaging frame rate, gain, thinning, pixel signal addition, accumulation, bit length, size of the imaging area, and position of the imaging area, and therefore can appropriately set the imaging conditions of the camera 3.

[0118] (10) The control device 4 changes at least one of the center positions of the attention area 82 and the center position of the imaging area 81 based on the detection results of information related to the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set, such as changing the position of the attention area 82 in accordance with changes in the vehicle speed V.

[0119] (11) The control device 4 changes at least one of the size of the attention area 82 and the size of the imaging area 81 based on the detection results of information related to the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set, such as changing the size of the attention area 82 in accordance with changes in the vehicle speed V.

[0120] (12) The control device 4 sets the imaging area 81 surrounding the attention area 82, so that the imaging conditions of the camera 3 can be set appropriately.

[0121] (13) The vehicle 1 is provided with a steering wheel (steering wheel 10) as an operating unit, and the control device 4 changes at least one of the center position of the attention area 82 and the center position of the imaging area 81 based on the operation of the steering wheel, so that it is possible to appropriately set the imaging conditions of the camera 3, such as changing the position of the attention area 82 in accordance with a change in the course of the vehicle 1. Note that, in the above-mentioned embodiment, the camera 3 is controlled by the control device 4, but part of the control of the camera 3 may be performed by the control unit 35 of the camera 3.

[0122] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 1) In the driving assist setting process, the control device 4 may be configured to change the position and size of the attention area 82 in response to an operation signal from the turn signal switch 11. As illustrated in Fig. 21, the control device 4 changes the size of the attention area 82 and sets the imaging conditions for the attention area 82 based on the initial setting value determined in step S24 and the turn signal direction by the operation of the turn signal switch 11.

[0123] For example, referring to FIG. 10, in the case of right-hand drive, left-hand traffic, and the initial setting value "4", the control device 4 performs control so that the attention area 82 includes the left edge of the road if the turn signal is to the left. Specifically, the attention area 82 in FIG. 10 is expanded to the left. The reason for expanding the attention area 82 to the left is to prevent a collision accident when turning left. Conversely, the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the right. Specifically, the attention area 82 in FIG. 10 is expanded to the right.

[0124] Explaining with reference to Fig. 14, in the case of a left-hand drive vehicle, right-hand traffic, and the initial setting value being "1", the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the left. Specifically, the attention area 82 in Fig. 14 is expanded to the left. Conversely, the control device 4 performs control so that the attention area 82 includes the right edge of the road if the turn signal is to the right. Specifically, the attention area 82 in Fig. 14 is expanded to the right. Expanding to the right is to prevent a hit-and-run accident when turning right.

[0125] Explaining with reference to Fig. 16, in the case of a vehicle with right-hand drive and right-hand traffic, where the initial setting value is "2", the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the left. Specifically, the attention area 82 in Fig. 16 is significantly expanded to the left. Conversely, the control device 4 performs control so that the attention area 82 includes the right edge of the road if the turn signal is to the right. Specifically, the attention area 82 in Fig. 16 is expanded slightly to the right. Expanding to the right is to prevent a hit-and-run accident when turning right.

[0126] Explaining with reference to FIG. 15, in the case of a left-hand drive vehicle, left-hand traffic, and the initial setting value being "3", the control device 4 performs control so that the left edge of the road is included in the attention area 82 if the turn signal is to the left. Specifically, the attention area 82 in FIG. 15 is expanded slightly to the left. The expansion to the left is to prevent collision accidents when turning left. Conversely, the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the right. Specifically, the attention area 82 in FIG. 15 is expanded significantly to the right.

[0127] Fig. 22 is a flowchart for explaining the processing when the turn signal switch 11 is operated according to the first modified example. When an operation signal is input from the turn signal switch 11 during the driving assist setting processing, the control device 4 starts the processing in Fig. 22 as a subroutine. In step S510 in Fig. 22, the control device 4 determines whether the turn signal direction is leftward. If the turn signal direction is leftward, the control device 4 makes an affirmative decision in step S510 and proceeds to step S520, and if the turn signal direction is rightward, the control device 4 makes a negative decision in step S510 and proceeds to step S530.

[0128] In step S520, the control device 4 determines whether the lane is on the left or not. If traffic is on the left, the control device 4 makes an affirmative decision in step S520 and proceeds to step S550, and if traffic is on the right, the control device 4 makes a negative decision in step S520 and proceeds to step S540.

[0129] In step S540, the control device 4 controls the imaging unit 32 to include the oncoming lane in the attention area 82, and ends the processing in Fig. 22. In step S550, the control device 4 controls the imaging unit 32 to include the left edge of the road in the attention area 82, and ends the processing in Fig. 22.

[0130] In step S530, the control device 4 determines whether the lane is on the left or not. If traffic is on the left, the control device 4 makes an affirmative decision in step S530 and proceeds to step S560, and if traffic is on the right, the control device 4 makes a negative decision in step S530 and proceeds to step S570.

[0131] In step S560, the control device 4 controls the imaging unit 32 to include the oncoming lane in the attention area 82, and ends the processing in Fig. 22. In step S570, the control device 4 controls the imaging unit 32 to include the right edge of the road in the attention area 82, and ends the processing in Fig. 22.

[0132] When the turn signal switch 11 is turned off after executing the process in Fig. 22, the control device 4 cancels the size change of the attention area 82 in Fig. 22. When the turn signal switch 11 is turned on, the frame rate, gain, thinning rate, and accumulation time of the attention area 82 may be set higher than those of the imaging area 81, even if the vehicle speed V is 0. However, when at least one of the frame rate, gain, thinning rate, accumulation time, etc. is to be made different between the imaging area 81 and the attention area 82, only the imaging condition to be made different is changed.

[0133] According to the above-described first modification, the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81 are set differently in response to the operation of the turn signal switch 11, so that, for example, when turning right or left at an intersection, the attention area 82 can be appropriately set by including an oncoming lane in the attention area 82 so that an oncoming vehicle can be reliably detected, or by including a road edge in the attention area 82 so that an entrapment accident can be prevented. Furthermore, the imaging conditions can be appropriately set in the imaging area 81 and the attention area 82, for example, by setting the frame rate of the attention area 82 higher than that of the imaging area 81.

[0134] (Variation 2) In the driving assistance setting process, the control device 4 may be configured to change the position of the attention area 82 and the size of the attention area 82 in accordance with changes in the distance between the vehicle 1 and an object such as a two-wheeled vehicle, a regular vehicle, a large vehicle, or a pedestrian.

[0135] In the second modification, for example, when vehicle 1 approaches a preceding vehicle and the distance L (inter-vehicle distance) to the preceding vehicle becomes shorter, control device 4 determines the position of attention area 82 so that the preceding vehicle is included in attention area 82. Here, the change in the inter-vehicle distance L from vehicle 1 to the preceding vehicle is obtained by distance measurement calculation unit 35a of camera 3 successively measuring distance L (inter-vehicle distance) to the preceding vehicle within the captured screen based on images acquired by camera 3 at predetermined time intervals synchronized with the acquisition timing of vehicle speed V.

[0136] The control device 4 calculates the position (Y coordinate) of the attention area 82 while traveling by using the inter-vehicle distance L instead of the depth Z in the above formula (5). As a result, when capturing an image of a preceding vehicle on a flat, straight road, the value of Yq indicating the position of the attention area 82 in the Y-axis direction on the imaging chip 113 increases as the inter-vehicle distance L increases and decreases as the inter-vehicle distance L decreases.

[0137] Furthermore, when the inter-vehicle distance L changes and becomes shorter, the preceding vehicle appears larger on the camera 3, so the control device 4 sets the size of the attention area 82 to be larger. Conversely, when the inter-vehicle distance L changes and becomes longer, the preceding vehicle appears smaller on the camera 3, so the control device 4 sets the size of the attention area 82 to be smaller. The control device 4 calculates the size of the attention area 82 while driving by substituting the inter-vehicle distance L instead of the depth Z in the above equations (7) and (8).

[0138] 17 described above, the size of attention area 82 is set to be small as vehicle speed V increases, but in Modification 2, even if vehicle speed V is fast, if inter-vehicle distance L to the preceding vehicle is short, the size of attention area 82 is set to be large, so that the preceding vehicle can be appropriately included in attention area 82. This makes it easier to detect changes in the traveling state of the preceding vehicle based on images captured by camera 3, compared to when the size of attention area 82 is continuously set small. With respect to the region of interest 82 whose size or position has been changed in this way, at least one of the frame rate, gain, thinning rate, accumulation time, etc. may be made different between the imaging region 81 and the region of interest 82 .

[0139] (Variation 3) In addition to the existing attention area 82, when the control device 4 detects an object around the vehicle 1, such as a two-wheeled vehicle, a normal vehicle, a large vehicle, or a pedestrian, the control device 4 may set a new attention area 82 including the object. In the third modification, when the detected object moves, the control device 4 sets a new attention area 82 including the object. For example, when the distance between the detected object and the vehicle 1 comes within a predetermined distance, the control device 4 sets a new attention area 82 including the object. Then, when the distance between the detected object and the vehicle 1 becomes greater than or equal to the predetermined distance, the control device 4 cancels the setting of the attention area 82 including the object. With regard to the region of interest 82 thus set, at least one of the frame rate, gain, thinning rate, accumulation time, and the like may be made different between the imaging region 81 and the region of interest 82 .

[0140] According to variant example 3, a control device 4 is provided that detects moving objects around the vehicle 1 based on information from the camera 3, and the control device 4 changes at least one of the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81 based on the detection result of the moving object, so that the imaging conditions of the camera 3 can be appropriately set depending on the presence or absence of a moving object.

[0141] In addition, when it is detected based on information from camera 3 that the distance between vehicle 1 and the moving object is approaching within a predetermined distance, the control device 4 changes at least one of the frame rate for imaging the attention area 82 and the frame rate for imaging the imaging area 81 to a higher frame rate, thereby making it easier to detect changes in the movement state of the moving object based on the image acquired by camera 3.

[0142] (Variation 4) The attention area 82 may be newly set based on the color of the image captured by the camera 3. The control device 4 sets an area in the image that includes a red object as the attention area 82. By adding an area that includes a red object to the attention area 82, it is possible to include in the attention area 82, for example, a red traffic light, a warning device at a railway crossing, a red light of an emergency vehicle, and the like. With regard to the region of interest 82 thus set, at least one of the frame rate, gain, thinning rate, accumulation time, and the like may be made different between the imaging region 81 and the region of interest 82 .

[0143] (Variation 5) The attention area 82 may be newly set based on sound information collected by the microphone 17 of the vehicle 1. For example, when the level of sound information from the right side of the vehicle 1 is input and exceeds a predetermined value, the control device 4 expands the attention area 82 to the right side of the imaging area 81, or sets a new attention area 82 to the right side of the imaging area 81. The attention area 82 is set to the right side in order to collect outside information on the right side of the vehicle 1.

[0144] Furthermore, for example, when the level of sound information from the left side of the vehicle 1 is input and exceeds a predetermined value, the control device 4 expands the attention area 82 to the left side of the imaging area 81, or sets a new attention area 82 to the left side of the imaging area 81. The attention area 82 is set to the left side in order to collect outside information on the left side of the vehicle 1. With regard to the region of interest 82 thus set, at least one of the frame rate, gain, thinning rate, accumulation time, and the like may be made different between the imaging region 81 and the region of interest 82 .

[0145] (Variation 6) In the above embodiment, a case has been described in which attention area 82 including a preceding vehicle is set, but attention area 82 may be set including an oncoming vehicle of vehicle 1. In Modification 6, control device 4 recognizes a vehicle that is closest to vehicle 1 as an oncoming vehicle from among objects that exist within the above-mentioned traveling area and travel in the opposite direction (opposite vehicle to vehicle 1).

[0146] The control device 4 sets an area corresponding to the position of the oncoming vehicle in the image acquired by the camera 3 as an attention area 82. The control device 4 sets the attention area 82 to include, in particular, the license plate of the oncoming vehicle and the face of the driver in the driver's seat of the oncoming vehicle.

[0147] By setting an area including the license plate of an oncoming vehicle or the face of the driver of the oncoming vehicle as attention area 82, an oncoming vehicle approaching vehicle 1 can be appropriately included in attention area 82. With regard to the region of interest 82 thus set, at least one of the frame rate, gain, thinning rate, accumulation time, and the like may be made different between the imaging region 81 and the region of interest 82 .

[0148] (Variation 7) In the above description, an example has been described in which the imaging area 81 includes the attention area 82 (surrounds the attention area 82), but the imaging area 81 and the attention area 82 may be set side by side. The sizes and positions of the imaging area 81 and the attention area 82 can be changed by moving the boundary between the imaging area 81 and the attention area 82 left and right. With respect to the attention area 82 set in this manner, at least one of the frame rate, gain, thinning rate, accumulation time, etc. may be made different between the imaging area 81 and the attention area 82.

[0149] In the above description, the distance is measured by the camera 3 through distance measurement calculation using image signals from focus detection pixels provided in the image sensor 100, but a distance measurement method using two images captured by a stereo camera may also be used. Also, a distance measurement method using a millimeter wave radar separate from the camera 3 may also be used.

[0150] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0151] 1. Vehicle 2. Driving support device 3. Camera 4. Control device 4b…Storage section 5...First driving control unit 6...Second driving control unit 7...Throttle control device 7a…Accelerator pedal 8...Brake control device 8a…Brake pedal 9...Steering control device 10…Steering wheel 11...Turn signal switch 12...Vehicle speed sensor 14...Display device 15...GPS device 16...Shift lever position detection device 17…Mike 31...Imaging optical system 32…Imaging unit 32a...Drive unit 33...Image processing unit 34…Work memory 35...Control section 35a…Distance calculation unit 36…Recording section 60...Focus detection pixel line 81…Imaging area 82, 82A, 82B…Area of ​​interest 83…Hibernation area 100...Image sensor 113...Imaging chip

Claims

1. An imaging device mounted on a vehicle, an input unit for inputting information about the speed of the vehicle; an imaging unit that captures an image of the outside of the vehicle; an imaging control unit that changes the imaging conditions of the imaging unit when information about the vehicle speed changes; and the imaging control unit, when it is determined that sudden braking has occurred based on information about the vehicle speed, increases a frame rate and captures an image of the outside of the vehicle. Imaging device.

2. The imaging surface of the imaging chip controlled by the imaging unit has an imaging area in which charge accumulation is performed and a region of interest in which charge accumulation is performed under conditions different from those of the imaging area, The imaging region and the region of interest have at least different thinning rates. The imaging device according to claim 1 .

3. The imaging control unit stops thinning out the area of ​​interest when it is determined that sudden braking has occurred. The imaging device according to claim 2 .

4. The imaging control unit changes the position of the attention area in response to a steering operation of the vehicle. The imaging device according to claim 2 .

5. The imaging control unit shifts the center of the attention area in a vertically upward direction as the speed of the vehicle increases. The imaging device according to claim 2 or 4.

Images