Driving support device

Abstract

To appropriately set the imaging conditions of the imaging device. The imaging device is mounted on a vehicle and includes an imaging unit that captures images of the exterior of the vehicle, and an imaging control unit that controls imaging conditions of the imaging unit based on the position of the vehicle's steering wheel.

Description

Technical Field

[0001] The present invention relates to a driving support device in the position .

Background Art

[0002] There has been developed a technique for detecting a driving environment of a vehicle based on an image acquired by a camera mounted on the vehicle, and performing automatic driving control such as following driving of a preceding vehicle and driving support such as warning, braking, and steering assistance based on the detected driving environment data (see Patent Document 1). In the prior art, a solid-state imaging device such as a CCD is used for an in-vehicle camera. The in-vehicle camera that continuously acquires images of roads and the like plays an important role in automatic driving control and driving support. However, there have not been many proposals for cameras on the premise of mounting on a vehicle, and the usability of the camera has not been sufficient.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] The driving support device according to the present invention includes an imaging unit that images the outside of the vehicle, a distance measuring unit that measures the distance to an object existing outside the vehicle, a positioning unit that measures the current position of the vehicle, and a control unit that generates driving support data, which is data for performing driving support of the vehicle, based on information from the imaging unit, the distance measuring unit, and the positioning unit. The control unit generates the driving support data using information on a target imaging object from an image captured by the imaging unit. Classify and extract motorcycles, ordinary vehicles, large vehicles, pedestrians, and utility poles, and set a target area including the extracted motorcycles, ordinary vehicles, large vehicles, and pedestrians. The where the target area is set above

Brief Description of the Drawings

[0005]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0006] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. <Usage Scenario of the Camera> FIG. 1 is a schematic configuration diagram of a driving support device 2 of a vehicle 1 equipped with a camera 3 according to an embodiment of the present invention. In FIG. 1, a driving support device 2 is mounted on a vehicle 1 such as an automobile. The driving support device 2 is composed of a camera 3, a control device 4, a first driving control unit 5, a second driving control unit 6, and the like. In this description, an example using an internal combustion engine as a drive source will be described, but one using a motor as a drive source may also be used.

[0007] The camera 3 includes an imaging optical system having a plurality of lenses and an image sensor (in this embodiment, a stacked image sensor (see FIG. 3)), and is attached, for example, in front of the ceiling inside the vehicle cabin. The camera 3 is directed forward of the vehicle 1, and its mounting height (the distance from the ground to the camera 3) is adjusted to, for example, 1.4 m. The camera 3 acquires an image in the traveling direction of the vehicle 1 and measures the distance (range measurement) to each subject (object) at a plurality of positions in the captured image based on the acquired image. The distance measurement is calculated by a range measurement operation using an image signal from a focus detection pixel provided in the stacked image sensor. The focus detection pixel and the range measurement will be described later. The image data and the range measurement data acquired by the camera 3 are sent to the control device 4. Note that the camera 3 may be provided outside the vehicle, or the cameras 3 inside and outside the vehicle may cooperate, and the number of cameras 3 may be set as appropriate. For example, the white line detection described later may use the camera 3 outside the vehicle, and the recognition of the object and the obstacle may be performed by the cooperation of the cameras 3 inside and outside the vehicle.

[0008] As illustrated in FIG. 2, the control device 4 includes a CPU 4a and a storage unit 4b. The CPU 4a performs various calculations using control parameters stored in the storage unit 4b and detection signals from various sensors described later based on various programs stored in the storage unit 4b.

[0009] The first driving control unit 5 performs a constant-speed driving control and a following driving control based on an instruction from the control device 4. The constant-speed driving control is a control for driving the vehicle 1 at a constant speed based on a predetermined control program. The following driving control is a control for driving the vehicle 1 while maintaining a constant inter-vehicle distance from a preceding vehicle when the speed of the preceding vehicle recognized by the control device 4 is equal to or lower than the target speed set for the vehicle 1 during the constant-speed driving control.

[0010] The second driving control unit 6 performs driving assistance control based on an instruction from the control device 4. The driving assistance control outputs a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road, or outputs a brake control signal to the brake control device 8 so as to avoid the vehicle 1 from colliding with an object, based on a predetermined control program.

[0011] Further shown in FIG. 1 are a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, a display device 14, a GPS device 15, a shift lever position detection device 16, and a microphone 17.

[0012] The throttle control device 7 controls the opening degree of a throttle valve (not shown) according to the depression amount of the accelerator pedal 7a. Further, the throttle control device 7 also controls the opening degree of the throttle valve according to a throttle control signal sent from the first driving control unit 5. The throttle control device 7 further sends a signal indicating the depression amount of the accelerator pedal 7a to the control device 4.

[0013] The brake control device 8 controls the opening degree of a brake valve (not shown) according to the depression amount of the brake pedal 8a. Further, the brake control device 8 also controls the opening degree of the brake valve according to a brake control signal from the second driving control unit 6. The brake control device 8 further sends a signal indicating the depression amount of the brake pedal 8a to the control device 4.

[0014] The steering control device 9 controls the steering angle of a steering device (not shown) according to the rotation angle of the steering wheel 10. Further, the steering control device 9 also controls the steering angle of the steering device according to a steering control signal from the second driving control unit 6. The steering control device 9 further sends a signal indicating the rotation angle of the steering wheel 10 to the first driving control unit 5 and the control device 4, respectively.

[0015] The turn signal switch 11 is a switch for operating a turn signal (winker) device (not shown). The turn signal device is a flashing light device indicating a change in the course of the vehicle 1. When the turn signal switch 11 is operated by an occupant of the vehicle 1, an operation signal from the turn signal switch 11 is sent to the turn signal device, the second travel control unit 6, and the control device 4, respectively. The vehicle speed sensor 12 detects the vehicle speed V of the vehicle 1 and sends a detection signal to the first travel control unit 5, the second travel control unit 6, and the control device 4, respectively.

[0016] The yaw rate sensor 13 detects the yaw rate of the vehicle 1 and sends a detection signal to the second travel control unit 6 and the control device 4, respectively. The yaw rate is the rate of change of the rotation angle in the turning direction of the vehicle 1. The display device 14 displays information indicating the control states by the first travel control unit 5 and the second travel control unit 6, etc. The display device 14 is constituted by, for example, a HUD (Head Up Display) that projects information onto the windshield. Note that, as the display device 14, the display unit of a navigation device (not shown) may be used.

[0017] The GPS device 15 receives radio waves from GPS satellites and calculates the position (latitude, longitude, etc.) of the vehicle 1 by performing predetermined calculations using the information carried on the radio waves. The position information calculated by the GPS device 15 is sent to a navigation device (not shown) and the control device 4. The shift lever position detection device 16 detects the position of a shift lever (not shown) operated by an occupant of the vehicle 1 (for example, parking (P), reverse (R), drive (D), etc.). The position information of the shift lever detected by the shift lever position detection device 16 is sent to the control device 4.

[0018] The microphone 17 is composed of, for example, a front microphone, a right-side microphone, and a left-side microphone. The front microphone has a directivity for collecting only the sounds in front of the vehicle 1. The right-side microphone has a directivity for collecting only the sounds on the right side of the vehicle 1. The left-side microphone has a directivity for collecting only the sounds on the left side of the vehicle 1. Each piece of sound information (front, right side, left side) collected by the microphone 17 is sent to the control device 4 respectively.

[0019] <Detection of Object> The control device 4 performs image processing on the image from the camera 3 as follows to detect the traveling road and objects of the vehicle 1. First, the control device 4 generates a distance image (depth distribution image) based on the distance measurement data at a plurality of positions within the photographed screen. Based on the data of the distance image, the control device 4 performs a well-known grouping process and compares it with frames (windows) such as three-dimensional road shape data, side wall data, and object data previously stored in the storage unit 4b, and extracts white line data (including white line data along the road and white line (stop line: intersection information) data crossing the road), side wall data such as guardrails and curbstones existing along the road, and classifies and extracts objects / obstacles into other objects such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, and utility poles. In this description, a white or yellow line drawn on the traveling road is called a white line. Also, both solid lines and dashed lines are included and called white lines.

[0020] <Driving Support> The control device 4 recognizes the traveling road and object / obstacle that becomes an obstacle based on each piece of information extracted as described above, that is, the white line data, guardrail side wall data, and object data, and based on the recognition result, causes the second driving control unit 6 to perform the above driving support control. That is, the vehicle 1 is made to travel along the road and the vehicle 1 is prevented from colliding with an object.

[0021] <Travel Control> The control device 4 estimates the traveling path of the host vehicle in, for example, the following four ways. (1) Estimation of the host vehicle traveling path based on the white line When white line data on both the left and right sides of the driving lane, or on either the left or right side alone, is obtained from the image acquired by the camera 3 and the shape of the lane in which the vehicle 1 is traveling can be estimated from these white line data, the control device 4 estimates that the own vehicle's travel path is parallel to the white line, taking into account the width of the vehicle 1 and the position of the vehicle 1 within the current lane.

[0022] (2) Estimation of own vehicle's travel path based on side wall data such as guardrails and curbstones When side wall data on both the left and right sides of the driving lane, or on either the left or right side alone, is obtained from the image acquired by the camera 3 and the shape of the lane in which the vehicle 1 is traveling can be estimated from these side wall data, the control device 4 estimates that the own vehicle's travel path is parallel to the side wall, taking into account the width of the vehicle 1 and the position of the vehicle 1 within the current lane.

[0023] (3) Estimation of own vehicle's travel path based on the preceding vehicle's trajectory The control device 4 estimates the own vehicle's travel path based on the past travel trajectory of the preceding vehicle stored in the storage unit 4b. The preceding vehicle refers to the vehicle closest to the vehicle 1 among the objects traveling in the same direction as the vehicle 1.

[0024] (4) Estimation of own vehicle's travel path based on the travel trajectory of the vehicle 1 The control device 4 estimates the own vehicle's travel path based on the driving state of the vehicle 1. For example, the own vehicle's travel path is estimated using the turning curvature based on the detection signal from the yaw rate sensor 13 and the detection signal from the vehicle speed sensor 12. The turning curvature Cua is calculated by Cua = dψ / dt / V. dψ / dt is the above-mentioned yaw rate (the change rate of the rotation angle in the turning direction), and V is the vehicle speed of the vehicle 1.

[0025] The control device 4 estimates, for each of the above objects, the driving area of the vehicle 1 at the position where the object exists based on the own vehicle's traveling path according to a predetermined driving control program stored in the storage unit 4b, compares this driving area with the object position, and determines whether each object is within the driving area. The control device 4 further recognizes the preceding vehicle based on the imaging result of the camera 3. That is, the control device 4 sets, as the preceding vehicle, the vehicle closest to the vehicle 1 among the objects existing within the driving area and traveling in the forward direction (the same direction as the vehicle 1).

[0026] The control device 4 outputs the inter-vehicle distance information between the preceding vehicle and the vehicle 1 and the vehicle speed information of the preceding vehicle to the first driving control unit 5 as external vehicle information. Here, the vehicle speed information of the preceding vehicle is calculated based on the vehicle speed V of the vehicle 1 acquired at predetermined time intervals and the change in the distance (inter-vehicle distance) to the preceding vehicle in the captured image acquired by the camera 3 at the same predetermined time intervals synchronized with the acquisition timing of the vehicle speed V.

[0027] The first driving control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a preset predetermined vehicle speed (target speed). Thereby, the throttle control device 7 feedback-controls the opening degree of a throttle valve (not shown), and automatically drives the vehicle 1 at a constant speed.

[0028] Further, when the vehicle speed information of the preceding vehicle input from the control device 4 is equal to or lower than the target speed set for the vehicle 1 during the driving control in the constant speed state, the first driving control unit 5 sends a throttle control signal to the throttle control device 7 based on the inter-vehicle distance information input from the control device 4. Specifically, an appropriate target value of the inter-vehicle distance is set based on the inter-vehicle distance from the vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of the vehicle 1, and a throttle control signal is sent to the throttle control device 7 so that the inter-vehicle distance measured based on the image acquired by the camera 3 converges to the target value of the inter-vehicle distance. Thereby, the throttle control device 7 feedback-controls the opening degree of a throttle valve (not shown), and makes the vehicle 1 follow the preceding vehicle.

[0029] <Description of the stacked image sensor> The stacked image sensor 100 provided in the camera 3 described above will be described. Note that this stacked image sensor 100 is the one described in Japanese Patent Application No. 2012-139026 previously filed by the applicant of the present application. FIG. 3 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 includes a back-illuminated imaging chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. These imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by bumps 109 having conductivity such as Cu.

[0030] As shown in the figure, incident light mainly enters in the +Z-axis direction indicated by the white arrow. In the present embodiment, in the imaging chip 113, the surface on the side where the incident light enters is referred to as the back surface (imaging surface). Also, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z-axis is defined as the +X-axis direction, and the front direction of the paper surface orthogonal to the Z-axis and the X-axis is defined as the +Y-axis direction. In some of the following figures, the coordinate axes are displayed based on the coordinate axes of FIG. 3 so that the orientation of each figure can be understood.

[0031] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges corresponding to incident light, and transistors 105 provided corresponding to the PDs 104.

[0032] A color filter 102 is provided via a passivation film 103 on the incident side of the incident light in the PD layer 106. The color filter 102 has a plurality of types that transmit different wavelength regions and has a specific arrangement corresponding to each of the PDs 104. The arrangement of the color filter 102 will be described later. A set of the color filter 102, PD 104, and transistor 105 forms one pixel.

[0033] On the incident side of the incident light in the color filter 102, a microlens 101 is provided corresponding to each pixel. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0034] The wiring layer 108 has a wiring 107 that transmits the pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multilayered, and passive elements and active elements may be provided.

[0035] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed or the like, so that the aligned bumps 109 are joined to each other and electrically connected.

[0036] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed or the like, so that the aligned bumps 109 are joined to each other and electrically connected.

[0037] Note that the joining between the bumps 109 is not limited to Cu bump joining by solid-phase diffusion, and microbump bonding by solder melting may be employed. Also, for example, about one bump 109 may be provided for one block described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in the peripheral region outside the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may be provided together.

[0038] The signal processing chip 111 has TSVs (through-silicon vias) 110 that connect the circuits provided on the front and back surfaces to each other. The TSVs 110 are preferably provided in the peripheral region. Also, the TSVs 110 may be provided in the peripheral region of the imaging chip 113 and the memory chip 112 as well.

[0039] FIG. 4 is a diagram for explaining the pixel array and the unit region 131 of the imaging chip 113. In particular, it shows how the imaging chip 113 is observed from the back surface (imaging surface) side. In the pixel region, for example, more than 20 million pixels are arranged in a matrix. In the example of FIG. 4, 16 pixels of 4 adjacent pixels × 4 pixels form one unit region 131. The grid lines in the figure indicate the concept that adjacent pixels are grouped to form the unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 64 pixels, or more or less.

[0040] As shown in the partial enlarged view of the pixel region, the unit region 131 in FIG. 4 encloses four so-called Bayer arrays each consisting of 4 pixels of green pixels Gb, Gr, blue pixel B, and red pixel R in the up, down, left, and right directions. The green pixels Gb, Gr are pixels having a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel having a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter 102 and receives light in the red wavelength band.

[0041] In the present embodiment, a plurality of blocks are defined such that each block includes at least one unit region 131, and each block can control the pixels included in each block with different control parameters. That is, imaging signals with different imaging conditions can be obtained between the pixel group included in one block and the pixel group included in another block. Examples of the control parameters include frame rate, gain, decimation rate, number of addition rows or addition columns for adding pixel signals, charge accumulation time or accumulation times, number of bits (word length) for digitization, and the like. The imaging device 100 can freely perform decimation not only in the row direction (X-axis direction of the imaging chip 113) but also in the column direction (Y-axis direction of the imaging chip 113). Further, the control parameter may be a parameter in image processing after obtaining an image signal from a pixel.

[0042] FIG. 5 is a diagram for explaining the circuit in the unit region 131. In the example of FIG. 5, one unit region 131 is formed by nine adjacent pixels of 3 pixels × 3 pixels. Note that the number of pixels included in the unit region 131 is not limited to this, and may be less than or more than this. The two-dimensional positions of the unit region 131 are indicated by symbols A to I.

[0043] The reset transistors of the pixels included in the unit region 131 are configured to be individually turnable on and off for each pixel. In FIG. 5, a reset wiring 300 for turning on and off the reset transistor of pixel A is provided, and a reset wiring 310 for turning on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 for turning on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. For the other pixels D to I, dedicated reset wirings for turning on and off their respective reset transistors are provided.

[0044] Regarding the transfer transistors of the pixels included in the unit area 131, they are also configured to be individually turned on and off for each pixel. In FIG. 5, a transfer wiring 302 for turning on and off the transfer transistor of pixel A, a transfer wiring 312 for turning on and off the transfer transistor of pixel B, and a transfer wiring 322 for turning on and off the transfer transistor of pixel C are provided separately. For other pixels D to I, dedicated transfer wirings for turning on and off their respective transfer transistors are provided.

[0045] Furthermore, regarding the selection transistors of the pixels included in the unit area 131, they are also configured to be individually turned on and off for each pixel. In FIG. 5, a selection wiring 306 for turning on and off the selection transistor of pixel A, a selection wiring 316 for turning on and off the selection transistor of pixel B, and a selection wiring 326 for turning on and off the selection transistor of pixel C are provided separately. For other pixels D to I, dedicated selection wirings for turning on and off their respective selection transistors are provided.

[0046] Note that the power supply wiring 304 is commonly connected from pixel A to pixel I included in the unit area 131. Similarly, the output wiring 308 is commonly connected from pixel A to pixel I included in the unit area 131. Also, the power supply wiring 304 is commonly connected between a plurality of unit areas, while the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0047] By individually turning on and off the reset transistor and the transfer transistor of the unit area 131, charge accumulation including the charge accumulation start time, the charge accumulation end time, and the transfer timing can be controlled independently for pixels A to I included in the unit area 131. Also, by individually turning on and off the selection transistors of the unit area 131, the pixel signals of each of pixels A to I can be output via the common output wiring 308.

[0048] Here, regarding pixels A to I included in the unit area 131, a so-called rolling shutter method is known, which controls charge accumulation in a regular order with respect to rows and columns. When pixels are selected row by row and then columns are specified by the rolling shutter method, in the example of FIG. 5, pixel signals are output in the order of "ABCDEFGHI".

[0049] By configuring the circuit based on the unit area 131 in this way, the charge accumulation time can be controlled for each unit area 131. In other words, pixel signals with different frame rates can be output between the unit areas 131. Also, while causing the unit areas 131 included in a part of the area in the imaging chip 113 to perform charge accumulation (imaging), by resting the unit areas 131 included in other areas, imaging can be performed only in a predetermined area of the imaging chip 113, and the pixel signals can be output. Furthermore, by switching the area where charge accumulation (imaging) is performed between frames (the area to be controlled for accumulation), sequential imaging can be performed in different areas of the imaging chip 113, and pixel signals can be output.

[0050] FIG. 6 is a block diagram showing the functional configuration of the imaging device 100 corresponding to the circuit illustrated in FIG. 5. The analog multiplexer 411 sequentially selects nine PDs 104 forming the unit area 131, and outputs the respective pixel signals to the output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed in the imaging chip 113 together with the PD 104.

[0051] The pixel signals output via the multiplexer 411 are subjected to correlated double sampling (CDS) and analog / digital (A / D) conversion by a signal processing circuit 412 formed in the signal processing chip 111 that performs CDS and A / D conversion. The A / D converted pixel signals are delivered to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. The demultiplexer 413 and the pixel memory 414 are formed in the memory chip 112.

[0052] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and delivers them to the subsequent image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that FIG. 6 shows the connection for one unit area 131, but in reality, these exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not necessarily exist for each unit area 131. For example, one arithmetic circuit 415 may sequentially process the values of the pixel memories 414 corresponding to the respective unit areas 131 while referring to them in order.

[0053] As described above, output wirings 308 are provided corresponding to each of the unit areas 131. Since the imaging device 100 stacks the imaging chip 113, the signal processing chip 111, and the memory chip 112, by using the electrical connection between the chips using the bumps 109 for these output wirings 308, the wirings can be routed without increasing the chips in the plane direction.

[0054] <Explanation of distance measurement> FIG. 7 is a diagram illustrating the positions of the focus detection pixels on the imaging surface of the imaging device 100. In the present embodiment, the focus detection pixels are discretely arranged along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of FIG. 7, 15 focus detection pixel lines 60 are provided at a predetermined interval. The focus detection pixels constituting the focus detection pixel line 60 output an image signal for distance measurement. Normal imaging pixels are provided at pixel positions other than the focus detection pixel line 60 in the imaging chip 113. The imaging pixels output an image signal for external vehicle monitoring.

[0055] FIG. 8 is an enlarged view of a region including a part of one of the focus detection pixel lines 60. In FIG. 8, a red pixel R, green pixels G (Gb, Gr), and a blue pixel B, and a focus detection pixel S1 and a focus detection pixel S2 are illustrated. The red pixel R, the green pixels G (Gb, Gr), and the blue pixel B are arranged according to the rules of the above-described Bayer array.

[0056] The square regions exemplified for the red pixel R, the green pixels G (Gb, Gr), and the blue pixel B indicate the light-receiving regions of the imaging pixels. Each imaging pixel receives a light beam passing through the exit pupil of the imaging optical system 31 (FIG. 9). That is, the red pixel R, the green pixels G (Gb, Gr), and the blue pixel B each have a square mask opening, and the light passing through these mask openings reaches the light-receiving portion of the imaging pixel.

[0057] Note that the shapes of the light-receiving regions (mask openings) of the red pixel R, the green pixels G (Gb, Gr), and the blue pixel B are not limited to quadrilaterals, and may be, for example, circular.

[0058] The semicircular regions exemplified for the focus detection pixel S1 and the focus detection pixel S2 indicate the light-receiving regions of the focus detection pixels. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 8, and the light passing through this mask opening reaches the light-receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 8, and the light passing through this mask opening reaches the light-receiving portion of the focus detection pixel S2. In this way, the focus detection pixel S1 and the focus detection pixel S2 each receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system 31 (FIG. 9).

[0059] Note that the position of the focus detection pixel line in the imaging chip 113 is not limited to the position exemplified in FIG. 7. Also, the number of focus detection pixel lines is not limited to the example of FIG. 7. Furthermore, the shapes of the mask openings in the focus detection pixel S1 and the focus detection pixel S2 are not limited to semicircles, and may be, for example, rectangular shapes obtained by horizontally dividing the square light-receiving regions (mask openings) in the imaging pixel R, the imaging pixel G, and the imaging pixel B.

[0060] In addition, the focus detection pixel lines in the imaging chip 113 may be provided by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 113. An image sensor in which imaging pixels and focus detection pixels are arranged two-dimensionally as shown in FIG. 8 is known, and detailed illustration and description of these pixels are omitted. In the example of FIG. 8, the configuration in which the focus detection pixels S1 and S2 receive one of a pair of light beams for focus detection, that is, the so-called 1PD structure, has been described. Instead, for example, as disclosed in Japanese Patent Application Laid-Open No. 2007-282107, a configuration in which the focus detection pixels receive both of a pair of light beams for focus detection, that is, a so-called 2PD structure, may be employed. By adopting the 2PD structure in this way, it becomes possible to read out image data also from the focus detection pixels, and the focus detection pixels will not become defective pixels.

[0061] In the present embodiment, based on the distance measurement image signals output from the focus detection pixel S1 and the focus detection pixel S2, the amount of image shift (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (FIG. 9) is detected, and thereby the focus adjustment state (defocus amount) of the imaging optical system 31 is calculated.

[0062] Generally, the above-mentioned pair of images approach each other in the so-called front pin state where the imaging optical system 31 forms a sharp image of an object (e.g., a preceding vehicle) in front of the planned focus plane, and conversely, move away from each other in the so-called rear pin state where the imaging optical system 31 forms a sharp image of the object behind the planned focus plane. In the in-focus state where a sharp image of the object is formed on the planned focus plane, the above-mentioned pair of images relatively coincide. Therefore, the relative position shift amount of the pair of images corresponds to the distance to the object (depth information).

[0063] Since the defocus amount calculation based on the above phase difference is known in the field of cameras, a detailed description thereof will be omitted. Here, since the defocus amount and the distance to the object correspond one-to-one, by obtaining the defocus amount for each object, the distance from the camera 3 to each object can be obtained. That is, distance measurement (ranging) to the above object can be performed at a plurality of positions on the imaging screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a mathematical formula or a look-up table and stored in the non-volatile memory 35b (FIG. 9).

[0064] <Description of the camera> FIG. 9 is a block diagram illustrating the configuration of the camera 3 having the imaging device 100 described above. In FIG. 9, the camera 3 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36.

[0065] The imaging optical system 31 guides a light beam from the object field to the imaging unit 32. The imaging unit 32 includes the imaging device 100 and a driving unit 32a, and photoelectrically converts an image of an object formed on the imaging chip 113 by the imaging optical system 31. The driving unit 32a generates a driving signal necessary to cause the imaging device 100 (imaging chip 113) to perform independent accumulation control in the above-described block units. Instructions such as the position, shape, range, and accumulation time of the above block are transmitted from the control unit 35 to the driving unit 32a.

[0066] The image processing unit 33 performs image processing on the image data captured by the imaging unit 32 in cooperation with the work memory 34. The image processing unit 33 performs color detection of an object included in the image in addition to image processing such as edge enhancement processing and gamma correction.

[0067] The work memory 34 temporarily stores image data before and after image processing. The recording unit 36 records image data and the like on a storage medium composed of a non-volatile memory or the like. The control unit 35 is constituted by, for example, a CPU, and controls the overall operation of the camera 3 according to a control signal from the control device 4. For example, a predetermined exposure calculation is performed based on the image signal captured by the imaging unit 32, and the accumulation time of the imaging chip 113 required for proper exposure is instructed to the drive unit 32a.

[0068] The control unit 35 includes a distance measurement calculation unit 35a and a non-volatile memory 35b. The distance measurement calculation unit 35a measures the distance to the object (distance measurement) at a plurality of positions on the shooting screen as described above. The image data acquired by the camera 3 and the distance measurement data calculated by the camera 3 are sent to the control device 4 (Fig. 1). The non-volatile memory 35b stores the program executed by the control unit 35a and the information necessary for distance measurement.

[0069] <Block control of the imaging device> The control device 4 causes the imaging device 100 (imaging chip 113) of the camera 3 to perform independent accumulation control in the above-described block units. For this purpose, the following signals are input to the control device 4 from each part of the vehicle 1 (Fig. 2). (1) Depression amount of the accelerator pedal 7a A signal indicating the depression amount of the accelerator pedal 7a is input from the throttle control device 7 to the control device 4. (2) Depression amount of the brake pedal 8a A signal indicating the depression amount of the brake pedal 8a is input from the brake control device 8 to the control device 4.

[0070] (3) Rotation angle of the steering wheel 10 A signal indicating the rotation angle of the steering wheel 10 is input from the steering control device 9 to the control device 4. The ratio between the rotation angle of the steering wheel 10 and the steering angle of the steering device is determined by the gear ratio of the steering. (4) Vehicle speed V of the vehicle 1 A detection signal from the vehicle speed sensor 12 is input to the control device 4.

[0071] (5) Operation signal of turn signal switch 11 The operation signal of turn signal switch 11 is input to control device 4. (6) Operation position of shift lever A signal indicating the operation position of the shift lever detected by shift lever position detection device 16 is input to control device 4.

[0072] (7) Position information of vehicle 1 The position information measured by GPS device 15 is input from GPS device 15 to control device 4. (8) Sound information around vehicle 1 Sound information from the front, right side, and left side of vehicle 1 collected by microphone 17 is input to control device 4 respectively.

[0073] FIG. 10 is a diagram illustrating the imaging surface of imaging chip 113, regions (imaging region 81 and region of interest 82) where charge accumulation (imaging) is performed in imaging chip 113, and regions (rest region 83) where charge accumulation (imaging) is not performed in the row direction and column direction. Region of interest 82 is a region where charge accumulation (imaging) is performed under conditions different from those of imaging region 81. The sizes and positions of imaging region 81 and region of interest 82 in imaging chip 113 are also one of the imaging conditions.

[0074] Control device 4 controls so as to perform imaging by setting first conditions for each unit region 131 included in imaging region 81, and also controls so as to perform imaging by setting second conditions for each unit region 131 included in region of interest 82. Further, control device 4 causes rest for unit regions 131 included in rest region 83 so as not to perform imaging. Note that a plurality of regions of interest 82 may be provided, or imaging conditions may be made different between the plurality of regions of interest. Also, rest region 83 may not be provided.

[0075] <Explanation of flowchart> Hereinafter, with reference to the flowcharts (FIGS. 11, 12, and 17), the method of determining the imaging area 81 and the attention area 82 will be described. FIG. 11 is a flowchart for explaining the flow of the control process of the camera 3 executed by the control device 4. A program for executing the process according to the flowchart of FIG. 11 is stored in the storage unit 4b of the control device 4. When the power supply is started from the vehicle 1 or the engine is started, for example, the control device 4 starts a program for performing the process according to FIG. 11.

[0076] In step S10 of FIG. 11, the control device 4 determines whether the flag a = 0. The flag a is a flag set to 1 when the initial setting is completed and 0 when the initial setting is not completed. When the flag a = 0, the control device 4 makes an affirmative determination in step S10 and proceeds to step S20. When the flag a ≠ 0, the control device 4 makes a negative determination in step S10 and proceeds to step S30.

[0077] In step S20, the control device 4 performs the initial setting process and proceeds to step S30. Details of the initial setting process will be described later. In step S30, the control device 4 performs the driving assist setting process and proceeds to step S40. In the driving assist setting process, the imaging area 81 and the attention area 82 are determined for the imaging element 100. Details of the driving assist setting process will be described later.

[0078] In step S40, the control device 4 sends an instruction to the camera 3 to drive the imaging area 81 and the attention area 82 in the imaging element 100 under predetermined conditions respectively to acquire an image. In the present embodiment, for example, when the vehicle speed V increases from 0, the control device 4 increases the frame rate, increases the gain, decreases the decimation rate, and shortens the accumulation time of the attention area 82 compared to the imaging area 81. Thereby, imaging by the camera 3 is performed, and distance measurement (range measurement) is performed at a plurality of positions on the imaging screen as described above. Note that it is not necessary to make all of the frame rate, gain, decimation rate, accumulation time, etc. different between the imaging region 81 and the region of interest 82, and it is sufficient to make at least one of them different. Note that the control device 4 may be set not to perform decimation for the region of interest 82.

[0079] In step S45, the control device 4 acquires image data and distance measurement data from the camera 3 and proceeds to step S50. In step S50, the control device 4 determines whether or not a setting for displaying information has been made. If the display setting has been made, the control device 4 makes an affirmative determination in step 50 and proceeds to step S60. If the display setting has not been made, the control device 4 makes a negative determination in step 50 and proceeds to step S70.

[0080] In step S60, the control device 4 sends display information to the display device 14 (FIG. 1) and proceeds to step S70. The display information is information corresponding to the state of the vehicle 1 determined in the driving assist setting process (S30), and causes the display device 14 to display messages such as "Stopped", "Emergency stop", "Turn right", and "Turn left". Note that instead of sending the display information, or together with the sending of the display information, an audio signal for causing the above-described message to be reproduced may be sent to an audio reproduction device (not shown). Also in this case, as the audio reproduction device (not shown), the audio device of a navigation device (not shown) may be used.

[0081] In step S70, the control device 4 determines whether or not it has been turned off. For example, when the control device 4 receives an off signal (for example, an engine off signal) from the vehicle 1, it makes an affirmative determination in step S70, performs a predetermined off process, and ends the process according to FIG. 11. If the control device 4 does not receive an off signal from the vehicle 1, for example, it makes a negative determination in step S70 and proceeds to step S80. In step S80, the control device 4 waits for a predetermined time (for example, 0.1 second) and returns to step S30. When returning to step S30, the above-described process is repeated.

[0082] <Initial setting process> Figure 12 is a flowchart for explaining the details of the initial setting process in step S20 of the flowchart in Figure 11. In step S21 of Figure 12, the control device 4 inputs the position information of the vehicle 1 from the GPS device 15 (Figure 1) and proceeds to step S22. In step S22, the control device 4 sets a flag indicating whether the traffic lane on which the vehicle 1 travels is on the left or right of the road, that is, whether it is left-hand traffic or right-hand traffic, based on the latitude and longitude included in the position information. Specifically, the country name in which the vehicle 1 is used is determined based on the latitude and longitude. Then, referring to a database (not shown), a flag indicating whether the roads in the country are left-hand traffic or right-hand traffic is set. The database showing the relationship between the country name and the left or right of the traffic lane is stored in the storage unit 4b in advance.

[0083] In step S23, the control device 4 sets a flag indicating the attachment position (right or left) of the steering wheel (steering wheel 10) in the vehicle 1 and proceeds to step S24. Information indicating whether it is a right-hand steering wheel or a left-hand steering wheel is stored in the storage unit 4b in advance as the specifications information of the vehicle 1. In step S24, the control device 4 determines the initial setting value based on the table illustrated in Figure 13 and proceeds to step S25. Note that the order of steps S21 and S23 may be swapped.

[0084] According to Figure 13, four initial setting values from "1" to "4" are prepared according to the combination of the attachment position (right or left) of the steering wheel 10 in the vehicle 1 and the position (right or left) of the traffic lane on the road. When it is a right-hand steering wheel and left-hand traffic, the initial setting value is "4".

[0085] In step S25, the control device 4 sets the initial position of the target area 82. The initial position of the target area 82 shall be a position corresponding to the initial setting value. Specifically, when the initial setting value is "1", the initial position of the target area 82 is set as (Xq1, Yq); when the initial setting value is "2", the initial position of the target area 82 is set as (Xq2, Yq); when the initial setting value is "3", the initial position of the target area 82 is set as (Xq3, Yq); when the initial setting value is "4", the initial position of the target area 82 is set as (Xq4, Yq).

[0086] In this description, in the coordinate system representing the imaging area 81, the position of the target area 82 is represented by the coordinates (Xq, Yq) at the center of the target area 82. FIG. 10 illustrates the target area 82 when the initial setting value is "4". Since it is left-hand drive and left-side traffic, the initial position (Xq4, Yq) is determined so that the target area 82 is set on the driver's seat side (right side) within the left-side traffic lane.

[0087] FIG. 14 illustrates the target area 82 when the initial setting value is "1". Since it is right-hand drive and right-side traffic, the initial position (Xq1, Yq) is determined so that the target area 82 is set on the driver's seat side (left side) within the right-side traffic lane.

[0088] FIG. 15 illustrates the target area 82 when the initial setting value is "3". Since it is left-hand drive and left-side traffic, the initial position (Xq3, Yq) is determined so that the target area 82 is set on the driver's seat side (left side) within the left-side traffic lane.

[0089] FIG. 16 illustrates the target area 82 when the initial setting value is "2". Since it is right-hand drive and right-side traffic, the initial position (Xq2, Yq) is determined so that the target area 82 is set on the driver's seat side (right side) within the right-side traffic lane.

[0090] In step S26 of FIG. 12, the control device 4 sets the initial size of the target area 82. In the present embodiment, the initial size (Px (X-axis direction) × Py (Y-axis direction)) of the target area 82 is determined based on the size (dimensions) of the object (for example, the preceding vehicle). When the image acquired by the camera 3 includes the preceding vehicle, the control device 4 estimates the size (dimensions) of the preceding vehicle based on the image height of the preceding vehicle imaged by the imaging chip 113, the known focal length of the imaging optical system 31, and the distance L from the vehicle 1 to the preceding vehicle obtained by distance measurement. Then, the number of pixels (Px (X-axis direction) × Py (Y-axis direction)) that make up the image obtained on the imaging chip 113 when imaging the preceding vehicle estimated to be of the size (for example, width 3 (m) × height 1.4 (m)) from 1 (m) behind is set as the initial size.

[0091] Px and Py are calculated by the following equations (1) and (2). Px = ox×L …(1) Py = oy×L …(2) However, ox is the number of pixels in the X-axis direction that make up the image of the preceding vehicle imaged by the imaging chip 113 at a distance of L (m). oy is the number of pixels in the Y-axis direction that make up the image of the preceding vehicle imaged by the imaging chip 113 at a distance of L (m). L is the inter-vehicle distance from the vehicle 1 to the preceding vehicle. In the coordinate system representing the imaging area 81, the above Yq representing the initial position corresponds to the center of the height of the image obtained on the imaging chip 113 when imaging the preceding vehicle 1 (m) away (in this example, the part with a height of 0.7 (m) of the preceding vehicle).

[0092] In step S27, the control device 4 sends display information to the display device 14 (FIG. 1), sets 1 in the flag a, and ends the process according to FIG. 12. The display information is information indicating that the initial setting process has ended, and for example, causes the message "Initial setting completed" to be displayed on the display device 14.

[0093] <Driving assist setting process> FIG. 17 is a flowchart for explaining the details of the travel assist setting process. In step S310 of FIG. 17, when the position information of the shift lever input from the shift lever position detection device 16 (FIG. 1) is "P" (parking), the control device 4 makes an affirmative determination in step S310 and proceeds to step S320. When the position information of the shift lever input from the shift lever position detection device 16 (FIG. 1) is not "P", the control device 4 makes a negative determination in step S310 and proceeds to step S420. Note that the determination in step S310 may also be applied when the shift lever is in the "N" (neutral) position.

[0094] In step S320, the control device 4 inputs the vehicle speed V from the vehicle speed sensor 12 and proceeds to step S330. The control device 4 changes the frame rate of the attention area 82 according to, for example, the vehicle speed V. As described above, when the frame rate of the attention area 82 is set higher than the frame rate of the imaging area 81, the control device 4 sets the frame rate of the attention area 82 higher as the vehicle speed V increases, and sets the frame rate of the attention area 82 lower as the vehicle speed V decreases. In this case, the control device 4 may also apply control such that the frame rate of the imaging area 81 other than the attention area 82 is also proportional to the vehicle speed V. In step S330, the control device 4 inputs the depression amount of the brake pedal 8a from the brake control device 8 (FIG. 1) and proceeds to step S340.

[0095] In step S340, the control device 4 determines whether the flag Em = 0 based on the vehicle speed V and the depression amount (depression angle) of the brake pedal 8a. The flag Em is a flag set based on the vehicle speed V and the change amount of the depression amount (depression angle) of the brake pedal 8a, as illustrated in FIG. 18. In the present embodiment, the case where Em = 1 is determined as an emergency brake (sudden brake), and the case where Em = 0 is determined as a normal brake. When Em = 0, the control device 4 makes an affirmative determination in step S340 and proceeds to step S350. When Em = 1, the control device 4 makes a negative determination in step S340 and proceeds to step S430. Note that instead of the amount of change (depression angle) of the depression of the brake pedal 8a, Em = 1 may be determined based on the amount of change in the opening degree of a brake valve (not shown). Further, Em = 1 may be determined based on the amount of change in the vehicle speed V, or Em = 1 may be determined based on the amount of change in the reduction ratio of a transmission (not shown).

[0096] In step S350, the control device 4 inputs the depression amount of the accelerator pedal 7a from the throttle control device 7 (FIG. 1) and proceeds to step S360. In step S360, the control device 4 inputs the rotation angle θ of the steering wheel 10 from the steering control device 9 and proceeds to step S370. In step S370, the control device 4 determines whether or not a steering operation has been performed. When the rotation angle θ is greater than a predetermined value, the control device 4 makes an affirmative determination in step S370 and proceeds to step S380. When the rotation angle θ is less than or equal to the predetermined value, the control device 4 makes a negative determination in step S370 and proceeds to step S440.

[0097] In step S380, the control device 4 calculates the movement amount Xdist of the attention area 82 in the X-axis direction based on the rotation angle θ of the steering wheel 10 and the vehicle speed V by the following formula (3). Xdist = θ×(V×0.2) …(3) According to the above formula (3), the larger the steering angle of the steering device (that is, the rotation angle θ of the steering wheel 10) and the larger the vehicle speed V, the larger the movement amount Xdist becomes.

[0098] In step S390, the control device 4 calculates the position (X coordinate) of the attention area 82 during traveling based on the initial position (XqN, Yq) of the attention area 82 set in the initial setting process by the following formula (4). Xq =XqN +Xdist …(4) However, N is any value from the initial setting values 1 to 4 determined in the initial setting process. Xdist is the movement amount of the target area 82 in the X-axis direction calculated in step S380, corresponding to the number of pixels in the X-axis direction. By the process of step S390, the position of the target area 82 changes according to the steering operation. Also, the position of the target area 82 changes depending on the magnitude of the vehicle speed V.

[0099] In step S400, the control device 4 calculates the position (Y coordinate) of the target area 82 during traveling based on the initial position (XqN, Yq) of the target area 82 set in the initial setting process by the following equation (5). Yq = Yq + P(Z) …(5) However, P(Z) is the movement amount of the target area 82 in the Y-axis direction, which is the number of pixels in the Y-axis direction corresponding to the depth Z (m). For example, it represents how many pixels in the Y-axis direction the image of a road with a depth of 20 (m) corresponds to. The relationship P(Z) between the depth Z and the number of pixels is stored in advance in the storage unit 4b (Figure 2).

[0100] Generally, when imaging the traveling direction on a flat straight road, the number of pixels in the Y-axis direction corresponding to the image of the road on the imaging chip 113 increases as the depth Z (m) from the vehicle 1 increases. Therefore, the value of Yq corresponding to the center of the height of the image when imaging a preceding vehicle 1 (m) ahead is increased as the preceding vehicle to be noted becomes farther away (i.e., the depth Z increases).

[0101] The control device 4 determines the depth Z of the preceding vehicle to be noted by the following equation (6). Z = Za + Zb …(6) However, Za is the braking distance (m) on a dry road, and Zb is the braking distance (m) on a wet road surface. Za and Zb are based on the values illustrated in FIG. 19. In the present embodiment, the position of the attention area 82 is determined so as to include a preceding vehicle at a depth Z (that is, a position Z (m) away from the vehicle 1) in front of the vehicle 1 in the attention area 82. This is based on the idea of paying attention to a distance farther than the distance required for stopping when applying an emergency brake. The value of the depth Z (Za + Zb) corresponding to the vehicle speed V is stored in advance in the storage unit 4b (FIG. 2). According to the process of step S400, the position of the attention area 82 changes according to the change in the vehicle speed V. For the attention area 82 whose position has been changed in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. is made different between the imaging area 81 and the attention area 82.

[0102] In step S410, based on the initial size (Px × Py) of the attention area 82 set in the initial setting process, the control device 4 calculates the size (X_wid, Y_wid) of the attention area 82 during running by the following expressions (7) and (8), and ends the process according to FIG. 17. X_wid = Px / Z …(7) Y_wid = Py / Z …(8) Here, Px is the number of pixels in the X-axis direction set in step S26, and Py is the number of pixels in the Y-axis direction set in step S26. According to the above expressions (7) and (8), the size (X_wid, Y_wid) of the attention area 82 during running becomes smaller than the initial size (Px × Py) of the attention area 82 as the preceding vehicle to be noticed moves farther away (the depth Z becomes deeper). According to the process of step S410, the size of the attention area 82 changes according to the change in the vehicle speed V. For the attention area 82 whose size has been changed in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. is made different between the imaging area 81 and the attention area 82.

[0103] In step S420, where the determination of step S310 described above is negative and the process proceeds, the control device 4 performs a setting process during stoppage and ends the process according to FIG. 17. The setting process during stoppage determines, for example, the position of the attention area 82 so as to include a preceding vehicle 1 m ahead in the attention area 82. Also, the size of the attention area 82 in the X-axis direction is maximized so that an object at a position close to the side portion of the vehicle 1 is also included in the attention area 82 as much as possible.

[0104] In step S430, where the determination of step S340 described above is negative and the process proceeds, the control device 4 performs a setting process at the time of sudden braking determination and ends the process according to FIG. 17. The setting process at the time of sudden braking determination stops, for example, thinning in the attention area 82, maximizes the frame rate, shortens the accumulation time, and sets the gain high. Note that the control device 4 may also increase the frame rate of the imaging area 81 other than the attention area 82. Further, the control device 4 gives an instruction to the camera 3 to save the image acquired by the camera 3 in the recording unit 36 for a predetermined time (for example, 5 seconds to 15 seconds) after the determination of step S340 is negative.

[0105] After the sudden stop, the control device 4 further moves the attention area 82 to the initial position of the attention area 82 set in the initial setting process (step S25 in FIG. 12) and changes the size of the attention area 82 to the initial size (Px × Py) of the attention area 82 set in the initial setting process (step S26 in FIG. 12). Thereby, the position and size of the attention area 82 that have changed according to the vehicle speed V during traveling return to positions and sizes suitable for the stop.

[0106] In step S440, where the determination of step S370 described above is negative and the process proceeds, the control device 4 performs a setting not to move the position (X coordinate) of the attention area 82 during traveling. That is, when the rotation angle θ of the steering wheel 10 is equal to or less than a predetermined value, θ←0 and the value of Xdist is also set to 0. That is, when the operation angle of the steering wheel 10 is less than the predetermined value, the position (X coordinate) of the attention area 82 is maintained. This helps to reduce the processing load during a minute operation that is not a turning operation.

[0107] FIG. 20(a) is a diagram illustrating the movement of the position of the attention area 82 and the change in the size of the attention area 82 when turning right at an intersection on a general road. According to the above driving assist setting process, when the vehicle 1 is waiting to turn right behind the preceding vehicle, the position of the attention area 82A is at the initial position, and the size of the attention area 82A is substantially the same as the initial size (Px×Py). When the driver starts a steering operation to the right while the vehicle 1 is moving forward, the position of the attention area 82B moves diagonally upward to the right. Since the vehicle speed V is low, the size of the attention area 82B is also substantially the same as the initial size (Px×Py).

[0108] FIG. 20(b) is a diagram illustrating the movement of the position of the attention area 82 and the change in the size of the attention area 82 when changing lanes while accelerating into the right passing lane on an expressway. According to the above driving assist setting process, when the vehicle 1 is traveling at a high speed, the position of the attention area 82A is above the initial position, and the size of the attention area 82A is smaller than the initial size (Px×Py). When the driver performs a steering operation to the right while the vehicle 1 is accelerating, the position of the attention area 82B moves diagonally upward to the right. Since the vehicle speed V is high, the size of the attention area 82B becomes even smaller. Note that FIG. 20 is an example for the case of driving on the left side, and it can also be appropriately used for a left turn on the right side or a lane change on the right side. Further, a line-of-sight detection device (not shown) (for example, a line-of-sight detection device is provided on the steering wheel) may be used to detect the driver's line of sight, and an area that the driver is not looking at or a blind spot area may be set as the attention area 82. Note that for line-of-sight detection, there are a corneal reflection method that reflects infrared rays from the driver's cornea to detect the user's line-of-sight direction, a limbus tracking method that utilizes the difference in reflectance of light between the cornea and the sclera, an image analysis method that captures an image of the eyeball with a camera and detects the line of sight by image processing, etc., and any line-of-sight detection method may be used.

[0109] According to the above-described embodiment, the following operational effects can be obtained. (1) A control device 4 that recognizes at least one of the specifications of the vehicle 1 on which it is mounted and the operations on the operation unit of the vehicle 1, and an imaging unit 32 that has at least a target area 82 and an imaging area 81 and images the outside of the vehicle 1. Since the control device 4 sets different imaging conditions for the target area 82 and the imaging conditions for the imaging area 81 based on the recognition result, the imaging conditions of the camera 3 can be set appropriately.

[0110] (2) Since the control device 4 recognizes the mounting position (right or left) of the steering wheel (steering wheel 10) in the vehicle 1, the imaging conditions of the camera 3 can be set appropriately according to the driver's seating position.

[0111] (3) Since the setting unit sets different frame rates for the target area 82 and the frame rate for the imaging area 81 according to the position of the steering wheel 10, for example, increasing the frame rate in the target area 82 on the driver's seat side (right), the imaging conditions of the camera 3 can be set appropriately.

[0112] (4) It is provided with a control device 4 that detects information regarding the vehicle speed V of the vehicle 1, and the control device 4 sets different imaging conditions for the target area 82 and the imaging conditions for the imaging area 81 according to the detection result of the information regarding the vehicle speed V. Therefore, the imaging conditions of the camera 3 can be set appropriately according to the vehicle speed V.

[0113] (5) When the information regarding the vehicle speed V increases and when it decreases, the control device 4 changes at least one of the imaging conditions for the target area 82 and the imaging conditions for the imaging area 81. Therefore, for example, the higher the vehicle speed V, the higher the frame rate can be increased, and the imaging conditions of the camera 3 can be set appropriately.

[0114] (6) When the rotation angle θ of the steering wheel (steering wheel 10) exceeds a predetermined value, the control device 4 increases at least one of the frame rates of the imaging of the target area 82 and the frame rate of the imaging of the imaging area 81. Therefore, in the case of a turning operation, the imaging conditions of the camera 3 can be changed.

[0115] (7) Since the control device 4 is provided to transmit display information to the display device 14 of the vehicle 1 based on the imaging result by the imaging unit 32, necessary information can be provided to the passengers of the vehicle 1.

[0116] (8) When the rotation angle θ of the steering wheel (steering wheel 10) does not reach a predetermined value, the control device 4 maintains the setting of at least one of the frame rate of imaging in the attention area 82 and the frame rate of imaging in the imaging area 81, so that it is possible to avoid changing the imaging conditions during a minute operation that is not a turning operation. Thereby, for example, it is possible to prevent the frame rate of the attention area 82 from being changed more finely than necessary, which helps to reduce the processing burden.

[0117] (9) The control device 4 includes at least one of the frame rate of imaging, gain, decimation, pixel signal addition, accumulation, bit length, size of the imaging area, and position of the imaging area in the imaging conditions that are made different between the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81, so that the imaging conditions of the camera 3 can be appropriately set.

[0118] (10) The control device 4 changes at least one of the center positions of the attention area 82 and the center position of the imaging area 81 based on the detection result of information regarding the vehicle speed V, so that the position of the attention area 82 can be changed as the vehicle speed V changes, etc., and the imaging conditions of the camera 3 can be appropriately set.

[0119] (11) The control device 4 changes at least one of the size of the attention area 82 and the size of the imaging area 81 based on the detection result of information regarding the vehicle speed V, so that the size of the attention area 82 can be changed as the vehicle speed V changes, etc., and the imaging conditions of the camera 3 can be appropriately set.

[0120] (12) The control device 4 sets the imaging area 81 surrounding the attention area 82, so that the imaging conditions of the camera 3 can be appropriately set.

[0121] (13) The vehicle 1 is provided with a steering wheel (steering wheel 10) as an operating unit, and the control device 4 changes at least one of the center position of the attention area 82 and the center position of the imaging area 81 based on the operation of the steering wheel, so that it is possible to appropriately set the imaging conditions of the camera 3, such as changing the position of the attention area 82 in accordance with a change in the course of the vehicle 1. Note that, in the above-mentioned embodiment, the camera 3 is controlled by the control device 4, but part of the control of the camera 3 may be performed by the control unit 35 of the camera 3.

[0122] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 1) In the driving assist setting process, the control device 4 may be configured to change the position and size of the attention area 82 in response to an operation signal from the turn signal switch 11. As illustrated in Fig. 21, the control device 4 changes the size of the attention area 82 and sets the imaging conditions for the attention area 82 based on the initial setting value determined in step S24 and the turn signal direction by the operation of the turn signal switch 11.

[0123] For example, referring to FIG. 10, in the case of right-hand drive, left-hand traffic, and the initial setting value being "4", the control device 4 performs control so that the left edge of the road is included in the attention area 82 if the turn signal is to the left. Specifically, the attention area 82 in FIG. 10 is expanded to the left. The reason for expanding the attention area 82 to the left is to prevent a collision accident when turning left. Conversely, the control device 4 performs control so that the oncoming lane is included in the attention area 82 if the turn signal is to the right. Specifically, the attention area 82 in FIG. 10 is expanded to the right.

[0124] Explaining with reference to Fig. 14, in the case of a left-hand drive vehicle, right-hand traffic, and the initial setting value being "1", the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the left. Specifically, the attention area 82 in Fig. 14 is expanded to the left. Conversely, the control device 4 performs control so that the attention area 82 includes the right edge of the road if the turn signal is to the right. Specifically, the attention area 82 in Fig. 14 is expanded to the right. Expanding to the right is to prevent a hit-and-run accident when turning right.

[0125] Explaining with reference to Fig. 16, in the case of a vehicle with right-hand drive and right-hand traffic, where the initial setting value is "2", the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the left. Specifically, the attention area 82 in Fig. 16 is significantly expanded to the left. Conversely, the control device 4 performs control so that the attention area 82 includes the right edge of the road if the turn signal is to the right. Specifically, the attention area 82 in Fig. 16 is expanded slightly to the right. Expanding to the right is to prevent a hit-and-run accident when turning right.

[0126] Explaining with reference to FIG. 15, in the case of a left-hand drive vehicle, left-hand traffic, and the initial setting value being "3", the control device 4 performs control so that the left edge of the road is included in the attention area 82 if the turn signal is to the left. Specifically, the attention area 82 in FIG. 15 is expanded slightly to the left. The expansion to the left is to prevent collision accidents when turning left. Conversely, the control device 4 performs control so that the attention area 82 includes the oncoming lane if the turn signal is to the right. Specifically, the attention area 82 in FIG. 15 is expanded significantly to the right.

[0127] FIG. 22 is a flowchart for explaining the processing when operating the turn signal switch 11 according to Modification 1. When an operation signal is input from the turn signal switch 11 during the running assist setting process, the control device 4 activates the processing according to FIG. 22 as a subroutine. In step S510 of FIG. 22, the control device 4 determines whether the turn signal direction is leftward. When the turn signal direction is leftward, the control device 4 makes an affirmative determination in step S510 and proceeds to step S520. When the turn signal direction is rightward, the control device 4 makes a negative determination in step S510 and proceeds to step S530.

[0128] In step S520, the control device 4 determines whether the position of the traffic lane is on the left. When driving on the left side, the control device 4 makes an affirmative determination in step S520 and proceeds to step S550. When driving on the right side, the control device 4 makes a negative determination in step S520 and proceeds to step S540.

[0129] In step S540, the control device 4 controls the imaging unit 32 to include the oncoming lane in the attention area 82 and ends the processing according to FIG. 22. In step S550, the control device 4 controls the imaging unit 32 to include the left end of the road in the attention area 82 and ends the processing according to FIG. 22.

[0130] In step S530, the control device 4 determines whether the position of the traffic lane is on the left. When driving on the left side, the control device 4 makes an affirmative determination in step S530 and proceeds to step S560. When driving on the right side, the control device 4 makes a negative determination in step S530 and proceeds to step S570.

[0131] In step S560, the control device 4 controls the imaging unit 32 to include the oncoming lane in the attention area 82 and ends the processing according to FIG. 22. In step S570, the control device 4 controls the imaging unit 32 to include the right end of the road in the attention area 82 and ends the processing according to FIG. 22.

[0132] When the turn signal switch 11 is turned off after executing the process in Fig. 22, the control device 4 cancels the size change of the attention area 82 in Fig. 22. When the turn signal switch 11 is turned on, the frame rate, gain, thinning rate, and accumulation time of the attention area 82 may be set higher than those of the imaging area 81, even if the vehicle speed V is 0. However, when at least one of the frame rate, gain, thinning rate, accumulation time, etc. is to be made different between the imaging area 81 and the attention area 82, only the imaging condition to be made different is changed.

[0133] According to the above-described first modification, the imaging conditions of the attention area 82 and the imaging conditions of the imaging area 81 are set differently in response to the operation of the turn signal switch 11, so that, for example, when turning right or left at an intersection, the attention area 82 can be appropriately set by including an oncoming lane in the attention area 82 so that an oncoming vehicle can be reliably detected, or by including a road edge in the attention area 82 so that an entrapment accident can be prevented. Furthermore, the imaging conditions can be appropriately set in the imaging area 81 and the attention area 82, for example, by setting the frame rate of the attention area 82 higher than that of the imaging area 81.

[0134] (Variation 2) In the driving assistance setting process, the control device 4 may be configured to change the position of the attention area 82 and the size of the attention area 82 in accordance with changes in the distance between the vehicle 1 and an object such as a two-wheeled vehicle, a regular vehicle, a large vehicle, or a pedestrian.

[0135] In the second modification, for example, when vehicle 1 approaches a preceding vehicle and the distance L (inter-vehicle distance) to the preceding vehicle becomes shorter, control device 4 determines the position of attention area 82 so that the preceding vehicle is included in attention area 82. Here, the change in the inter-vehicle distance L from vehicle 1 to the preceding vehicle is obtained by distance measurement calculation unit 35a of camera 3 successively measuring distance L (inter-vehicle distance) to the preceding vehicle within the captured screen based on images acquired by camera 3 at predetermined time intervals synchronized with the acquisition timing of vehicle speed V.

[0136] The control device 4 calculates the position (Y coordinate) of the target area 82 during traveling by using the inter-vehicle distance L instead of the depth Z in the above formula (5). As a result, when imaging the preceding vehicle on a flat straight road, the value of Yq indicating the position of the target area 82 in the Y-axis direction on the imaging chip 113 increases as the inter-vehicle distance L increases, and decreases as the inter-vehicle distance L decreases.

[0137] Also, when the inter-vehicle distance L changes and becomes shorter, the preceding vehicle appears larger in the camera 3, so the control device 4 sets the size of the target area 82 to be larger. Conversely, when the inter-vehicle distance L changes and becomes longer, the preceding vehicle appears smaller in the camera 3, so the control device 4 sets the size of the target area 82 to be smaller. The control device 4 substitutes the inter-vehicle distance L instead of the depth Z in the above formulas (7) and (8) to calculate the size of the target area 82 during traveling.

[0138] According to the processing according to FIG. 17 described above, when the vehicle speed V increases, the size of the target area 82 is set to be smaller. In the second modification, however, when the inter-vehicle distance L to the preceding vehicle is short even if the vehicle speed V is high, the size of the target area 82 is set to be larger, so that the preceding vehicle can be appropriately included in the target area 82. Therefore, compared with the case where the size of the target area 82 is continuously set to be smaller, it becomes easier to detect changes in the traveling state of the preceding vehicle based on the image acquired by the camera 3. Regarding the target area 82 whose size and position are changed in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the target area 82.

[0139] (Second Modification) When the control device 4 detects an object around the vehicle 1 such as a two-wheeled vehicle, a passenger vehicle, a large vehicle, or a pedestrian, in addition to the existing target area 82, the control device 4 may newly set a target area 82 including this object. In Modification 3, when the detected object moves, the control device 4 newly sets a target area 82 including this object. For example, when the distance between the detected object and the vehicle 1 approaches within a predetermined distance, the control device 4 newly sets a target area 82 including this object. Then, when the distance between the detected object and the vehicle 1 exceeds the predetermined distance, the control device 4 cancels the setting of the target area 82 including this object. Regarding the target area 82 set in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the target area 82.

[0140] According to Modification 3, a control device 4 that detects a moving object around the vehicle 1 based on information from the camera 3 is provided. The control device 4 changes at least one imaging condition of the imaging condition of the target area 82 and the imaging condition of the imaging area 81 based on the detection result of the moving object. Therefore, the imaging condition of the camera 3 can be appropriately set according to the presence or absence of the moving object.

[0141] Also, when it is detected that the distance between the vehicle 1 and the moving object approaches within a predetermined distance based on the information from the camera 3, the control device 4 increases at least one frame rate of the imaging frame rate of the target area 82 and the imaging frame rate of the imaging area 81. Therefore, it becomes easy to detect a change in the moving state of the moving object based on the image acquired by the camera 3.

[0142] (Modification 4) The target area 82 may be newly set based on the color of the image acquired by the camera 3. The control device 4 sets an area including a red object in the image as the target area 82. By adding an area including a red object to the target area 82, for example, a red signal, a warning device at a railway crossing, a red light of an emergency vehicle, etc. can be included in the target area 82. Regarding the attention area 82 set in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the attention area 82.

[0143] (Modification Example 5) The attention area 82 may be newly set based on the sound information collected by the microphone 17 of the vehicle 1. For example, when the level of the sound information on the right side of the vehicle 1 is input exceeding a predetermined value, the control device 4 expands the attention area 82 to the right in the imaging area 81, or newly sets the attention area 82 to the right in the imaging area 81. The reason for providing the attention area 82 to the right is to collect the out-of-vehicle information on the right side of the vehicle 1.

[0144] Also, for example, when the level of the sound information on the left side of the vehicle 1 is input exceeding a predetermined value, the control device 4 expands the attention area 82 to the left in the imaging area 81, or newly sets the attention area 82 to the left in the imaging area 81. The reason for providing the attention area 82 to the left is to collect the out-of-vehicle information on the left side of the vehicle 1. Regarding the attention area 82 set in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the attention area 82.

[0145] (Modification Example 6) In the above embodiment, the case of setting the attention area 82 including the preceding vehicle has been described, but the attention area 82 including the oncoming vehicle of the vehicle 1 may be set. In Modification Example 6, the control device 4 recognizes the vehicle closest to the vehicle 1 as the oncoming vehicle from among the objects traveling in the reverse direction (opposite to the vehicle 1) existing in the above-described traveling area.

[0146] The control device 4 sets, as the attention area 82, the area corresponding to the position of the oncoming vehicle in the image acquired by the camera 3. The control device 4 sets the attention area 82 including, in particular, the license plate of the oncoming vehicle and the face of the driver driving in the driver's seat of the oncoming vehicle.

[0147] By setting the area including the license plate of the oncoming vehicle and the face of the driver of the oncoming vehicle as the attention area 82, the oncoming vehicle approaching the vehicle 1 can be appropriately included in the attention area 82. Regarding the attention area 82 set in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the attention area 82.

[0148] (Modification Example 7) In the above description, an example where the imaging area 81 includes (surrounds) the attention area 82 has been described, but the imaging area 81 and the attention area 82 may be set side by side left and right. By moving the boundary line between the imaging area 81 and the attention area 82 left and right, the sizes and positions of the imaging area 81 and the attention area 82 can be changed. Regarding the attention area 82 set in this way, at least one of the frame rate, gain, decimation rate, accumulation time, etc. may be made different between the imaging area 81 and the attention area 82.

[0149] In the above description, as the distance measurement performed by the camera 3, a method of calculating by distance measurement operation using the image signal from the focus detection pixels provided in the imaging device 100 has been used, but a method of performing distance measurement using two images by a stereo camera may also be used. Further, a method of performing distance measurement using a millimeter wave radar separately from the camera 3 may also be used.

[0150] In the above, various embodiments and modification examples have been described, but the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

Explanation of Reference Numerals

[0151] 1... Vehicle 2... Driving Support Device 3... Camera 4... Control Device 4b... Storage Unit 5... First Travel Control Unit 6... Second Travel Control Unit 7... Throttle Control Device 7a... Accelerator Pedal 8… Brake control device 8a… Brake pedal 9… Steering control device 10… Steering wheel 11… Turn signal switch 12… Vehicle speed sensor 14… Display device 15… GPS device 16… Shift lever position detection device 17… Microphone 31… Imaging optical system 32… Imaging unit 32a… Driving unit 33… Image processing unit 34… Work memory 35… Control unit 35a… Distance measurement calculation unit 36… Recording unit 60… Focus detection pixel line 81… Imaging area 82, 82A, 82B… Region of interest 83… Rest area 100… Image sensor 113… Imaging chip

Claims

1. An imaging unit that images the exterior of the vehicle, A distance measuring unit that measures the distance to an object existing outside the vehicle, A positioning unit that measures the current position of the vehicle, A control unit that generates driving support data, which is data for performing driving support of the vehicle based on information from the imaging unit, the distance measuring unit, and the positioning unit, The control unit classifies and extracts a two-wheeled vehicle, a passenger vehicle, a large vehicle, a pedestrian, and a utility pole from the image captured by the imaging unit, sets a region of interest including the extracted two-wheeled vehicle, passenger vehicle, large vehicle, and pedestrian, and generates the driving support data using the information of the object of interest for which the region of interest is set. A driving support device.

2. The driving support device according to claim 1, further comprising a vehicle speed detection unit that detects the speed of the vehicle, The control unit changes the range of the object of interest for imaging based on information from the vehicle speed detection unit. The driving support device according to claim 1.

Images