Imaging device

Abstract

To properly set imaging conditions of an imaging apparatus.SOLUTION: The imaging apparatus includes an imaging unit mounted on a vehicle and imaging the exterior of the vehicle, and an imaging control unit for controlling imaging conditions of the imaging unit based on the position of a steering wheel of the vehicle.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] There has been developed a technique for detecting a driving environment of a vehicle based on an image acquired by a camera mounted on the vehicle, and performing automatic driving control such as following driving of a preceding vehicle or driving support such as warning, braking, and steering assistance based on the detected driving environment data (see Patent Document 1). In the prior art, a solid-state imaging device such as a CCD is used for an in-vehicle camera. An in-vehicle camera that continuously acquires images of roads and the like plays an important role in automatic driving control and driving support. However, there have been few proposals for cameras on the premise of mounting on a vehicle, and the usability of the camera has not been sufficient.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] The imaging device according to the present invention is an imaging device mounted on a vehicle, and includes an input unit that inputs information related to the speed of the vehicle, an imaging unit that images the outside of the vehicle, and an imaging control unit that changes the imaging conditions of the imaging unit when the information related to the speed of the vehicle changes. Based on information regarding the vehicle's speed, the imaging control unit sets the frame rate of the area of ​​interest, which is a portion of the imaging area of ​​the imaging unit, lower as the vehicle's speed decreases. However, even if the vehicle's speed decreases, if the imaging control unit determines that sudden braking has occurred, it increases the frame rate of the area of ​​interest. .

Brief Description of the Drawings

[0005] [Figure 1] It is a schematic configuration diagram of a vehicle driving support device. [Figure 2] It is a block diagram illustrating the configuration of a control device. [Figure 3] It is a cross-sectional view of a stacked imaging device. [Figure 4] It is a diagram for explaining a pixel array and a unit region of an imaging chip. [Figure 5] This is a diagram illustrating a circuit in a unit region. [Figure 6] This is a block diagram showing the functional configuration of an image sensor. [Figure 7] This diagram illustrates the position of focus detection pixels on the imaging surface of an image sensor. [Figure 8] This is a magnified view of the region containing part of the focus detection pixel line. [Figure 9] This is a block diagram illustrating the configuration of a camera having an image sensor. [Figure 10] This diagram illustrates the imaging surface of an imaging chip, the imaging area, the area of ​​interest, and the rest area. [Figure 11] This is a flowchart illustrating the flow of camera control processing performed by the control unit. [Figure 12] This is a flowchart that explains the details of the initial setup process. [Figure 13] This diagram illustrates a table of initial settings. [Figure 14] This diagram illustrates the imaging surface of an imaging chip, the imaging area, the area of ​​interest, and the rest area. [Figure 15] This diagram illustrates the imaging surface of an imaging chip, the imaging area, the area of ​​interest, and the rest area. [Figure 16] This diagram illustrates the imaging surface of an imaging chip, the imaging area, the area of ​​interest, and the rest area. [Figure 17] This is a flowchart that explains the details of the driving assist setting process. [Figure 18] This is a diagram explaining the flag Em. [Figure 19] This is a diagram illustrating the distance Z. [Figure 20] Figure 20(a) illustrates the movement and size changes of the area of ​​interest when making a right turn at an intersection on a regular road. Figure 20(b) illustrates the movement and size changes of the area of ​​interest when changing lanes while accelerating on a highway. [Figure 21] This diagram illustrates the direction of the turn signal and the resizing of the area of ​​interest. [Figure 22] This is a flowchart illustrating the process when operating the turn signal switch according to Modification 1. [Modes for carrying out the invention]

[0006] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. <Camera usage scenarios> Figure 1 is a schematic diagram of a driver assistance device 2 for a vehicle 1 equipped with a camera 3 according to one embodiment of the present invention. In Figure 1, the driver assistance device 2 is installed in a vehicle 1 such as an automobile. The driver assistance device 2 consists of a camera 3, a control device 4, a first driving control unit 5, a second driving control unit 6, etc. In this description, an example using an internal combustion engine as the drive source is described, but a motor may also be used as the drive source.

[0007] Camera 3 comprises an imaging optical system with multiple lenses and an image sensor (in this embodiment, a stacked image sensor (see Figure 3)), and is mounted, for example, on the front of the ceiling inside the vehicle. Camera 3 is directed forward of the vehicle 1, and its mounting height (distance from the ground to Camera 3) is adjusted to, for example, 1.4 m. Camera 3 acquires an image in the direction of travel of the vehicle 1 and measures the distance to each subject (object) at multiple positions within the captured image based on the acquired image. The distance measurement is calculated by a distance measurement calculation using image signals from focus detection pixels provided in the stacked image sensor. Focus detection pixels and distance measurement will be described later. The image data and distance measurement data acquired by Camera 3 are sent to the control device 4. Note that Camera 3 may be installed outside the vehicle, or the interior and exterior cameras 3 may work together, and the number of cameras 3 may be set as appropriate. For example, the exterior camera 3 may be used for white line detection, which will be described later, and the interior and exterior cameras 3 may work together for object and obstacle recognition.

[0008] As illustrated in FIG. 2, the control device 4 includes a CPU 4a and a storage unit 4b. Based on various programs stored in the storage unit 4b, the CPU 4a performs various calculations using control parameters stored in the storage unit 4b, detection signals from various sensors described later, and the like.

[0009] Based on an instruction from the control device 4, the first driving control unit 5 performs constant-speed driving control and following driving control. The constant-speed driving control is control for driving the vehicle 1 at a constant speed based on a predetermined control program. The following driving control is control for driving the vehicle 1 while maintaining a constant inter-vehicle distance from a preceding vehicle when the speed of the preceding vehicle recognized by the control device 4 is less than or equal to the target speed set for the vehicle 1 during the constant-speed driving control.

[0010] Based on an instruction from the control device 4, the second driving control unit 6 performs driving support control. The driving support control is control for outputting a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road and outputting a brake control signal to the brake control device 8 so as to avoid a collision between the vehicle 1 and an object, based on a predetermined control program.

[0011] FIG. 1 further illustrates a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, a display device 14, a GPS device 15, a shift lever position detection device 16, and a microphone 17.

[0012] The throttle control device 7 controls the opening degree of a throttle valve (not shown) according to the depression amount of an accelerator pedal 7a. Further, the throttle control device 7 also controls the opening degree of the throttle valve according to a throttle control signal sent from the first driving control unit 5. The throttle control device 7 further sends a signal indicating the depression amount of the accelerator pedal 7a to the control device 4.

[0013] The brake control device 8 controls the opening degree of a brake valve (not shown) according to the amount the brake pedal 8a is pressed. The brake control device 8 also controls the opening degree of the brake valve according to the brake control signal from the second driving control unit 6. Furthermore, the brake control device 8 sends a signal indicating the amount the brake pedal 8a is pressed to the control device 4.

[0014] The steering control device 9 controls the steering angle of a steering device (not shown) according to the rotation angle of the steering wheel 10. The steering control device 9 also controls the steering angle of the steering device according to the steering control signal from the second driving control unit 6. Furthermore, the steering control device 9 sends signals indicating the rotation angle of the steering wheel 10 to the first driving control unit 5 and the control device 4, respectively.

[0015] The turn signal switch 11 is a switch for activating a turn signal (indicator signal) device (not shown). The turn signal device is a flashing light-emitting device that indicates a change of lane for vehicle 1. When the turn signal switch 11 is operated by an occupant of vehicle 1, an operation signal is sent from the turn signal switch 11 to the turn signal device, the second driving control unit 6, and the control device 4, respectively. The vehicle speed sensor 12 detects the vehicle speed V of vehicle 1 and sends a detection signal to the first driving control unit 5, the second driving control unit 6, and the control device 4, respectively.

[0016] The yaw rate sensor 13 detects the yaw rate of the vehicle 1 and sends a detection signal to the second driving control unit 6 and the control device 4, respectively. The yaw rate is the rate at which the rotation angle of the vehicle 1 changes in the turning direction. The display device 14 displays information such as the control status by the first driving control unit 5 and the second driving control unit 6. The display device 14 is composed of, for example, a HUD (Head Up Display) that projects information onto the windshield. Note that the display unit of a navigation device (not shown) may be used as the display device 14.

[0017] The GPS device 15 receives radio waves from GPS satellites and calculates the position of the vehicle 1 (latitude, longitude, etc.) by performing predetermined calculations using the information carried by the radio waves. The position information calculated by the GPS device 15 is sent to a navigation device and control device 4 (not shown). The shift lever position detection device 16 detects the position of the shift lever (not shown) operated by the occupant of the vehicle 1 (for example, parking (P), reverse (R), drive (D), etc.). The position information of the shift lever detected by the shift lever position detection device 16 is sent to the control device 4.

[0018] The microphone 17 is composed of, for example, a front microphone, a right-side microphone, and a left-side microphone. The front microphone has a directivity that exclusively collects sound from the front of the vehicle 1. The right-side microphone has a directivity that exclusively collects sound from the right side of the vehicle 1. The left-side microphone has a directivity that exclusively collects sound from the left side of the vehicle 1. The sound information collected by the microphone 17 (front, right side, left side) is sent to the control device 4.

[0019] <Detection of objects> The control device 4 performs the following image processing on the image from the camera 3 in order to detect the vehicle 1's travel path and objects. First, the control device 4 generates a distance image (depth distribution image) based on distance measurement data at multiple positions within the captured image. Based on the data in the distance image, the control device 4 performs a well-known grouping process and compares it with frames (windows) of 3D road shape data, side wall data, object data, etc., which are stored in the storage unit 4b in advance. It extracts white line data (including white line data along the road and white line data crossing the road (stop line: intersection information)), guardrails, curb data, and other side wall data that exist along the road, and also classifies and extracts objects and obstacles into two-wheeled vehicles, regular vehicles, large vehicles, pedestrians, utility poles, and other objects. In this explanation, the white or yellow lines drawn on the road surface are referred to as "white lines." This term also includes both solid and dashed lines.

[0020] <Driving assistance> The control device 4 recognizes the road and any obstructing objects or obstacles based on the information extracted as described above, namely the white line data, guardrail side wall data, and object data, and instructs the second driving control unit 6 to perform the above-mentioned driving assistance control based on the recognition results. In other words, it causes vehicle 1 to travel along the road and avoids collisions with objects.

[0021] <Driving control> The control device 4 estimates the vehicle's path in one of the following four ways, for example. (1) Estimation of the vehicle's path based on white lines If the camera 3 has obtained white line data for both the left and right sides of the road, or for one side of either the left or right, and the shape of the lane in which vehicle 1 is traveling can be estimated from this white line data, the control device 4 will estimate that the vehicle's path is parallel to the white line, taking into account the width of vehicle 1 and its current position within the lane.

[0022] (2) Estimation of the vehicle's path based on data from side walls such as guardrails and curbs If the camera 3 has obtained side wall data for both the left and right sides of the road, or for one of the left or right sides, and the shape of the lane in which vehicle 1 is traveling can be estimated from this side wall data, the control device 4 will estimate that the vehicle's path is parallel to the side wall, taking into account the width of vehicle 1 and its current position within the lane.

[0023] (3) Estimation of the vehicle's path based on the trajectory of the preceding vehicle The control device 4 estimates the vehicle's path based on the past travel trajectory of the preceding vehicle stored in the memory unit 4b. The preceding vehicle refers to the vehicle closest to vehicle 1 among objects traveling in the same direction as vehicle 1.

[0024] (4) Estimation of the vehicle's travel path based on the travel trajectory of vehicle 1 The control device 4 estimates the vehicle's path based on the vehicle's driving state. For example, it estimates the vehicle's path using a turning curvature based on the detection signal from the yaw rate sensor 13 and the detection signal from the vehicle speed sensor 12. The turning curvature Cua is calculated using the formula Cua = dψ / dt / V, where dψ / dt is the yaw rate (the rate of change of the rotation angle in the turning direction) and V is the vehicle speed of vehicle 1.

[0025] The control device 4, in accordance with a predetermined driving control program stored in the memory unit 4b, estimates the driving area of ​​vehicle 1 at the location where each object exists based on the vehicle's path, compares this driving area with the object's location, and determines whether each object is within the driving area. The control device 4 further recognizes the preceding vehicle based on the image captured by the camera 3. That is, the control device 4 identifies the vehicle closest to vehicle 1 from among the objects that are within the driving area and traveling in the forward direction (the same direction as vehicle 1) as the preceding vehicle.

[0026] The control device 4 outputs information on the distance between the preceding vehicle and vehicle 1, and the speed of the preceding vehicle, as external information to the first driving control unit 5. Here, the speed of the preceding vehicle is calculated based on the speed V of vehicle 1 acquired at predetermined intervals and the change in the distance to the preceding vehicle (interval distance) in the captured screen, which is measured based on the image acquired by camera 3 at predetermined intervals in synchronization with the timing of the acquisition of the vehicle speed V.

[0027] The first driving control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a predetermined vehicle speed (target speed) that has been set in advance. As a result, the throttle control device 7 provides feedback control to the opening degree of a throttle valve (not shown), and the vehicle 1 is automatically driven at a constant speed.

[0028] Furthermore, when the first driving control unit 5 is performing constant-speed driving control, if the vehicle speed information of the preceding vehicle input from the control device 4 is less than or equal to the target speed set for vehicle 1, it sends a throttle control signal to the throttle control device 7 based on the inter-vehicle distance information input from the control device 4. Specifically, it sets an appropriate target value for the inter-vehicle distance based on the inter-vehicle distance from vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of vehicle 1, and sends a throttle control signal to the throttle control device 7 so that the inter-vehicle distance measured based on the image acquired by the camera 3 converges to the above target value for the inter-vehicle distance. As a result, the throttle control device 7 provides feedback control of the opening degree of a throttle valve (not shown), causing vehicle 1 to follow the preceding vehicle.

[0029] <Explanation of stacked image sensors> The stacked image sensor 100 provided in the camera 3 described above will now be explained. This stacked image sensor 100 is the same as the one described in Japanese Patent Application No. 2012-139026, which was previously filed by the applicant of this application. Figure 3 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 comprises a back-illuminated imaging chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. These imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0030] As shown in the figure, the incident light mainly enters in the Z-axis positive direction, indicated by the white arrow. In this embodiment, the side of the imaging chip 113 that the incident light enters is referred to as the back surface (imaging surface). Also, as shown in the coordinate axes, the direction to the left of the paper perpendicular to the Z-axis is the X-axis positive direction, and the direction towards the front of the paper perpendicular to the Z-axis and X-axis is the Y-axis positive direction. In some of the following figures, the coordinate axes are displayed with the coordinate axes of Figure 3 as the reference point to indicate the orientation of each figure.

[0031] An example of an imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is located on the back side of the wiring layer 108. The PD layer 106 is arranged two-dimensionally and has a plurality of PDs (photodiodes) 104 that accumulate charge in response to incident light, and transistors 105 provided corresponding to the PDs 104.

[0032] A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. The color filter 102 has multiple types that transmit different wavelength regions and has a specific arrangement corresponding to each PD 104. The arrangement of the color filter 102 will be described later. A set of color filter 102, PD 104, and transistor 105 forms a single pixel.

[0033] Microlenses 101 are provided on the incident light side of the color filter 102, corresponding to each pixel. The microlenses 101 focus the incident light toward the corresponding PD 104.

[0034] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multilayer, and may also be provided with passive and active elements.

[0035] Multiple bumps 109 are arranged on the surface of the wiring layer 108. These multiple bumps 109 are aligned with multiple bumps 109 provided on the opposing surface of the signal processing chip 111, and when the imaging chip 113 and the signal processing chip 111 are pressed together, the aligned bumps 109 are joined together and electrically connected.

[0036] Similarly, multiple bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. When these bumps 109 are aligned with each other and pressure is applied to the signal processing chip 111 and the memory chip 112, the aligned bumps 109 are joined together and electrically connected.

[0037] Furthermore, the joining of bumps 109 is not limited to Cu bump joining by solid-phase diffusion; microbump joining by solder melting may also be employed. Also, it is sufficient to provide approximately one bump 109 per block, for example, as described later. Therefore, the size of the bumps 109 may be larger than the pitch of PD104. In addition, in peripheral regions other than the pixel regions where pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel regions may also be provided.

[0038] The signal processing chip 111 has TSVs (Through-Silicon Electrodes) 110 that connect circuits provided on its front and back surfaces. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0039] Figure 4 illustrates the pixel arrangement and unit region 131 of the imaging chip 113. In particular, it shows the imaging chip 113 as observed from the back (imaging surface) side. The pixel region contains, for example, more than 20 million pixels arranged in a matrix. In the example in Figure 4, 16 adjacent pixels (4x4) form one unit region 131. The grid lines in the figure illustrate the concept that adjacent pixels are grouped together to form a unit region 131. The number of pixels forming a unit region 131 is not limited to this; it could be around 1000, for example, 32x64 pixels, or more or less.

[0040] As shown in the magnified view of the pixel region, the unit region 131 in Figure 4 contains four so-called Bayer arrays, each consisting of four pixels: green pixels Gb and Gr, blue pixel B, and red pixel R, arranged vertically and horizontally. The green pixels Gb and Gr are pixels that have a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter 102 and receives light in the red wavelength band.

[0041] In this embodiment, multiple blocks are defined such that each block contains at least one unit region 131, and each block can control the pixels contained within it using different control parameters. In other words, it is possible to acquire imaging signals with different imaging conditions for a group of pixels contained in one block and a group of pixels contained in another block. Examples of control parameters include frame rate, gain, decimation rate, number of rows or columns to add the pixel signals, charge accumulation time or number of accumulations, and the number of bits (word length) for digitization. The image sensor 100 can freely perform decimation not only in the row direction (X-axis direction of the imaging chip 113) but also in the column direction (Y-axis direction of the imaging chip 113). Furthermore, the control parameters may also be parameters used in image processing after acquiring image signals from pixels.

[0042] Figure 5 is a diagram illustrating the circuit in a unit region 131. In the example in Figure 5, one unit region 131 is formed by nine adjacent 3x3 pixel arrangements. As mentioned above, the number of pixels included in a unit region 131 is not limited to this and may be less or more. The two-dimensional positions of the unit region 131 are indicated by symbols A to I.

[0043] The reset transistors of the pixels included in the unit region 131 are configured to be individually switched on and off for each pixel. In Figure 5, a reset wire 300 is provided to switch the reset transistor of pixel A on and off, and a reset wire 310 is provided separately from the reset wire 300 to switch the reset transistor of pixel B on and off. Similarly, a reset wire 320 is provided separately from the reset wires 300 and 310 to switch the reset transistor of pixel C on and off. Dedicated reset wires are also provided for switching the reset transistors of other pixels D to pixel I.

[0044] The transfer transistors of the pixels included in the unit region 131 are also configured to be individually switched on and off for each pixel. In Figure 5, separate transfer lines are provided for switching the transfer transistor of pixel A on and off (302), for switching the transfer transistor of pixel B on and off (312), and for switching the transfer transistor of pixel C on and off (322). Dedicated transfer lines are also provided for switching the transfer transistors of other pixels D to pixel I.

[0045] Furthermore, the selection transistors for each pixel within the unit region 131 are also configured to be individually switched on and off. In Figure 5, separate selection wires are provided for switching the selection transistor of pixel A on and off, for switching the selection transistor of pixel B on and off, and for switching the selection transistor of pixel C on and off, respectively. Dedicated selection wires are also provided for switching the selection transistors of other pixels D to pixel I.

[0046] The power supply wiring 304 is commonly connected from pixel A to pixel I within the unit region 131. Similarly, the output wiring 308 is commonly connected from pixel A to pixel I within the unit region 131. In addition, the power supply wiring 304 is commonly connected between multiple unit regions, but the output wiring 308 is provided individually for each unit region 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0047] By individually switching the reset transistor and transfer transistor of the unit region 131 on and off, charge accumulation, including the charge accumulation start time, accumulation end time, and transfer timing, can be controlled independently for pixels A through I contained within the unit region 131. Furthermore, by individually switching the selection transistor of the unit region 131 on and off, the pixel signals from each pixel A through I can be output via a common output wiring 308.

[0048] Here, a known rolling shutter method is used to control charge accumulation in a regular order for rows and columns for pixels A to I contained in the unit region 131. When pixels are selected row by row and then columns are specified using the rolling shutter method, the pixel signals are output in the order "ABCDEFGHI" in the example shown in Figure 5.

[0049] By configuring the circuit based on the unit region 131 in this way, the charge accumulation time can be controlled for each unit region 131. In other words, pixel signals with different frame rates can be output for each unit region 131. Furthermore, by allowing charge accumulation (imaging) to occur in some areas of the imaging chip 113 while resting the unit regions 131 in other areas, imaging can be performed only in predetermined areas of the imaging chip 113, and the resulting pixel signals can be output. Moreover, by switching the area where charge accumulation (imaging) is performed between frames (the area targeted for accumulation control), sequential imaging can be performed in different areas of the imaging chip 113, and pixel signals can be output.

[0050] Figure 6 is a block diagram showing the functional configuration of the image sensor 100 corresponding to the circuit illustrated in Figure 5. The analog multiplexer 411 sequentially selects nine PDs 104 that form a unit region 131 and outputs the respective pixel signals to output wiring 308 provided corresponding to that unit region 131. The multiplexer 411 is formed on the imaging chip 113 together with the PDs 104.

[0051] The pixel signals output via the multiplexer 411 are subjected to CDS and A / D conversion by a signal processing circuit 412 formed on the signal processing chip 111, which performs correlated double sampling (CDS) and analog-to-digital (A / D) conversion. The A / D converted pixel signals are then passed to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. The demultiplexer 413 and the pixel memory 414 are formed on the memory chip 112.

[0052] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to the subsequent image processing unit. The arithmetic circuit 415 may be located on the signal processing chip 111 or on the memory chip 112. Although Figure 6 shows the connections for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not necessarily exist for each unit area 131; for example, a single arithmetic circuit 415 may sequentially process by referencing the values ​​in the pixel memory 414 corresponding to each unit area 131.

[0053] As described above, output wiring 308 is provided corresponding to each of the unit regions 131. Since the image sensor 100 is made up of stacked imaging chip 113, signal processing chip 111, and memory chip 112, by using inter-chip electrical connections with bumps 109 for these output wirings 308, the wiring can be routed without increasing the size of each chip in the planar direction.

[0054] <Explanation of distance measurement> Figure 7 illustrates the position of focus detection pixels on the imaging surface of the image sensor 100. In this embodiment, focus detection pixels are discretely arranged along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of Figure 7, 15 focus detection pixel lines 60 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 60 output an image signal for distance measurement. In the imaging chip 113, normal imaging pixels are provided at pixel positions other than the focus detection pixel lines 60. The imaging pixels output an image signal for external vehicle monitoring.

[0055] Figure 8 is an enlarged view of a region containing a portion of one of the focus detection pixel lines 60 described above. In Figure 8, red pixels R, green pixels G(Gb, Gr), and blue pixels B, as well as focus detection pixels S1 and S2, are shown as examples. The red pixels R, green pixels G(Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0056] The square-shaped regions exemplified for the red pixel R, green pixel G(Gb, Gr), and blue pixel B represent the light-receiving areas of the imaging pixels. Each imaging pixel receives the light beam passing through the exit pupil of the imaging optical system 31 (Figure 9). That is, the red pixel R, green pixel G(Gb, Gr), and blue pixel B each have square-shaped mask openings, and the light passing through these mask openings reaches the light-receiving area of ​​the imaging pixel.

[0057] Furthermore, the shape of the light-receiving area (mask aperture) of the red pixel R, green pixel G (Gb, Gr), and blue pixel B is not limited to a rectangle; for example, it may be circular.

[0058] The semicircular regions illustrated for focus detection pixels S1 and S2 indicate the light-receiving areas of the focus detection pixels. Specifically, focus detection pixel S1 has a semicircular mask opening to the left of its pixel position in Figure 8, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S1. On the other hand, focus detection pixel S2 has a semicircular mask opening to the right of its pixel position in Figure 8, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S2. In this way, focus detection pixels S1 and S2 each receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system 31 (Figure 9).

[0059] Note that the position of the focus detection pixel line on the imaging chip 113 is not limited to the position exemplified in Figure 7. Furthermore, the number of focus detection pixel lines is not limited to the example in Figure 7. In addition, the shape of the mask aperture in the focus detection pixel S1 and focus detection pixel S2 is not limited to a semicircle; for example, the rectangular light-receiving area (mask aperture) in the imaging pixel R, imaging pixel G, and imaging pixel B may be divided horizontally to form a rectangle.

[0060] Furthermore, the focus detection pixel line on the imaging chip 113 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 113. Image sensors in which imaging pixels and focus detection pixels are arranged in a two-dimensional manner, as shown in Figure 8, are well known, and detailed illustrations and explanations of these pixels are omitted. In the example shown in Figure 8, a configuration in which focus detection pixels S1 and S2 each receive one of the pair of light beams used for focus detection, known as a 1PD structure, was described. Alternatively, a configuration in which each focus detection pixel receives both of the pair of light beams used for focus detection, known as a 2PD structure, may be used, for example, as disclosed in Japanese Patent Application Publication No. 2007-282107. By using this 2PD structure, it becomes possible to read image data from the focus detection pixels as well, and the focus detection pixels do not become defective pixels.

[0061] In this embodiment, the focusing state (defocus amount) of the imaging optical system 31 is calculated by detecting the amount of image shift (phase difference) between a pair of images caused by a pair of light beams passing through different regions of the imaging optical system 31 (Figure 9) based on the distance-measuring image signals output from the focus-detection pixel S1 and the focus-detection pixel S2.

[0062] Generally, the pair of images described above move closer to each other in a so-called front-focus state where the imaging optical system 31 forms a sharp image of the object (e.g., a preceding vehicle) in front of the planned focal plane, and conversely, they move further apart in a so-called back-focus state where the sharp image of the object is formed behind the planned focal plane. In the focused state where a sharp image of the object is formed at the planned focal plane, the pair of images relatively coincide. Therefore, the amount of relative positional displacement between the pair of images corresponds to the distance to the object (depth information).

[0063] The calculation of the defocus amount based on the above phase difference is well known in the field of cameras, so a detailed explanation will be omitted. Here, since there is a one-to-one correspondence between the defocus amount and the distance to the object, the distance from camera 3 to each object can be determined by determining the defocus amount for each object. In other words, distance measurement (distance measurement) to the above object can be performed at multiple positions on the shooting screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a mathematical formula or lookup table and stored in non-volatile memory 35b (Figure 9).

[0064] <Camera Description> Figure 9 is a block diagram illustrating the configuration of a camera 3 having the image sensor 100 described above. In Figure 9, the camera 3 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36.

[0065] The imaging optical system 31 guides the light beam from the field of view to the imaging unit 32. The imaging unit 32 includes the image sensor 100 and the drive unit 32a, and converts the image of the object formed on the imaging chip 113 by the imaging optical system 31 into photoelectric signals. The drive unit 32a generates drive signals necessary to cause the image sensor 100 (imaging chip 113) to perform independent accumulation control in the block units described above. Instructions such as the position and shape of the block, its range, and accumulation time are transmitted from the control unit 35 to the drive unit 32a.

[0066] The image processing unit 33 works in cooperation with the work memory 34 to perform image processing on the image data captured by the imaging unit 32. In addition to image processing such as edge enhancement and gamma correction, the image processing unit 33 also performs color detection of objects contained in the image.

[0067] The work memory 34 temporarily stores image data before and after image processing. The recording unit 36 ​​records image data on a storage medium, such as a non-volatile memory. The control unit 35 is composed of, for example, a CPU, and controls the overall operation of the camera 3 according to control signals from the control device 4. For example, it performs a predetermined exposure calculation based on the image signal captured by the imaging unit 32 and instructs the drive unit 32a on the storage time of the imaging chip 113 required for proper exposure.

[0068] The control unit 35 includes a distance measurement calculation unit 35a and a non-volatile memory 35b. As described above, the distance measurement calculation unit 35a measures the distance to the target object at multiple positions on the captured image screen. The image data acquired by the camera 3 and the distance measurement data calculated by the camera 3 are sent to the control device 4 (Figure 1). The non-volatile memory 35b stores the program executed by the control unit 35a and the information necessary for distance measurement.

[0069] <Block control of the image sensor> The control device 4 causes the image sensor 100 (imaging chip 113) of the camera 3 to perform independent accumulation control in the block units described above. For this reason, the control device 4 receives the following signals from various parts of the vehicle 1 (Figure 2). (1) Amount of depression of the accelerator pedal 7a A signal indicating the amount the accelerator pedal 7a is pressed is input from the throttle control device 7 to the control device 4. (2) Amount of depression of the brake pedal 8a A signal indicating the amount the brake pedal 8a is pressed is input from the brake control device 8 to the control device 4.

[0070] (3) Rotation angle of the steering wheel 10 A signal indicating the rotation angle of the steering wheel 10 is input from the steering control device 9 to the control device 4. The ratio of the rotation angle of the steering wheel 10 to the steering angle of the steering device is determined by the steering gear ratio. (4) Vehicle speed V of vehicle 1 The detection signal from the vehicle speed sensor 12 is input to the control device 4.

[0071] (5) Operation signal of the turn signal switch 11 The operation signal from the turn signal switch 11 is input to the control device 4. (6) Operating position of the shift lever A signal indicating the operating position of the shift lever, detected by the shift lever position detection device 16, is input to the control device 4.

[0072] (7) Location information of Vehicle 1 Location information determined by the GPS device 15 is input from the GPS device 15 to the control device 4. (8) Sound information of the surroundings of vehicle 1 Sound information collected by microphone 17 from the front, right side, and left side of vehicle 1 is input to control device 4, respectively.

[0073] Figure 10 illustrates the imaging surface of the imaging chip 113, the regions on the imaging chip 113 where charge accumulation (imaging) takes place (imaging region 81 and region of interest 82), and regions where charge accumulation (imaging) in the row and column directions does not take place (rest region 83). Region of interest 82 is a region where charge accumulation (imaging) takes place under different conditions than imaging region 81. The size and position of imaging region 81 and region of interest 82 on the imaging chip 113 are also part of the imaging conditions.

[0074] The control device 4 controls the unit regions 131 included in the imaging region 81 to be imaged under a first condition, and controls the unit regions 131 included in the area of ​​interest 82 to be imaged under a second condition. The control device 4 also pauses imaging of the unit regions 131 included in the pause area 83. Multiple areas of interest 82 may be provided, and the imaging conditions may differ between multiple areas of interest. Furthermore, the pause area 83 may not be provided.

[0075] <Explanation of the flowchart> The following describes how to determine the imaging area 81 and the area of ​​interest 82, referring to flowcharts (Figures 11, 12, and 17). Figure 11 is a flowchart illustrating the flow of control processing for the camera 3 executed by the control device 4. The program for executing the processing according to the flowchart in Figure 11 is stored in the memory unit 4b of the control device 4. For example, when power is supplied from the vehicle 1 or the engine is started, the control device 4 starts the program that performs the processing according to Figure 11.

[0076] In step S10 of Figure 11, the control device 4 determines whether flag a = 0 or not. Flag a is set to 1 if the initial setup is complete, and to 0 if the initial setup is not complete. If flag a = 0, the control device 4 affirms step S10 and proceeds to step S20, and if flag a ≠ 0, it negates step S10 and proceeds to step S30.

[0077] In step S20, the control device 4 performs an initial setup process and proceeds to step S30. Details of the initial setup process will be described later. In step S30, the control device 4 performs a driving assist setting process and proceeds to step S40. In the driving assist setting process, the imaging area 81 and the area of ​​interest 82 are determined for the image sensor 100. Details of the driving assist setting process will be described later.

[0078] In step S40, the control device 4 sends an instruction to the camera 3 to drive the imaging area 81 and the area of ​​interest 82 of the image sensor 100 under predetermined conditions to acquire an image. In this embodiment, for example, when the vehicle speed V increases from 0, the control device 4 sets the frame rate of the area of ​​interest 82 to be higher than that of the imaging area 81, increases the gain, decreases the decimation rate, and shortens the storage time. As a result, the camera 3 takes images, and distance measurement (distance measurement) is performed at multiple positions on the captured screen as described above. It is not necessary to make all parameters such as frame rate, gain, decimation rate, and storage time different between the imaging area 81 and the area of ​​interest 82; it is sufficient to make at least one of them different. The control device 4 may also be set not to perform decimation for the area of ​​interest 82.

[0079] In step S45, the control device 4 acquires image data and distance measurement data from the camera 3 and proceeds to step S50. In step S50, the control device 4 determines whether or not the setting to display information has been made. If the display setting has been made, the control device 4 affirms step 50 and proceeds to step S60. If the display setting has not been made, the control device 4 negates step 50 and proceeds to step S70.

[0080] In step S60, the control device 4 sends display information to the display device 14 (Figure 1) and proceeds to step S70. The display information is information corresponding to the state of vehicle 1 determined in the driving assist setting process (S30), and for example, messages such as "Stopped", "Emergency stop", "Turning right", and "Turning left" are displayed on the display device 14. Alternatively, instead of sending display information, or along with sending display information, an audio signal may be sent to an audio playback device (not shown) to play the above message. In this case, the audio playback device (not shown) may be the audio device of a navigation system (not shown).

[0081] In step S70, the control device 4 determines whether or not the device has been turned off. If the control device 4 receives an off signal from, for example, the vehicle 1 (for example, an engine off signal), it affirms step S70, performs a predetermined off process, and terminates the process shown in Figure 11. If the control device 4 does not receive an off signal from, for example, the vehicle 1, it negates step S70 and proceeds to step S80. In step S80, the control device 4 waits for a predetermined time (for example, 0.1 seconds) and returns to step S30. If it returns to step S30, it repeats the process described above.

[0082] <Initial setup process> Figure 12 is a flowchart illustrating the details of the initial setup process in step S20 of the flowchart in Figure 11. In step S21 of Figure 12, the control device 4 receives location information of vehicle 1 from the GPS device 15 (Figure 1) and proceeds to step S22. In step S22, the control device 4 sets a flag indicating whether the lane in which vehicle 1 is traveling is on the left or right side of the road, i.e., left-hand traffic or right-hand traffic, based on the latitude and longitude included in the location information. Specifically, it determines the country in which vehicle 1 is used based on the latitude and longitude. Then, it refers to a database (not shown) and sets a flag indicating whether the roads in that country are left-hand traffic or right-hand traffic. The database showing the relationship between country names and left / right lane orientations is stored in the storage unit 4b in advance.

[0083] In step S23, the control device 4 sets a flag indicating the mounting position (right or left) of the steering wheel 10 in the vehicle 1 and proceeds to step S24. Information indicating whether it is right-hand drive or left-hand drive is stored in the storage unit 4b in advance as part of the vehicle 1's specifications. In step S24, the control device 4 determines the initial setting value based on the table illustrated in Figure 13 and proceeds to step S25. Note that the order of steps S21 and S23 may be reversed.

[0084] According to Figure 13, four initial settings, from "1" to "4," are available depending on the combination of the mounting position of the steering wheel 10 in vehicle 1 (right or left) and the position of the traffic lane on the road (right or left). For right-hand drive vehicles driving on the left side of the road, the initial setting is "4."

[0085] In step S25, the control device 4 sets the initial position of the region of interest 82. The initial position of the region of interest 82 is set to a position corresponding to the initial setting value. Specifically, if the initial setting value is "1", the initial position of the region of interest 82 is set to (Xq1, Yq), if the initial setting value is "2", the initial position of the region of interest 82 is set to (Xq2, Yq), if the initial setting value is "3", the initial position of the region of interest 82 is set to (Xq3, Yq), and if the initial setting value is "4", the initial position of the region of interest 82 is set to (Xq4, Yq).

[0086] In this explanation, the position of the area of ​​interest 82 is represented by the coordinates (Xq, Yq) of its center in the coordinate system representing the imaging area 81. Figure 10 shows an example of the area of ​​interest 82 when the initial setting is "4". Since it is a right-hand drive vehicle with left-hand traffic, the initial position (Xq4, Yq) is set so that the area of ​​interest 82 is located on the driver's side (to the right) within the left-hand traffic lane.

[0087] Figure 14 illustrates the area of ​​interest 82 when the initial setting is "1". Since it is a left-hand drive vehicle with right-hand traffic, the initial position (Xq1, Yq) is set so that the area of ​​interest 82 is located on the driver's side (left side) within the right-hand traffic lane.

[0088] Figure 15 illustrates the area of ​​interest 82 when the initial setting is "3". Since it is a left-hand drive vehicle and drives on the left side of the road, the initial position (Xq3, Yq) is set so that the area of ​​interest 82 is located on the driver's side (left side) within the left-hand traffic lane.

[0089] Figure 16 shows an example of the area of ​​interest 82 when the initial setting is "2". Since it is a right-hand drive vehicle and traffic is on the right side, the initial position (Xq2, Yq) is set so that the area of ​​interest 82 is located on the driver's side (to the right) within the right-hand traffic lane.

[0090] In step S26 of Figure 12, the control device 4 sets the initial size of the area of ​​interest 82. In this embodiment, the initial size of the area of ​​interest 82 (Px (X-axis direction) × Py (Y-axis direction)) is determined based on the size (dimensions) of the object (e.g., the preceding vehicle). If the preceding vehicle is included in the image acquired by the camera 3, the control device 4 estimates the size (dimensions) of the preceding vehicle based on the image height of the preceding vehicle captured by the imaging chip 113, the known focal length of the imaging optical system 31, and the distance L from vehicle 1 to the preceding vehicle obtained by distance measurement. The initial size is then set to the number of pixels (Px (X-axis direction) × Py (Y-axis direction)) that make up the image obtained on the imaging chip 113 when the estimated size (e.g., width 3 (m) × height 1.4 (m)) of the preceding vehicle is captured from 1 (m) behind.

[0091] Px and Py are calculated using the following equations (1) and (2). Px = ox × L …(1) Py = oy × L …(2) However, ox is the number of pixels in the X-axis direction that constitute the image of the preceding vehicle captured by the imaging chip 113 at a distance of L(m). oy is the number of pixels in the Y-axis direction that constitute the image of the preceding vehicle captured by the imaging chip 113 at a distance of L(m). L is the distance between vehicle 1 and the preceding vehicle. In the coordinate system representing the imaging area 81, the Yq representing the initial position corresponds to the height center of the image obtained on the imaging chip 113 when imaging a preceding vehicle located 1 m away (in this example, the point at a height of 0.7 m of the preceding vehicle).

[0092] In step S27, the control device 4 sends display information to the display device 14 (Figure 1), sets flag a to 1, and terminates the process shown in Figure 12. The display information indicates that the initial setup process has been completed, for example, by displaying the message "Initial setup completed" on the display device 14.

[0093] <Driving assist setting process> Figure 17 is a flowchart illustrating the details of the driving assist setting process. In step S310 of Figure 17, the control device 4 affirms step S310 and proceeds to step S320 if the shift lever position information input from the shift lever position detection device 16 (Figure 1) is "P" (parking). If the shift lever position information input from the shift lever position detection device 16 (Figure 1) is not "P", the control device 4 negates step S310 and proceeds to step S420. Note that the judgment in step S310 may also be applied when the shift lever is in the "N" (neutral) position.

[0094] In step S320, the control device 4 receives the vehicle speed V from the vehicle speed sensor 12 and proceeds to step S330. The control device 4 changes the frame rate of the area of ​​interest 82 according to the vehicle speed V, for example. As described above, when the frame rate of the area of ​​interest 82 is set higher than the frame rate of the imaging area 81, the control device 4 sets the frame rate of the area of ​​interest 82 higher as the vehicle speed V increases and sets the frame rate of the area of ​​interest 82 lower as the vehicle speed V decreases. In this case, the control device 4 may also apply control such that the frame rate of the imaging area 81 other than the area of ​​interest 82 is also proportional to the vehicle speed V. In step S330, the control device 4 receives the amount of depression of the brake pedal 8a from the brake control device 8 (Figure 1) and proceeds to step S340.

[0095] In step S340, the control device 4 determines whether the flag Em=0 based on the vehicle speed V and the amount (angle) of depression of the brake pedal 8a. The flag Em is a flag that is set based on the vehicle speed V and the change in the amount (angle) of depression of the brake pedal 8a, as illustrated in Figure 18. In this embodiment, if Em=1, it is determined to be an emergency brake (sudden brake), and if Em=0, it is determined to be a normal brake. If Em=0, the control device 4 affirms step S340 and proceeds to step S350. If Em=1, the control device 4 negates step S340 and proceeds to step S430. Alternatively, Em=1 may be determined based on the change in the opening degree of a brake valve (not shown) instead of the change in the amount (angle) of depression of the brake pedal 8a. Furthermore, Em=1 may be determined based on the change in vehicle speed V, or based on the change in the reduction ratio of a transmission (not shown).

[0096] In step S350, the control device 4 receives input from the throttle control device 7 (Figure 1) regarding the amount the accelerator pedal 7a is pressed, and proceeds to step S360. In step S360, the control device 4 receives input from the steering control device 9 regarding the rotation angle θ of the steering wheel 10, and proceeds to step S370. In step S370, the control device 4 determines whether or not a steering operation has been performed. If the rotation angle θ is greater than a predetermined value, the control device 4 affirms step S370 and proceeds to step S380, and if the rotation angle θ is less than or equal to the predetermined value, the control device 4 denies step S370 and proceeds to step S440.

[0097] In step S380, the control device 4 calculates the amount of movement Xdist of the region of interest 82 in the X-axis direction based on the rotation angle θ of the steering wheel 10 and the vehicle speed V using the following equation (3). Xdist = θ × (V × 0.2) …(3) According to equation (3) above, the larger the steering angle of the steering device (i.e., the rotation angle θ of the steering wheel 10), and the larger the vehicle speed V, the larger the displacement Xdist.

[0098] In step S390, the control device 4 calculates the position (X coordinate) of the region of interest 82 during travel using the following equation (4), based on the initial position (XqN, Yq) of the region of interest 82 set in the initial setup process. Xq = XqN + Xdist …(4) However, N is one of the initial setting values ​​1 to 4 determined in the initial setup process. Xdist is the amount of movement of the area of ​​interest 82 in the X-axis direction calculated in step S380, and corresponds to the number of pixels in the X-axis direction. The position of the area of ​​interest 82 changes according to the steering operation as a result of the processing in step S390. The position of the area of ​​interest 82 also changes depending on the magnitude of the vehicle speed V.

[0099] In step S400, the control device 4 calculates the position (Y coordinate) of the area of ​​interest 82 during travel using the following equation (5), based on the initial position (XqN, Yq) of the area of ​​interest 82 set in the initial setup process. Yq = Yq + P(Z) …(5) However, P(Z) is the amount of movement of the region of interest 82 in the Y-axis direction, and is the number of pixels in the Y-axis direction corresponding to the depth Z(m). For example, it represents how many pixels in the Y-axis direction correspond to an image of a road with a depth of 20(m). The relationship P(Z) between depth Z and the number of pixels is stored in the memory unit 4b (Figure 2) beforehand.

[0100] Generally, when imaging the direction of travel on a flat, straight road, the number of pixels in the Y-axis direction corresponding to the road image on the imaging chip 113 increases as the depth Z(m) from the vehicle 1 increases. Therefore, the value of Yq, which corresponds to the height center of the image when imaging a preceding vehicle located 1(m) away, is increased as the preceding vehicle of interest becomes farther away (i.e., the depth Z increases).

[0101] The control device 4 determines the depth Z of the preceding vehicle of interest using the following equation (6). Z = Za + Zb …(6) However, Za is the braking distance (m) on a dry road, and Zb is the braking distance (m) on a wet road. Za and Zb are based on the values ​​illustrated in Figure 19. In this embodiment, the position of the area of ​​focus 82 is determined so that the preceding vehicle at a depth Z in front of vehicle 1 (i.e., at a distance of Z(m) from vehicle 1) is included in the area of ​​focus 82. This is based on the idea of ​​focusing on an area farther than the distance required to stop in the event of emergency braking. The value of depth Z (Za + Zb) corresponding to the vehicle speed V is stored in advance in the memory unit 4b (Figure 2). According to the processing in step S400, the position of the area of ​​focus 82 changes in accordance with the change in vehicle speed V. For the region of interest 82 whose position has been changed in this way, at least one of the following is made different between the imaging region 81 and the region of interest 82: frame rate, gain, decimation rate, storage time, etc.

[0102] In step S410, the control device 4 calculates the size of the area of ​​interest 82 during travel (X_wid, Y_wid) based on the initial size (Px × Py) of the area of ​​interest 82 set in the initial setup process using the following equations (7) and (8), and then terminates the process shown in Figure 17. X_wid = Px / Z …(7) Y_wid = Py / Z …(8) However, Px is the number of pixels in the X-axis direction set in step S26, and Py is the number of pixels in the Y-axis direction set in step S26. According to equations (7) and (8) above, the size of the region of interest 82 during driving (X_wid, Y_wid) becomes smaller than the initial size of the region of interest 82 (Px × Py) as the preceding vehicle of interest moves further away (the depth Z becomes deeper). According to the processing in step S410, the size of the region of interest 82 changes in accordance with the change in vehicle speed V. For the resized region 82 of interest, at least one of the frame rate, gain, decimation rate, or storage time is made different between the imaging region 81 and the region of interest 82.

[0103] In step S420, which proceeds after determining that step S310 described above was negative, the control device 4 performs a stop setting process and terminates the process shown in Figure 17. The stop setting process determines the position of the area of ​​interest 82 so that, for example, a preceding vehicle located 1 m away is included in the area of ​​interest 82. In addition, the size of the area of ​​interest 82 in the X-axis direction is maximized so that objects located close to the side of the vehicle 1 are included in the area of ​​interest 82 as much as possible.

[0104] In step S430, which proceeds after a negative determination of step S340 described above, the control device 4 performs setting processing for sudden braking detection and terminates the processing shown in Figure 17. The setting processing for sudden braking detection includes, for example, stopping decimation in the area of ​​interest 82, increasing the frame rate to the maximum, shortening the storage time, and setting the gain to a high level. The control device 4 may also increase the frame rate of the imaging area 81 other than the area of ​​interest 82. Furthermore, for a predetermined time (for example, 5 to 15 seconds) after the negative determination of step S340, the control device 4 instructs the camera 3 to record the images acquired by the camera 3 and save them to the recording unit 36.

[0105] After a sudden stop, the control device 4 further moves the area of ​​interest 82 to the initial position set in the initial setup process (step S25 in Figure 12), and changes the size of the area of ​​interest 82 to the initial size (Px × Py) set in the initial setup process (step S26 in Figure 12). As a result, the position and size of the area of ​​interest 82, which changed due to the vehicle speed V while driving, returns to a position and size suitable for when the vehicle is stopped.

[0106] In step S440, which proceeds after a negative determination of step S370 described above, the control device 4 sets the position (X coordinate) of the area of ​​interest 82 during driving to not move. That is, if the rotation angle θ of the steering wheel 10 is less than or equal to a predetermined value, θ←0 is set, and the value of Xdist is also set to 0. In other words, if the operating angle of the steering wheel 10 is less than a predetermined value, the position (X coordinate) of the area of ​​interest 82 is maintained. This helps to reduce the processing load during minute operations that are not turning operations.

[0107] Figure 20(a) illustrates the movement of the focus area 82 and the change in its size when making a right turn at an intersection on a public road. According to the above driving assist setting process, when vehicle 1 is waiting to make a right turn behind a preceding vehicle, the focus area 82A is at its initial position, and its size is approximately the same as the initial size (Px × Py). When vehicle 1 is moving forward and the driver begins to steer to the right, the focus area 82B moves diagonally upward to the right. Since the vehicle speed V is low, the size of the focus area 82B is also approximately the same as the initial size (Px × Py).

[0108] Figure 20(b) illustrates the movement of the focus area 82 and the change in its size when changing lanes while accelerating to the right-hand passing lane on a highway. According to the above driving assist setting process, when vehicle 1 is traveling at high speed, the position of focus area 82A is higher than the initial position, and the size of focus area 82A is smaller than the initial size (Px × Py). When the driver steers to the right while vehicle 1 is accelerating, the position of focus area 82B moves diagonally upward to the right. Because the vehicle speed V is high, the size of focus area 82B becomes even smaller. Note that Figure 20 is an example for left-hand traffic, but it can be used as appropriate for left turns in right-hand traffic or lane changes in right-hand traffic. In addition, the driver's gaze may be detected by a gaze detection device (for example, a gaze detection device installed on the steering wheel), and areas that the driver is not looking at or blind spots may be set as the focus area 82. Furthermore, gaze detection methods include corneal reflection, which detects the user's gaze direction by reflecting infrared light off the driver's cornea; limbus tracking, which utilizes the difference in light reflectivity between the cornea and sclera; and image analysis, which detects gaze by capturing images of the eyeball with a camera and processing the images. Any of these gaze detection methods may be used.

[0109] According to the above-described embodiment, the following effects and advantages can be obtained. (1) The camera 3 is equipped with a control device 4 that recognizes at least one of the specifications of the mounted vehicle 1 and the operation of the vehicle 1's control unit, an imaging unit 32 that has at least a focus area 82 and an imaging area 81 and images the outside of the vehicle 1, and a control device 4 that sets different imaging conditions for the focus area 82 and the imaging area 81 based on the recognition results by the control device 4, so that the imaging conditions of the camera 3 can be set appropriately.

[0110] (2) The control device 4 recognizes the mounting position (right or left) of the steering wheel 10 in the vehicle 1, and can appropriately set the imaging conditions of the camera 3 according to the driver's seating position.

[0111] (3) The setting unit sets the frame rate of the area of ​​interest 82 and the frame rate of the imaging area 81 to be different depending on the position of the steering wheel 10, so that the imaging conditions of the camera 3 can be appropriately set, for example, by increasing the frame rate in the area of ​​interest 82 on the driver's side (right).

[0112] (4) The system includes a control device 4 that detects information regarding the vehicle speed V of the vehicle 1. The control device 4 sets the imaging conditions for the area of ​​interest 82 and the imaging conditions for the imaging area 81 differently according to the detection result of the information regarding the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set according to the vehicle speed V.

[0113] (5) The control device 4 changes at least one of the imaging conditions of the area of ​​interest 82 and the imaging conditions of the imaging area 81 when the information regarding the vehicle speed V increases or decreases, so that the imaging conditions of the camera 3 can be appropriately set, for example, by increasing the frame rate when the vehicle speed V is fast.

[0114] (6) When the rotation angle θ of the steering wheel 10 exceeds a predetermined value, the control device 4 changes the frame rate of at least one of the frame rates of imaging the area of ​​interest 82 and imaging the area of ​​interest 81 to a higher value, thereby changing the imaging conditions of the camera 3 in the case of a rotation operation.

[0115] (7) The vehicle is equipped with a control device 4 that transmits display information to the display device 14 of the vehicle 1 based on the imaging results from the imaging unit 32, so that necessary information can be provided to the occupants of the vehicle 1.

[0116] (8) When the rotation angle θ of the handle (steering wheel 10) does not reach a predetermined value, the control device 4 maintains the frame rate setting of at least one of the frame rates, which is the frame rate for imaging the area of ​​interest 82 and the frame rate for imaging the imaging area 81. This avoids changing the imaging conditions during minute operations that are not rotational operations. This prevents, for example, the frame rate of the area of ​​interest 82 from being changed more finely than necessary, which helps to reduce the processing load.

[0117] (9) The control device 4 includes at least one of the imaging conditions for which the imaging conditions for the area of ​​interest 82 differ from those for the imaging area 81, such as the frame rate of imaging, gain, decimation, pixel signal addition, accumulation, bit length, size of the imaging area, and position of the imaging area, so that the imaging conditions for the camera 3 can be set appropriately.

[0118] (10) The control device 4 changes at least one center position between the center position of the area of ​​interest 82 and the center position of the imaging area 81 based on the detection result of information regarding the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set, such as changing the position of the area of ​​interest 82 in accordance with the change in vehicle speed V.

[0119] (11) The control device 4 changes the size of at least one of the size of the area of ​​interest 82 and the size of the imaging area 81 based on the detection result of information regarding the vehicle speed V, so that the imaging conditions of the camera 3 can be appropriately set, such as changing the size of the area of ​​interest 82 in accordance with changes in the vehicle speed V.

[0120] (12) The control device 4 sets an imaging area 81 surrounding the area of ​​interest 82, so that the imaging conditions of the camera 3 can be set appropriately.

[0121] (13) The vehicle 1 is equipped with a steering wheel (10) as an operating unit, and the control device 4 changes at least one center position between the center position of the area of ​​interest 82 and the center position of the imaging area 81 based on the operation of the steering wheel, so that the imaging conditions of the camera 3 can be appropriately set, such as changing the position of the area of ​​interest 82 in accordance with the change in the path of the vehicle 1. In the above embodiment, the camera 3 was controlled by the control device 4, but a part of the control of the camera 3 may be performed by the control unit 35 of the camera 3.

[0122] The following modifications are also within the scope of the present invention, and it is possible to combine one or more of these modifications with the embodiments described above. (Variation 1) In the driving assist setting process, the control device 4 may be configured to change the position and size of the area of ​​interest 82 in response to the operation signal from the turn signal switch 11. As illustrated in Figure 21, the control device 4 changes the size of the area of ​​interest 82 or sets the imaging conditions for the area of ​​interest 82 based on the initial setting value determined in step S24 and the direction of the turn signal due to the operation of the turn signal switch 11.

[0123] For example, referring to Figure 10, if the vehicle is right-hand drive and drives on the left side of the road, and the initial setting is "4", the control device 4 will control the vehicle so that the area of ​​focus 82 includes the left edge of the road if the turn signal is to the left. Specifically, it expands the area of ​​focus 82 in Figure 10 to the left. Expanding the area of ​​focus 82 to the left is to prevent accidents caused by vehicles being pulled over when turning left. Conversely, if the turn signal is to the right, the control device 4 will control the vehicle so that the area of ​​focus 82 includes the oncoming lane. Specifically, it expands the area of ​​focus 82 in Figure 10 to the right.

[0124] Referring to Figure 14, if the vehicle is left-hand drive and drives on the right side of the road, and the initial setting is "1", the control device 4 will control the vehicle so that the area of ​​focus 82 includes the oncoming lane if the turn signal is to the left. Specifically, it will expand the area of ​​focus 82 in Figure 14 to the left. Conversely, if the turn signal is to the right, the control device 4 will control the vehicle so that the area of ​​focus 82 includes the right edge of the road. Specifically, it will expand the area of ​​focus 82 in Figure 14 to the right. The reason for expanding it to the right is to prevent accidents caused by vehicles being pulled over when turning right.

[0125] Referring to Figure 16, if the vehicle is right-hand drive and drives on the right side of the road, and the initial setting is "2", the control device 4 will control the vehicle so that the area of ​​focus 82 includes the oncoming lane if the turn signal is to the left. Specifically, it will greatly expand the area of ​​focus 82 in Figure 16 to the left. Conversely, if the turn signal is to the right, the control device 4 will control the vehicle so that the area of ​​focus 82 includes the right edge of the road. Specifically, it will slightly expand the area of ​​focus 82 in Figure 16 to the right. The expansion to the right is to prevent accidents caused by vehicles being pulled over when turning right.

[0126] Referring to Figure 15, if the vehicle is left-hand drive and drives on the left side of the road, and the initial setting is "3", the control device 4 will control the vehicle so that the area of ​​focus 82 includes the left edge of the road if the turn signal is to the left. Specifically, it will slightly expand the area of ​​focus 82 in Figure 15 to the left. This expansion to the left is to prevent accidents caused by vehicles being pulled over when turning left. Conversely, if the turn signal is to the right, the control device 4 will control the vehicle so that the area of ​​focus 82 includes the oncoming lane. Specifically, it will significantly expand the area of ​​focus 82 in Figure 15 to the right.

[0127] Figure 22 is a flowchart illustrating the processing when the turn signal switch 11 is operated according to Modification 1. When the control device 4 receives an operation signal from the turn signal switch 11 during the driving assist setting process, it activates the processing shown in Figure 22 as a subroutine. In step S510 of Figure 22, the control device 4 determines whether the turn signal direction is left or right. If the turn signal direction is left, the control device 4 affirms step S510 and proceeds to step S520, and if the turn signal direction is right, it negates step S510 and proceeds to step S530.

[0128] In step S520, the control device 4 determines whether the lane is on the left or right. If it is left-hand traffic, the control device 4 affirms step S520 and proceeds to step S550; if it is right-hand traffic, it denies step S520 and proceeds to step S540.

[0129] In step S540, the control device 4 controls the imaging unit 32 to include the oncoming lane in the area of ​​interest 82, and terminates the processing shown in Figure 22. In step S550, the control device 4 controls the imaging unit 32 to include the left edge of the road in the area of ​​interest 82, and terminates the processing shown in Figure 22.

[0130] In step S530, the control device 4 determines whether the lane is on the left or right. If it is left-hand traffic, the control device 4 affirms step S530 and proceeds to step S560; if it is right-hand traffic, it denies step S530 and proceeds to step S570.

[0131] In step S560, the control device 4 controls the imaging unit 32 to include the oncoming lane in the area of ​​interest 82, and terminates the processing shown in Figure 22. In step S570, the control device 4 controls the imaging unit 32 to include the right edge of the road in the area of ​​interest 82, and terminates the processing shown in Figure 22.

[0132] Furthermore, if the turn signal switch 11 is turned off after the processing shown in Figure 22, the control device 4 cancels the resizing of the area of ​​interest 82 as shown in Figure 22. Also, if the turn signal switch 11 is turned on, even if the vehicle speed V is 0, the frame rate of the area of ​​interest 82 may be set higher, the gain higher, the decimation rate lower, and the storage time shorter compared to the imaging area 81. However, if at least one of the frame rate, gain, decimation rate, or storage time is to be different between the imaging area 81 and the area of ​​interest 82, only the imaging condition that is different should be changed.

[0133] According to the modified example 1 described above, the imaging conditions for the area of ​​interest 82 and the imaging conditions for the imaging area 81 are set differently depending on the operation of the turn signal switch 11. For example, when turning right or left at an intersection, the area of ​​interest 82 can be appropriately set by including the oncoming lane in the area of ​​interest 82 to ensure reliable detection of oncoming vehicles, or by including the edge of the road in the area of ​​interest 82 to prevent accidents involving vehicles being pulled over. Furthermore, the imaging conditions can be appropriately set in both the imaging area 81 and the area of ​​interest 82, such as by setting a higher frame rate for the area of ​​interest 82 compared to the imaging area 81.

[0134] (Modification 2) In the driving assist setting process, the control device 4 may be configured to change the position and size of the focus area 82 in accordance with changes in the distance between the vehicle 1 and objects such as motorcycles, regular vehicles, large vehicles, and pedestrians.

[0135] In the modified example 2, the control device 4 determines the position of the area of ​​interest 82 so that the preceding vehicle is included in the area of ​​interest 82 when, for example, vehicle 1 approaches the preceding vehicle and the distance L (inter-vehicle distance) to the preceding vehicle decreases. Here, the change in the inter-vehicle distance L from vehicle 1 to the preceding vehicle is obtained by the distance measurement calculation unit 35a of the camera 3 sequentially measuring the distance L (inter-vehicle distance) to the preceding vehicle in the captured image based on images acquired by the camera 3 at predetermined time intervals in synchronization with the timing of acquisition of the vehicle speed V.

[0136] The control device 4 calculates the position (Y coordinate) of the area of ​​interest 82 while driving using the inter-vehicle distance L instead of the depth Z in equation (5) above. As a result, when imaging a preceding vehicle on a flat, straight road, the value of Yq, which indicates the position of the area of ​​interest 82 in the Y-axis direction on the imaging chip 113, increases as the inter-vehicle distance L increases and decreases as the inter-vehicle distance L decreases.

[0137] Furthermore, when the inter-vehicle distance L changes and becomes shorter, the control device 4 increases the size of the area of ​​interest 82 because the preceding vehicle appears larger in the camera 3. Conversely, when the inter-vehicle distance L changes and becomes longer, the control device 4 decreases the size of the area of ​​interest 82 because the preceding vehicle appears smaller in the camera 3. The control device 4 calculates the size of the area of ​​interest 82 while driving by substituting the inter-vehicle distance L for the depth Z in equations (7) and (8) above.

[0138] According to the process shown in Figure 17 above, the size of the area of ​​interest 82 is set to be smaller when the vehicle speed V increases. In the modified example 2, however, even if the vehicle speed V is high, if the distance L to the preceding vehicle is short, the size of the area of ​​interest 82 is set to be larger, so that the preceding vehicle can be appropriately included in the area of ​​interest 82. Therefore, compared to the case where the size of the area of ​​interest 82 is kept small, it becomes easier to detect changes in the driving state of the preceding vehicle based on the image acquired by the camera 3. In this way, even with the size and position of the area of ​​interest 82, at least one of the frame rate, gain, decimation rate, or storage time may be different between the imaging area 81 and the area of ​​interest 82.

[0139] (Variation 3) In addition to the existing focus area 82, the control device 4 may also set a new focus area 82 that includes objects such as motorcycles, passenger cars, large vehicles, and pedestrians when it detects objects around the vehicle 1. In modified example 3, the control device 4 sets a new focus area 82 that includes a detected object when the detected object moves. For example, the control device 4 sets a new focus area 82 that includes a detected object when the distance between the detected object and the vehicle 1 approaches within a predetermined distance. Then, the control device 4 releases the setting of the focus area 82 that includes the detected object when the distance between the detected object and the vehicle 1 exceeds the predetermined distance. In this way, the area of ​​interest 82 may also have at least one difference between the imaging area 81 and the area of ​​interest 82, such as frame rate, gain, decimation rate, or storage time.

[0140] According to Modification 3, the vehicle 1 is equipped with a control device 4 that detects moving objects around the vehicle 1 based on information from the camera 3. The control device 4 changes at least one of the imaging conditions, between the imaging conditions of the area of ​​interest 82 and the imaging conditions of the imaging area 81, based on the detection result of the moving object. This allows the imaging conditions of the camera 3 to be appropriately set depending on the presence or absence of a moving object.

[0141] Furthermore, when the control device 4 detects that the distance between the vehicle 1 and the moving object is approaching a predetermined distance based on information from the camera 3, it increases the frame rate of at least one of the frame rates for imaging the area of ​​interest 82 and the frame rate for imaging the imaging area 81, thereby facilitating the detection of changes in the movement state of the moving object based on the image acquired by the camera 3.

[0142] (Modification 4) A new area of ​​interest 82 may be set based on the color of the image acquired by camera 3. The control device 4 sets the area of ​​the image that contains a red object as the area of ​​interest 82. By adding an area containing a red object to the area of ​​interest 82, for example, red traffic lights, railway crossing warning lights, and emergency vehicle red lights can be included in the area of ​​interest 82. In this way, the area of ​​interest 82 may also have at least one difference between the imaging area 81 and the area of ​​interest 82, such as frame rate, gain, decimation rate, or storage time.

[0143] (Variation 5) A new focus area 82 may be set based on sound information collected by the microphone 17 of vehicle 1. For example, if the level of sound information on the right side of vehicle 1 exceeds a predetermined value, the control device 4 expands the focus area 82 to the right of the imaging area 81, or sets a new focus area 82 on the right side of the imaging area 81. The purpose of setting the focus area 82 on the right side is to collect external information on the right side of vehicle 1.

[0144] Furthermore, if, for example, the sound information level on the left side of the vehicle 1 exceeds a predetermined value, the control device 4 expands the focus area 82 to the left within the imaging area 81, or sets a new focus area 82 on the left side within the imaging area 81. The purpose of setting the focus area 82 on the left side is to collect external information on the left side of the vehicle 1. In this way, the area of ​​interest 82 may also have at least one difference between the imaging area 81 and the area of ​​interest 82, such as frame rate, gain, decimation rate, or storage time.

[0145] (Experimental variation 6) In the above embodiment, a case in which a focus area 82 including a preceding vehicle is set was described, but a focus area 82 including oncoming vehicles of vehicle 1 may also be set. In modified example 6, the control device 4 recognizes the vehicle closest to vehicle 1 as an oncoming vehicle from among the objects that are located within the above-described driving area and are traveling in the opposite direction (facing vehicle 1).

[0146] The control device 4 sets the area corresponding to the position of the oncoming vehicle in the image acquired by the camera 3 as the area of ​​focus 82. In particular, the control device 4 sets the area of ​​focus 82 to include the license plate of the oncoming vehicle and the face of the driver in the driver's seat of the oncoming vehicle.

[0147] By defining the area including the license plate and the face of the driver of the oncoming vehicle as the area of ​​focus 82, it is possible to appropriately include the oncoming vehicle approaching vehicle 1 in the area of ​​focus 82. In this way, the area of ​​interest 82 may also have at least one difference between the imaging area 81 and the area of ​​interest 82, such as frame rate, gain, decimation rate, or storage time.

[0148] (Example 7) In the above explanation, an example was described in which the imaging area 81 includes (surrounds) the area of ​​interest 82, but the imaging area 81 and the area of ​​interest 82 may also be set side by side. By moving the boundary line between the imaging area 81 and the area of ​​interest 82 left or right, the size and position of the imaging area 81 and the area of ​​interest 82 can be changed. Even for the area of ​​interest 82 set in this way, at least one of the frame rate, gain, decimation rate, storage time, etc., may be different between the imaging area 81 and the area of ​​interest 82.

[0149] In the above explanation, the distance measurement performed by camera 3 was calculated using a distance measurement calculation based on the image signal from the focus detection pixel provided in the image sensor 100. However, a method of measuring the distance using two images from a stereo camera may also be used. Alternatively, a method of measuring the distance using millimeter-wave radar separately from camera 3 may also be used.

[0150] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0151] 1…Vehicle 2…Driving assistance systems 3…Camera 4…Control device 4b…Storage section 5…First driving control unit 6…Second driving control unit 7…Throttle control device 7a...Accelerator pedal 8…Brake control device 8a...Brake pedal 9… Steering control device 10… Steering wheel 11…Turn signal switch 12…Vehicle speed sensor 14...Display device 15...GPS device 16... Shift lever position detection device 17... Mike 31…Imaging Optical System 32…Imaging Unit 32a... Drive unit 33…Image Processing Unit 34…Work Memory 35…Control Unit 35a…Distance calculation unit 36…Records Department 60…Focus detection pixel line 81…Imaging area 82, 82A, 82B… Areas of Interest 83…Hibernation area 100... Image sensor 113…Imaging chip

Claims

1. In an imaging device installed in a vehicle, An input unit for inputting information regarding the speed of the vehicle, An imaging unit for capturing images of the exterior of the vehicle, An imaging control unit that changes the imaging conditions of the imaging unit when information regarding the speed of the vehicle changes, It has, Based on information regarding the vehicle's speed, the imaging control unit sets the frame rate of the area of ​​interest, which is a part of the imaging area of ​​the imaging unit, to a lower value as the vehicle's speed decreases. Even if the vehicle's speed decreases, if the imaging control unit determines that it is a sudden brake, it increases the frame rate of the area of ​​interest. Imaging device.

2. The imaging surface of the imaging chip controlled by the imaging unit has an imaging region in which charge accumulation is performed, and a region of interest in which charge accumulation is performed under different conditions from that of the imaging region. The imaging region and the region of interest have at least different decimation rates. The imaging apparatus according to claim 1.

3. If the imaging control unit determines that a sudden brake has occurred, it stops the thinning in the area of ​​interest. The imaging apparatus according to claim 1 or 2.

4. The imaging control unit changes the position of the area of ​​interest in response to the steering operation of the vehicle. The imaging apparatus according to any one of claims 1 to 3.

5. The imaging control unit changes the center of the area of ​​interest in a vertical upward direction as the vehicle speed increases. The imaging apparatus according to any one of claims 1 to 4.

6. The imaging control unit determines whether the degree of reduction in the vehicle's speed is due to sudden braking, based on the vehicle's speed and the amount the brake pedal is pressed. The imaging apparatus according to any one of claims 1 to 5.

7. The amount of depression of the brake pedal is determined according to the depression angle of the brake pedal. The imaging device according to claim 6.

8. The imaging control unit determines, based on the amount of change in the vehicle's speed, whether the degree of decrease in the vehicle's speed is due to sudden braking. The imaging apparatus according to any one of claims 1 to 5.

Images