Imaging apparatus

Abstract

To appropriately set imaging conditions of an imaging unit.SOLUTION: An imaging device includes an imaging unit including an imaging element capable of independently setting imaging conditions of a plurality of areas, and a setting unit that sets the imaging conditions of the plurality of areas based on a speed and a moving direction of a moving object, the moving object being at least one of an automobile and a pedestrian imaged by the imaging unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Technologies have been developed that detect the vehicle's driving environment based on images acquired by a camera mounted on the vehicle, and then, based on the detected driving environment data, provide automatic driving control such as following a preceding vehicle, as well as driving assistance such as warnings, braking, and steering assistance (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-79424 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is expected that in the future there will be a mixture of vehicles with automatic driving control and vehicles that are driven manually by a driver, but there were not many proposals regarding this point. [Means for solving the problem]

[0005] According to a first aspect of the present invention, an imaging device includes an imaging unit equipped with an imaging element that can independently set imaging conditions for multiple areas, and a setting unit that sets imaging conditions for the multiple areas based on the speed and direction of movement of at least one of a vehicle and a pedestrian imaged by the imaging unit, the vehicle being a moving object. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a block diagram showing a configuration of an imaging system according to an embodiment; [Figure 2] FIG. 1 is a diagram illustrating an example of the arrangement of traffic lights at an intersection. [Figure 3] FIG. 1 illustrates an example of a traffic light for automobiles. [Figure 4] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 5] 1 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Figure 6] FIG. 2 is a diagram illustrating a circuit of a unit area. [Figure 7] FIG. 2 is a block diagram showing the functional configuration of an imaging element. [Figure 8] FIG. 2 is a diagram illustrating an example of the positions of focus detection pixels on an imaging surface. [Figure 9] FIG. 2 is an enlarged view of a region including a portion of a focus detection pixel line. [Figure 10] 4 is a flowchart illustrating control by a control unit of the automobile. [Figure 11] Figure 11(a) is a diagram explaining the positional relationship of the automobile, Figure 11(b) is a diagram showing a schematic view of the subject image captured by the front camera, and Figure 11(c) is a diagram showing a schematic view of the subject image captured by the rear camera. [Figure 12] 10 is a flowchart illustrating control by a control unit of a traffic light. [Figure 13] FIG. 10 is a diagram illustrating the control of an imaging unit of a traffic light for automobiles. [Figure 14] FIG. 10 is a diagram illustrating the control of an imaging unit of a traffic light for automobiles. [Figure 15] 10A and 10B are diagrams illustrating the control of the imaging unit of a pedestrian traffic light. [Figure 16] FIG. 16(a) is a diagram illustrating an example of an image captured by the image capturing unit of a pedestrian traffic light, and FIG. 16(b) is a diagram for explaining the settings of image capturing conditions. [Figure 17] 1A and 1B are diagrams illustrating an example of an image captured by an imaging unit of a traffic light for automobiles; DETAILED DESCRIPTION OF THE INVENTION

[0007] 1 is a block diagram illustrating the configuration of an imaging system 1 including an imaging device according to an embodiment. The imaging system 1 uses an automobile 10, another vehicle 20, a traffic signal generating device 30, and a traffic light 40. Instead of the traffic light 40 or in combination with the traffic light 40, an information providing system installed on the road or VICS (registered trademark: Vehicle Information and Communication System) may be used.

[0008] (Automobile 10) The automobile 10 includes an automobile operation unit 11, a GPS device 12, a navigation system 13, an optical system 14, a photoelectric conversion unit 15, a communication unit 16, a memory unit 17, a sensor 18, and a control unit 19. The automobile 10 has the basic configuration of an automobile, although a detailed description thereof will be omitted.

[0009] The vehicle operation unit 11 includes various operation members related to the operation of the vehicle, such as a steering wheel, a turn signal switch, a shift lever, an accelerator, a brake, and a switch for switching between an automatic driving mode and a manual driving mode.

[0010] The GPS device 12 calculates the position (longitude, latitude, etc.) of the automobile 10 based on signals obtained by receiving radio waves from GPS satellites. The position information calculated by the GPS device 12 is output to the navigation system 13 and the control unit 19.

[0011] The navigation system 13 detects the current position of the automobile 10 using the GPS device 12 or the like, obtains map data corresponding to the current position from a storage medium or a network, displays it on an LCD monitor, and provides guidance along a route to an input destination. The navigation system 13 includes an operation unit that accepts operations from the user, the LCD monitor mentioned above, a speaker that provides voice guidance, a reader that reads map data, and the like.

[0012] The optical system 14 is composed of multiple lenses and forms an image of a subject on the photoelectric conversion unit 15. When the optical system 14 is directed forward of the automobile 10, the photoelectric conversion unit 15 acquires an image in the direction in which the automobile 10 is traveling. When the optical system 14 is directed backward of the automobile 10, the photoelectric conversion unit 15 acquires an image in the direction opposite to the direction in which the automobile 10 is traveling. The optical system 14 has an angle of view that corresponds to multiple driving lanes (two or three lanes, etc.). A plurality of optical systems 14 may be provided to form a stereo camera.

[0013] The photoelectric conversion unit 15 includes an imaging element 100 configured by stacking an imaging chip that outputs pixel signals in response to light incident from the optical system 14, a signal processing chip that processes the pixel signals, and a memory chip that stores the pixel signals. As will be described in detail later, the imaging element 100 can set imaging conditions (including the case where no imaging is performed) individually for each pixel or for each unit area consisting of multiple pixels (for example, 16 pixels × 16 pixels).

[0014] In this embodiment, the optical system 14 and the photoelectric conversion unit 15 constitute the camera imaging unit 5, which captures images of objects (moving bodies, obstacles, etc.) around the automobile 10 and white lines on the road (including lines of other colors such as yellow). The automobile 10 is equipped with a front imaging unit 5 that captures images of the front of the automobile 10, and a rear imaging unit 5 that captures images of the rear of the automobile 10. In this description, a white line refers to a line such as a white line drawn on a roadway. The term white line also refers to both solid and broken lines. Although not shown, a radar may be provided, and surrounding objects may be detected by this radar and the imaging unit 5 (optical system 14, photoelectric conversion unit 15).

[0015] The communication unit 16 performs wireless communication (including optical beacons, radio beacons, and visible light communication) with external devices such as other vehicles 20 and traffic lights 40. Any communication method may be used.

[0016] The storage unit 17 is configured by a nonvolatile semiconductor memory such as a flash memory, and stores various programs and control parameters for driving the automobile 10 (including automatic driving).

[0017] The sensor 18 includes one or more vehicle speed sensors, yaw rate sensors, and other various sensors. The vehicle speed sensors detect the vehicle speed V of the automobile 10 and send detection signals to the control unit 19, etc. The yaw rate sensors detect the yaw rate of the automobile 10 and send detection signals to the control unit 19, etc. The yaw rate is the rate of change in the rotation angle of the vehicle in the turning direction.

[0018] The control unit 19 controls the entire automobile 10 and includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. In this embodiment, the control unit 19 sets and controls the imaging conditions for each unit area of ​​the image sensor 100 of the photoelectric conversion unit 15. Furthermore, when the autonomous driving mode is set by the vehicle operation unit 11, the control unit 19 detects white lines on the road using the image capture unit 5 (optical system 14, photoelectric conversion unit 15), and also detects moving objects, obstacles, etc. around the automobile 10 using the image capture unit 5, and cooperates with the navigation system 13 to perform autonomous driving to the destination inputted to the navigation system 13.

[0019] In this embodiment, the autonomous driving mode refers to a state in which the steering wheel, accelerator, brake, turn signal switch, shift lever, and other operations are all performed automatically under the control of the control unit 19. Manual driving refers to a state in which the driver performs operations such as the steering wheel, accelerator, brake, turn signal switch, and shift lever, and can be either an automatic transmission or a manual transmission. The autonomous driving mode includes not only fully autonomous driving, in which all operations are performed under the control of the control unit 19, but also semi-autonomous driving, in which the control unit 19 stops or decelerates the vehicle 10 to avoid collisions, etc., based on the outputs of the imaging unit 5, GPS 12, communication unit 16, sensor 18, and the like, even when the user is operating the vehicle operation unit 11. This allows the user to enjoy driving the vehicle 10 while ensuring safety. Semi-autonomous driving also includes a state in which the control unit 19 controls some operations, such as the steering wheel, accelerator, brake, turn signal switch, and shift lever, on behalf of the driver.

[0020] (20 other cars) The other cars 20 are equipped with a communication unit 21, a vehicle operation unit 22, a memory unit 23, an imaging unit 24, a control unit 25, etc., and the functions of each unit are the same as those of the automobile 10. Although the other cars 20 are omitted from FIG. 1, they also have the basic configuration of an automobile. However, some of the other cars 20 may not be equipped with the communication unit 21. Furthermore, the other cars 20 include a mixture of autonomously driven cars and manually driven cars. Of these, at least the vehicles equipped with an autonomous driving mode are capable of communicating with each other via the communication unit 21, and are configured to be able to send and receive information regarding whether they are in autonomous or manual driving mode, as well as image data acquired by the imaging unit 24.

[0021] (Traffic signal generator 30) The traffic signal generating device 30 is a device that controls the signal lights displayed on the display unit 42 of the traffic light 40, and includes a signal information generating unit 31, a memory unit 32, a communication unit 33, and a control unit 34. The traffic signal generating device 30 can be installed at each of the multiple traffic lights 40 provided at an intersection or the like, but a single traffic signal generating device 30 may also control the multiple traffic lights 40.

[0022] The traffic signal information generating unit 31 generates traffic signals based on the types and installation positions of multiple traffic signals 40 installed at intersections and the like, and traffic-related instructions from a traffic control center (not shown).

[0023] The storage unit 32 is configured by a non-volatile semiconductor memory such as a flash memory, and stores various programs and control parameters of the traffic signal generating device 30.

[0024] The communication unit 33 transmits the traffic signals generated by the signal information generation unit 31 to one or more traffic lights 40 via wired or wireless communication. The communication unit 33 also transmits and receives information to and from the traffic control center.

[0025] The control unit 34 controls the entire traffic signal generating device 30 and includes a CPU, RAM, ROM, etc. The control unit 34 also analyzes traffic conditions based on traffic volume and the like, and can control the signal information generating unit 31 based on the results.

[0026] (Traffic Light 40) The traffic light 40 has a communication unit 41, a display unit 42, an optical system 43, a photoelectric conversion unit 44, a memory unit 45, and a control unit 46. Although only one traffic light 40 is shown in FIG. 1, a plurality of traffic lights 40 are usually provided. For example, in the case of an intersection, as shown in FIG. 2, four traffic lights 40a for automobiles and eight traffic lights 40b for pedestrians are provided. The traffic lights 40 receive traffic signals corresponding to their installation positions from the traffic signal generating device 30 and turn on or flash the indicator lights of the display unit 42.

[0027] The communication unit 41 receives, via wire or wireless, the traffic signals generated by the traffic signal information generation unit 31. The communication unit 41 also transmits and receives various information, such as vehicle driving information and traffic information, between the automobile 10, other vehicles 20, and other traffic lights 40.

[0028] The display unit 42 has signal lights and displays them according to the traffic signals received by the communication unit 41. Specifically, the display unit 42 turns on, blinks, or turns off the signal lights to allow or restrict movement such as proceeding or stopping for vehicles traveling on the road and pedestrians crossing the road. Note that the display unit 42 may turn on not only red, yellow, and blue lights but also arrow lights indicating that it is OK to go straight, turn left, or turn right at an intersection.

[0029] In this embodiment, the optical system 43 and the photoelectric conversion unit 44 constitute the imaging unit 50 of the camera. The optical system 43 is made up of a plurality of lenses and forms an image of a subject on the photoelectric conversion unit 44. When the imaging unit 50 is provided in a traffic light 40a for automobiles, the optical system 43 is used mainly to capture images of vehicles. When the imaging unit 50 is provided in a traffic light 40b for pedestrians, the optical system 43 is used mainly to capture images of pedestrians (including bicycles). The imaging unit 50 is provided near the display unit 42 (signal light).

[0030] The photoelectric conversion unit 44 includes an imaging element 100 configured by stacking an imaging chip that outputs pixel signals in response to light incident from the optical system 43, a signal processing chip that processes the pixel signals, and a memory chip that stores the pixel signals. This has the same configuration as the photoelectric conversion unit 15 in the automobile 10 described above. The imaging element 100 can set imaging conditions in accordance with traffic signals for each pixel or for each unit area consisting of multiple pixels (for example, 16 pixels x 16 pixels).

[0031] Fig. 3 is a diagram illustrating an example of a traffic light 40a for automobiles placed at an intersection. In Fig. 3, an imaging unit 50 is installed near the bottom of a display unit 42a of the traffic light 40a. The imaging unit 50 has, for example, a wide-angle lens as an optical system 43, and has an angle of view that includes four lanes on both sides of the multiple driving lanes (e.g., two lanes on each side) of the intersection illustrated in Fig. 2.

[0032] The storage unit 45 is configured by a nonvolatile semiconductor memory such as a flash memory, and stores image data acquired by the photoelectric conversion unit 44 and the like.

[0033] The control unit 46 controls the entire traffic light 40 and includes a CPU, RAM, ROM, etc. In this embodiment, the control unit 46 controls the display of the signal lights of the display unit 42 in accordance with the traffic signal, and also controls imaging using the imaging element 100 of the photoelectric conversion unit 44.

[0034] In addition, if the shooting range of the imaging unit 50 of the traffic light 40a for automobiles includes the shooting range of the imaging unit 50 of the traffic light 40b for pedestrians, the imaging unit 50 (optical system 43, photoelectric conversion unit 44) of the traffic light 40b for pedestrians may be omitted.

[0035] <Explanation of stacked image sensor> The stacked imaging element 100 provided in the imaging section of the automobile 10, the other vehicle 20, and the traffic light 40 described above will now be described. This stacked imaging element 100 is described in WO13 / 164915, a patent application filed by the applicant and published earlier. FIG. 4 is a cross-sectional view of the stacked imaging element 100. The imaging element 100 includes a back-illuminated imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0036] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface (imaging surface). As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in FIG. 4.

[0037] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

[0038] A color filter 102 is provided on the incident side of the PD layer 106, on which incident light is incident, via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

[0039] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0040] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may be provided with passive elements and active elements.

[0041] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0042] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0043] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 109 per block, as described below. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in a peripheral region other than the pixel region where pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

[0044] The signal processing chip 111 has through-silicon vias (TSVs) 110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0045] FIG. 5 is a diagram illustrating the pixel array of the imaging chip 113 and the unit area 131. In particular, it shows the imaging chip 113 as observed from the back side (imaging surface) side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In the example of FIG. 5, 16 pixels, 4 pixels by 4 pixels adjacent to each other, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be around 1000, for example, 32 pixels by 64 pixels, or may be more or less than that.

[0046] 5 includes four so-called Bayer arrays, each consisting of four pixels: green pixels Gb and Gr, a blue pixel B, and a red pixel R, arranged vertically and horizontally. The green pixels Gb and Gr have a green filter as the color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixel B has a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R has a red filter as the color filter 102 and receives light in the red wavelength band.

[0047] In this embodiment, multiple blocks are defined, each containing at least one unit area 131, and each block can control the pixels contained in that block using different control parameters. In other words, imaging signals can be acquired under different imaging conditions for a group of pixels contained in one block and a group of pixels contained in another block. Examples of control parameters include frame rate, gain, thinning rate, number of rows or columns for adding pixel signals, charge accumulation time or number of accumulations, and number of digitization bits (word length). The image sensor 100 can freely perform thinning not only in the row direction (the X-axis direction of the image sensor 113) but also in the column direction (the Y-axis direction of the image sensor 113). Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels.

[0048] Fig. 6 is a diagram illustrating the circuitry in a unit area 131. In the example of Fig. 6, one unit area 131 is formed by nine pixels, 3 pixels x 3 pixels adjacent to each other. As mentioned above, the number of pixels included in the unit area 131 is not limited to this and may be more or less than this. The two-dimensional positions of the unit area 131 are indicated by symbols A to I.

[0049] The reset transistors of the pixels included in unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 6, reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and reset wiring 310 is provided to turn on and off the reset transistor of pixel B, separately from the reset wiring 300. Similarly, reset wiring 320 is provided to turn on and off the reset transistor of pixel C, separately from the reset wirings 300 and 310. Dedicated reset wiring is also provided for the other pixels D to I to turn on and off their respective reset transistors.

[0050] The transfer transistors of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 6, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. Dedicated transfer wiring is also provided for the other pixels D to I to turn on and off their respective transfer transistors.

[0051] Furthermore, the selection transistors of the pixels included in unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 6, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. Dedicated selection wiring is also provided for the other pixels D to I to turn on and off their respective selection transistors.

[0052] The power supply wiring 304 is commonly connected to pixels A to I included in the unit area 131. Similarly, the output wiring 308 is commonly connected to pixels A to I included in the unit area 131. Furthermore, the power supply wiring 304 is commonly connected among a plurality of unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0053] By individually turning on and off the reset transistor and transfer transistor of the unit area 131, it is possible to control charge accumulation, including the accumulation start time, accumulation end time, and transfer timing, independently for pixels A to I included in the unit area 131. Furthermore, by individually turning on and off the selection transistor of the unit area 131, it is possible to output pixel signals of each of pixels A to I via a common output wiring 308.

[0054] Here, the so-called rolling shutter method is known, which controls charge accumulation in a regular order for rows and columns for pixels A to I included in unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, pixel signals are output in the order "ABCDEFGHI" in the example of Figure 6.

[0055] By configuring the circuit based on the unit area 131 in this way, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output pixel signals at different frame rates between the unit areas 131. Furthermore, by causing unit areas 131 included in some areas of the imaging chip 113 to accumulate charge (imaging) while unit areas 131 included in other areas are rested, it is possible to cause imaging to be performed only in a predetermined area of ​​the imaging chip 113 and output the corresponding pixel signals. Furthermore, it is also possible to switch the area where charge accumulation (imaging) is performed between frames (the area subject to accumulation control) to cause different areas of the imaging chip 113 to sequentially perform imaging and output pixel signals.

[0056] Fig. 7 is a block diagram showing the functional configuration of the image sensor 100 corresponding to the circuit illustrated in Fig. 6. An analog multiplexer 411 sequentially selects nine PDs 104 forming a unit area 131 and outputs the respective pixel signals to output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed on the image sensor chip 113 together with the PDs 104.

[0057] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 that performs CDS and A / D conversion and is formed in the signal processing chip 111. The A / D converted pixel signals are passed to a demultiplexer 413 and stored in pixel memories 414 corresponding to the respective pixels. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 112.

[0058] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that while Fig. 7 shows connections for one unit area 131, in reality, there is one arithmetic circuit for each unit area 131 and they operate in parallel. However, there does not need to be a arithmetic circuit 415 for each unit area 131; for example, one arithmetic circuit 415 may perform sequential processing while referring to the values ​​of the pixel memories 414 corresponding to each unit area 131 in order.

[0059] As described above, output wiring 308 is provided corresponding to each unit area 131. Since the imaging element 100 has the imaging chip 113, signal processing chip 111, and memory chip 112 stacked on top of each other, by using bumps 109 for electrical connection between the chips for these output wiring 308, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0060] <Explanation of distance measurement> FIG. 8 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 100. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of FIG. 8, 15 focus detection pixel lines 60 are arranged at predetermined intervals. The focus detection pixels that make up the focus detection pixel lines 60 output image signals for ranging. Normal imaging pixels are provided at pixel positions other than those of the focus detection pixel lines 60 on the imaging chip 113. The imaging pixels output image signals for monitoring moving objects, obstacles, etc.

[0061] 9 is an enlarged view of a region including a portion of one of the focus detection pixel lines 60. Illustrated in FIG. 9 are red pixels R, green pixels G (Gb, Gr), and blue pixels B, as well as focus detection pixels P1 and P2. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0062] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B represent the light-receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system. That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and light that passes through these mask openings reaches the light-receiving regions of the imaging pixels.

[0063] The shape of the light receiving regions (mask openings) of the red pixel R, green pixel G (Gb, Gr), and blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0064] The semicircular regions illustrated for the focus-detection pixels P1 and P2 indicate the light-receiving regions of the focus-detection pixels. Specifically, the focus-detection pixel P1 has a semicircular mask opening to the left of the pixel position in FIG. 9, and light passing through this mask opening reaches the light-receiving portion of the focus-detection pixel P1. Meanwhile, the focus-detection pixel P2 has a semicircular mask opening to the right of the pixel position in FIG. 9, and light passing through this mask opening reaches the light-receiving portion of the focus-detection pixel P2. In this way, the focus-detection pixel P1 and the focus-detection pixel P2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system.

[0065] The positions of the focus detection pixel lines on the imaging chip 113 are not limited to the positions illustrated in Fig. 8. The number of focus detection pixel lines is also not limited to the example in Fig. 8. Furthermore, the shape of the mask openings in the focus detection pixels P1 and P2 is not limited to semicircular, and may be rectangular, for example, formed by horizontally dividing the square-shaped light-receiving regions (mask openings) in the imaging pixels R, G, and B.

[0066] Furthermore, the focus detection pixel line on the imaging chip 113 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 113. Image sensors in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Figure 9 are well known, and detailed illustration and description of these pixels will be omitted.

[0067] 9, a so-called 1PD structure has been described in which each of the focus detection pixels P1 and P2 receives one of a pair of light beams used for focus detection. Alternatively, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2007-282107, a so-called 2PD structure in which each of the focus detection pixels receives both of a pair of light beams used for focus detection may be used. Using this 2PD structure makes it possible to read image data from the focus detection pixels, preventing the focus detection pixels from becoming defective pixels.

[0068] In this embodiment, the focus adjustment state (defocus amount) of the imaging optical system is calculated by detecting the image shift amount (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system based on the ranging image signals output from the focus detection pixel P1 and the focus detection pixel P2.

[0069] Generally, the pair of images move closer to each other in a so-called front-focus state, where the imaging optical system forms a sharp image of the object (e.g., a preceding vehicle) in front of the intended focal plane, and move farther away from each other in a so-called back-focus state, where the imaging optical system forms a sharp image of the object behind the intended focal plane. In a focused state where a sharp image of the object is formed on the intended focal plane, the pair of images are relatively aligned. Therefore, the amount of relative positional deviation between the pair of images corresponds to the distance to the object (depth information).

[0070] The calculation of the defocus amount based on the phase difference is well known in the field of cameras, and therefore a detailed description thereof will be omitted. Here, since there is a one-to-one correspondence between the defocus amount and the distance to the object, the distance from the camera to each object can be calculated by calculating the defocus amount for each object. In other words, the distance to each object can be measured (ranging) at multiple positions on the shooting screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a formula or a lookup table and stored in a non-volatile memory or the like.

[0071] <Automobile control> The control executed by the control unit 19 of the automobile 10 will be described below with reference to the flowchart of Fig. 10. Note that this flowchart is initiated by starting the automobile 10, for example, by starting the engine or the driving system. A program for executing the processing according to the flowchart of Fig. 10 is stored in a storage medium such as a ROM in the control unit 19 or in the storage unit 17 of the automobile 10.

[0072] In step S1, the control unit 19 starts imaging by the imaging element 100 of the photoelectric conversion unit 15. As described above, the imaging unit 5 (optical system 14, photoelectric conversion unit 15) of the automobile 10 acquires images of the front and rear of the automobile 10, respectively.

[0073] In step S2, the control unit 19 communicates with other vehicles 20 (vehicles traveling in the same lane as the automobile 10, as well as vehicles traveling in a different lane but in the same traveling direction as the automobile 10) via the communication unit 16. In this embodiment, as shown in FIG. 11(a), it is assumed that the preceding vehicle 73A traveling in the same lane as the automobile 10 is an automatically driven vehicle, and the preceding vehicle 72A traveling in the adjacent traveling lane (same traveling direction) is a manually driven vehicle. It is also assumed that the following vehicle 73B traveling in the same lane as the automobile 10 is an automatically driven vehicle, and the following vehicle 72B traveling in the adjacent traveling lane (same traveling direction) is a manually driven vehicle.

[0074] Whether a vehicle around the automobile 10 is an autonomous vehicle or a manually driven vehicle is determined based on the communication results of known vehicle-to-vehicle communication by the communication unit 16. Furthermore, if an identification mark or the like is displayed on the other vehicle 20, the determination may be based on the imaging results of the imaging unit 5 (optical system 14, photoelectric conversion unit 15). The identification mark may be a predetermined mark or code displayed on the body of the vehicle, or identification information may be displayed on a display unit (not shown) provided on the roof or the like of the vehicle. Note that, for other vehicles 20 for which it is not possible to determine whether they are autonomous vehicles or manually driven vehicles based on the communication results (communication failure) or imaging results, the control unit 19 presumes that the other vehicle 20 is a manually driven vehicle.

[0075] In step S3, the control unit 19 sets imaging conditions individually for each of the unit regions 131 (FIG. 5) of the image sensor 100 of the photoelectric conversion unit 15. FIGS. 11(b) and 11(c) are diagrams schematically showing images of a subject (object) formed on the image sensor 100 that captures images of the front and rear of the automobile 10, respectively. In reality, an inverted, reversed image is formed, but for ease of understanding, it is shown as an upright, normal image. White line 80a represents the dividing line on the left side of the road as viewed in the direction of travel, white line 80b represents the boundary line of the driving lane, and white line 80c represents the dividing line on the right side of the road.

[0076] As described above, since the vehicles 72A and 72B traveling in the lane next to the automobile 10 are manually driven vehicles, the control unit 19 sets the area including the vehicle 72A in Fig. 11(b) as the attention area 71A. Then, the frame rate of the unit area of ​​the image sensor 100 corresponding to the attention area 71A is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%, which is lower than that of the normal area.

[0077] 11(c), the control unit 19 designates the area including the vehicle 72B as the attention area 71B. Then, the control unit 19 sets the frame rate of the unit area of ​​the image sensor 100 corresponding to the attention area 71B higher than the frame rate of the normal area (for example, 60 fps), and sets the thinning rate to 0 to 20%, which is lower than that of the normal area. The control unit 19 further changes the setting of this thinning rate in accordance with the moving speed of the automobile 10 or the relative moving speed between the automobile 10 and other vehicles 20. For example, the faster the relative moving speed becomes, the lower the thinning rate becomes.

[0078] While FIG. 11(b) illustrates an example in which all areas except the attention area 71A are normal areas, the area surrounding the autonomous vehicle 73A may be a quasi-attention area 74A, and the area excluding the attention area 71A and the quasi-attention area 74A may be normal areas. Also, FIG. 11(c) illustrates an example in which all areas except the attention area 71B are normal areas, the area surrounding the autonomous vehicle 73B may be a quasi-attention area 74B, and the area excluding the attention area 71B and the quasi-attention area 74B may be normal areas. The control unit 19 may set different imaging conditions for the quasi-attention areas 74A and 74B for fully autonomous driving and semi-autonomous driving. In this case, the control unit 19 may set the frame rate of the image sensor 100 for semi-autonomous driving to be higher than the frame rate of the image sensor 100 for fully autonomous driving. Also, the control unit 19 may set the imaging area as a normal area for fully autonomous driving.

[0079] The control unit 19 sets the frame rate of the unit areas of the imaging element 100 corresponding to the quasi-attention areas 74A and 74B, which respectively capture images of the front and rear of the automobile 10, to be lower than the frame rate of the normal area (for example, 30 fps), and sets the thinning rate to approximately 30 to 60%.

[0080] Furthermore, the control unit 19 may set an area including a white line on the road as an attention area in addition to the attention areas 71A and 71B. Then, for the image pickup device 100 capturing images in front of and behind the automobile 10, the frame rate of the unit area of ​​the image pickup device 100 corresponding to the attention area is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%. In this way, by varying the imaging conditions for each unit area of ​​the imaging element 100 when imaging a manually driven vehicle and an autonomously driven vehicle, the imaging element 100 can be used efficiently, and power consumption and heat generation can be reduced.

[0081] 10, the control unit 19 determines whether communication with the traffic light 40 is possible via the communication unit 16, i.e., whether the vehicle is approaching the traffic light 40 (intersection). If communication with the traffic light 40 is not possible (outside the communication area), the control unit 19 makes a negative decision in step S4 and proceeds to step S7. On the other hand, if communication with the traffic light 40 is possible (within the communication area), the control unit 19 makes a positive decision in step S4 and proceeds to step S5.

[0082] In step S5, the control unit 19 receives information based on an image acquired by the photoelectric conversion unit 44 of the traffic light 40 (traffic light 40a for automobiles or traffic light 40b for pedestrians). For example, when the automobile 10 is turning left (or turning right in areas where vehicles keep to the right, such as the United States), the control unit 19 receives information about a person from the traffic light 40b for pedestrians. In this case, the control unit 46 of the traffic light 40 determines the presence or absence of a pedestrian based on the image acquired by the photoelectric conversion unit 44, and the control unit 19 receives the information about the pedestrian determined by the control unit 46. The control unit 19 of the automobile 10 may receive image data acquired by the photoelectric conversion unit 44 of the traffic light 40, and determine the presence or absence of a pedestrian based on the received image data.

[0083] When the automobile 10 makes a right turn (or a left turn in an area where vehicles drive on the right), the control unit 19 receives information from the automobile traffic light 40a, such as whether a vehicle moving straight in the oncoming lane is an autonomous vehicle or a manually driven vehicle. Furthermore, the control unit 19 receives information about the switching of traffic signals (for example, signal switching information such as a change from green to red in a few seconds) from the automobile traffic light 40a.

[0084] In step S6, based on the information obtained in step S5, the control unit 19 sets the imaging conditions of the image sensor 100 of the photoelectric conversion unit 15. For example, when the automobile 10 turns left at an intersection, the control unit 19 sets the frame rate of the unit area of ​​the image sensor 100 corresponding to the left side of the imaging screen to be higher than the frame rate of the unit area corresponding to the right side of the imaging screen, or changes the frame rate according to the speed of the automobile 10 or the moving speed of pedestrians (for example, 4 km / h).

[0085] For example, if automobile 10 traveling at 50 km / h slows down to about 10 km / h to make a left turn, the frame rate of the unit area corresponding to the left side of the shooting screen is reduced compared to before the speed reduction. Furthermore, when automobile 10 turns left, control unit 19 sets the imaging conditions of image sensor 100 depending on whether a pedestrian crossing the sidewalk is approaching automobile 10 or moving away from automobile 10. That is, control unit 19 increases the frame rate and decreases the thinning rate for the unit area of ​​image sensor 100 corresponding to a person approaching automobile 10, and decreases the frame rate and increases the thinning rate for the unit area of ​​image sensor 100 corresponding to a pedestrian moving away from automobile 10 (especially a person who has already crossed the crosswalk where automobile 10 is scheduled to pass).

[0086] Furthermore, when automobile 10 turns right at an intersection, if another vehicle 20 coming straight from the oncoming lane is a manually driven vehicle, control unit 19 makes the frame rate of the unit area corresponding to the right side of the captured image relatively higher than the frame rate of the unit area corresponding to the left side of the captured image, thereby lowering the thinning rate. Furthermore, if a pedestrian crossing the crosswalk ahead of the right turn is approaching automobile 10, control unit 19 further increases the frame rate of the unit area of ​​image sensor 100 corresponding to the approaching pedestrian, thereby further lowering the thinning rate.

[0087] When the automobile 10 turns right or left, the control unit 19 may predict the imaging area that should be focused on (predict the imaging area where pedestrians, etc. may be captured) depending on the operating state of the turn signal switch and the amount of steering wheel operation, and change the imaging conditions.

[0088] The control unit 19 also receives information about the change in traffic signal from the automobile traffic light 40a and changes the imaging conditions of the image sensor 100 based on that information. For example, the control unit 19 receives information about the change in traffic signal, that is, the signal will change from green to red in a few seconds, from the automobile traffic light 40a, and when the automobile 10 decelerates, the control unit 19 controls the image sensor 100 to capture the forward image so as to lower the frame rate or increase the thinning rate compared to before deceleration. On the other hand, the control unit 19 controls the image sensor 100 to capture the rearward image so as to maintain the imaging conditions as they are. In addition, since it is expected that a following vehicle will approach the automobile 10 when the automobile 10 decelerates, the frame rate of the image sensor 100 capturing images of the rear may be controlled to be higher than before deceleration or the thinning rate may be controlled to be lower. When the speed of the automobile 10 changes, the control unit 19 may predict the speed change according to the amount of operation of the brake or accelerator (amount of depression of the pedal) and change the settings of the imaging conditions.

[0089] In step S7, the control unit 19 determines whether the engine (or driving system) is on. If the engine (or driving system) is on, the control unit 19 makes an affirmative decision in step S7 and repeats the processing from step S2 onwards. If the engine (or driving system) is off, the control unit 19 makes a negative decision in step S7 and ends the processing according to this flowchart.

[0090] <Traffic signal control> Next, the control executed by the control unit 46 of the traffic light 40 will be described with reference to the flowchart of Fig. 12. A program for executing the processing according to the flowchart of Fig. 12 is stored in a storage medium such as a ROM in the control unit 46 or in the storage unit 45.

[0091] In step S10, the control unit 46 determines whether a traffic signal has been received from the traffic signal generating device 30. When the control unit 46 receives a traffic signal from the traffic signal generating device 30, it makes a positive determination in step S10 and proceeds to step S11. When the control unit 46 has not received a traffic signal from the traffic signal generating device 30, it makes a negative determination in step S10 and waits for reception.

[0092] In step S11, the control unit 46 controls the display of the display unit 42. For example, the control unit 46 controls the signal light display of the display unit 42 to change from red to green in accordance with the traffic signal received from the traffic signal generating device 30.

[0093] In step S12, the control unit 46 communicates with one or more vehicles or other traffic lights 40. The vehicles with which the control unit 46 communicates may include the automobile 10 and other vehicles 20 equipped with a communication unit 21. The communication target vehicle may be a vehicle within a predetermined range from the intersection or traffic light 40, or may be a vehicle with which communication can be made via an information providing system (not shown) installed on the road.

[0094] The control unit 46 acquires information from the communication target vehicle regarding the travel mode of the vehicle or vehicles around the vehicle, such as whether the vehicle or vehicles around the vehicle are autonomous or manually driven. Furthermore, the control unit 46 acquires information from the communication target vehicle regarding the driving conditions of the vehicle or vehicles around the vehicle. For example, the control unit 19 or 25 of the communication target vehicle predicts a course change (right or left turn) at an intersection based on the state of a turn signal switch for activating a blinker (directional indicator). The control unit 46 acquires right turn prediction information or left turn prediction information predicted by the communication target vehicle from the communication target vehicle as information regarding the driving conditions.

[0095] The control unit 46 may determine whether a vehicle is an autonomous vehicle or a manually driven vehicle based on the results of communication with the communication target vehicle. Also, if an identification mark or the like is displayed on the vehicle, the control unit 46 may determine this based on the image capture results obtained by the imaging unit (optical system 43, photoelectric conversion unit 44). Furthermore, with regard to a change of path (right or left turn) of the vehicle, the control unit 46 may determine this from the operating state of the turn signal of the vehicle based on the image capture results obtained by the imaging unit (optical system 43, photoelectric conversion unit 44), or may determine whether the vehicle is in a left turn lane or a right turn lane.

[0096] The control unit 46 also communicates with other traffic lights 40, including the traffic light 40a for automobiles or the traffic light 40b for pedestrians, to acquire information on traffic conditions at intersections, etc. The control unit 46 further communicates with the traffic signal generating device 30 involved in generating the traffic signals for each traffic light, as necessary, to acquire information on the display status of the traffic signal.

[0097] In step S13, the control unit 46 sets the imaging conditions of the image sensor 100 of the photoelectric conversion unit 44 based on the display status of the signal lamp on the display unit 42 and the information acquired in step S12. Details of setting the imaging conditions of the image sensor 100 of the photoelectric conversion unit 44 will be described later.

[0098] In step S14, the control unit 46 causes the imaging unit (optical system 43, photoelectric conversion unit 44) to capture an image under the imaging conditions set in step S13.

[0099] In step S15, the control unit 46 transmits the image data acquired in step S14 to the automobile 10, another vehicle 20, another traffic light 40, etc. via the communication unit 41. The control unit 46 also transmits various pieces of information extracted by performing image processing or image analysis on the image data, such as data on the direction and speed of the vehicle, and information on the driving lane of the vehicle estimated from identifying the operating state of the turn signals, etc.

[0100] Furthermore, the control unit 46 may recognize the estimated driving lane of the communication target vehicle (the automobile 10 or another vehicle 20) and obstacles (including vehicles and pedestrians) based on the analyzed information, generate a message based on the recognition result, and transmit the message from the communication unit 41 to the communication target vehicle. The message may be, for example, "A motorcycle is approaching from behind," "A pedestrian is crossing," or "An oncoming vehicle is proceeding straight." The control unit 46 repeatedly executes the processes from step S10 to step S15.

[0101] <Setting imaging conditions for traffic lights> Figure 13 is a diagram showing an example of control of imaging conditions in the imaging element 100 of the imaging unit 50-1, which is installed integrally with or in the vicinity of a traffic light 40a for automobiles, when the traffic light 40a for automobiles is displaying a green light.

[0102] 13 shows a traffic light 40a for automobiles for driving lane A, an imaging unit 50-1, and a traffic signal generation device 30-1. Other traffic lights and the like at the intersection are not shown. A control unit 46 of the traffic light 40a for automobiles sets the range of the driving lane in which the vehicle is moving with the green light within the imaging area 70 of the imaging unit 50-1 as a region of interest 71 (the shaded area).

[0103] The control unit 46 controls the unit region of the image sensor 100 corresponding to the attention region 71 by increasing the frame rate or decreasing the thinning rate compared to unit regions other than the attention region 71. Furthermore, when a vehicle 72 that is a manually driven vehicle and not an autonomous vehicle is present in the attention region 71 of FIG. 13 , the control unit 46 sets the region surrounding the vehicle 72 as a special attention region 75. The control unit 46 then increases the frame rate of the unit region of the image sensor 100 corresponding to the special attention region 75 even more than the unit region corresponding to the attention region 71. The control unit 46 also controls the thinning rate of the unit region of the image sensor 100 corresponding to the special attention region 75 so that it is even lower than the thinning rate of the unit region corresponding to the attention region 71.

[0104] In addition, the control unit 46 can also capture high-resolution images of areas within the attention area 71 that include vehicles 76 and 77 that are temporarily stopped to turn right or left by setting a thinning rate lower than that of the unit area corresponding to the attention area 71.

[0105] Figure 14 is a diagram showing an example of control of imaging conditions in the imaging element 100 of the imaging unit 50-1 installed integrally with or in the vicinity of a traffic light 40a for automobiles when the traffic light 40a for automobiles is displaying a red light.

[0106] 14 shows a traffic light 40a for automobiles for driving lane A, an imaging unit 50-1, and a traffic signal generation device 30-1. Other traffic lights and the like at the intersection are not shown. The control unit 46 sets a crosswalk where pedestrian 90 can enter and its vicinity as a region of interest 71 (shaded area) within the imaging area 70 of the imaging unit 50-1.

[0107] The control unit 46 controls the unit area of ​​the image sensor 100 corresponding to the attention area 71 by increasing the frame rate or decreasing the thinning rate compared to unit areas other than the attention area 71. Furthermore, when the control unit 46 recognizes a pedestrian 90, it sets an area of ​​a predetermined range including the pedestrian 90 as a special attention area 75. Then, it increases the frame rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 even more than the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 even more lower than the thinning rate of the unit area corresponding to the attention area 71.

[0108] 15 is a diagram showing an example of control of the imaging conditions of an imaging unit 50-2 installed integrally with or adjacent to a pedestrian traffic light 40b. In FIG. 15, the pedestrian traffic light 40b, the imaging unit 50-2, and the traffic signal generating device 30-2 are shown. Other traffic lights and the like at the intersection are not shown. The control unit 46 of the pedestrian traffic light 40b sets the crosswalk where a pedestrian 90 can enter and its vicinity as a region of interest 71 (shaded area) within the imaging area 70 of the imaging unit 50-2.

[0109] The control unit 46 controls the unit area of ​​the image sensor 100 corresponding to the attention area 71 by increasing the frame rate or decreasing the thinning rate compared to unit areas other than the attention area 71. Furthermore, when the control unit 46 recognizes a pedestrian 90, it sets an area of ​​a predetermined range including the pedestrian 90 as a special attention area 75. Then, it increases the frame rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 even more than the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 even more lower than the thinning rate of the unit area corresponding to the attention area 71.

[0110] FIG. 16(a) is a diagram illustrating an example of a scene captured by an imaging unit 50-2 installed at a pedestrian traffic light 40b. FIG. 16(b) is a diagram illustrating the setting of imaging conditions based on subject recognition results using image data acquired by the imaging unit 50-2. In FIG. 16(a), the imaging unit 50-2 having the above-described imaging element 100 is installed at the pedestrian traffic light 40b. The imaging element 100 according to this embodiment can measure movement not only in the up / down and left / right directions but also in the depth direction, making it possible to measure the speed Vo of a pedestrian 90, who is the subject.

[0111] 16(b), the control unit 46 sets an area within the imaging area 70 of the imaging unit 50-2 that includes a pedestrian 90 crossing the crosswalk as an attention area 71. Then, the control unit 46 makes the imaging conditions for the unit area of ​​the imaging element 100 that corresponds to the attention area 71 different from those for the unit areas other than the attention area 71. At this time, the control unit 46 makes the imaging conditions different depending on the speed Vo of the pedestrian 90.

[0112] For example, when the absolute value of the speed Vo of the pedestrian 90 is large, the control unit 46 controls the unit area of ​​the image sensor 100 corresponding to the attention area 71 including the pedestrian 90 by increasing the frame rate or decreasing the thinning rate compared to unit areas other than the attention area 71.

[0113] Furthermore, the control unit 46 may change the imaging conditions for the attention area 71 based on the positional relationship between the pedestrian 90 and nearby vehicles or buildings. For example, if a vehicle with its blinker on to turn right or left at an intersection is present in the imaging area 70, there is a possibility that the vehicle will enter the crosswalk. Therefore, the control unit 46 sets the area of ​​the attention area 71 that is closer to the vehicle as a special attention area. Then, the control unit 46 sets the frame rate of the unit area of ​​the image sensor 100 that corresponds to the special attention area to be higher than that of the unit area that corresponds to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of ​​the image sensor 100 that corresponds to the special attention area to be lower than that of the unit area that corresponds to the attention area 71.

[0114] Furthermore, the control unit 46 may also set different imaging conditions for multiple pixels or regions 78P indicated by diagonal lines and multiple pixels or regions 78S not indicated by diagonal lines in the region of interest 71. In the example of Fig. 16(b), different imaging conditions are set for pixels or regions adjacent to each other in the vertical and horizontal directions in a checkerboard pattern, but this is not a limitation.

[0115] Furthermore, when control unit 46 recognizes other objects such as pedestrians or bicycles within imaging area 70, control unit 46 may add areas including each of these multiple objects to attention area 71. Then, in the multiple attention areas 71, imaging conditions may be different between multiple pixels or areas 78P indicated by diagonal lines and multiple pixels or areas 78S not indicated by diagonal lines.

[0116] Fig. 17 is a diagram illustrating an example of a scene captured by an imaging unit 50-1 installed in a traffic light 40a for automobiles at an intersection. In Fig. 17, the traffic light 40a for automobiles is equipped with a display unit 42a having indicator lights that indicate whether to go straight, turn left, or turn right. An example of setting imaging conditions based on subject recognition results using image data captured by the imaging unit 50-1 will be described.

[0117] 17 shows a state in which the display unit 42a of a traffic light 40a has the indicator lights indicating that straight travel is permitted and that a left turn is permitted lit, and a right turn is being waited for. The control unit 46 of the traffic light 40a for automobiles controls the frame rate of the unit area of ​​the image sensor 100 corresponding to the driving lane A, through which vehicles traveling straight or turning left pass, and the driving lane B, through which vehicles traveling straight pass, by setting the frame rate higher or the thinning rate lower than the frame rate of the unit area corresponding to the other driving lane C. In other words, the control unit 46 controls the frame rate of the unit area corresponding to the driving lane C by setting the frame rate lower or the thinning rate higher. Note that lowering the frame rate also includes setting the unit area so that no image is captured.

[0118] In controlling the imaging conditions of the image sensor 100 in the automobile 10 or traffic light 40 (traffic light 40a for automobiles, traffic light 40b for pedestrians) according to the embodiments described above, the imaging conditions may be changed almost simultaneously with the timing of switching the display on the display unit 42, or the imaging conditions may be changed at a fixed time interval from the timing of switching the display. Alternatively, for a fixed time immediately after switching the display, the imaging conditions may be changed using both the attention area set before the display switching and the attention area to be set after the display switching as the attention area.

[0119] According to the above-described embodiment, the following advantageous effects can be obtained. (1) The imaging device of the automobile 10 (or traffic light 40) includes an imaging unit 5 (or imaging unit 50) having an imaging element 100 capable of setting imaging conditions for multiple areas, and a control unit 19 (or control unit 41) that sets imaging conditions for multiple unit areas based on the driving style of other surrounding vehicles 20. This allows the imaging element 100 to set imaging conditions suitable for the driving style of the surrounding vehicles 20.

[0120] (2) The control unit 19 (or the control unit 41) sets different imaging conditions for each area in which other vehicles 20 with different driving styles are imaged, and therefore different imaging conditions can be set for the imaging element 100 between areas in which vehicles 20 with different driving styles are imaged.

[0121] (3) The driving modes of the other vehicles 20 are an automatic driving mode and a manual driving mode, and the control unit 19 (or control unit 41) sets different imaging conditions for the area for imaging the vehicles 20 using the automatic driving mode and the area for imaging the vehicles 20 using the manual driving mode, so that different imaging conditions can be set for the imaging element 100 between the area for imaging the vehicles 20 using the automatic driving mode and the area for imaging the vehicles 20 using the manual driving mode.

[0122] (4) The control unit 19 (or control unit 41) sets the frame rate of the area for capturing images of the manually driven vehicle 20 higher than the frame rate of the area for capturing images of the automatically driven vehicle 20. This increases the frequency with which the manually driven vehicle 20 is captured compared to the frequency with which the automatically driven vehicle 20 is captured, thereby increasing the attention paid to the manually driven vehicle 20. In other words, it becomes possible to quickly and accurately obtain information about the unpredictable behavior of the manually driven vehicle 20.

[0123] (5) The control unit 19 (or control unit 41) sets the pixel thinning rate of the area capturing an image of the manually driven vehicle 20 to be lower than the pixel thinning rate of the area capturing an image of the automatically driven vehicle 20. This allows the amount of information about the manually driven vehicle 20 to be greater than the amount of information about the automatically driven vehicle 20. In other words, it becomes possible to obtain more accurate information about the unpredictable behavior of the manually driven vehicle 20.

[0124] (6) The vehicle is provided with a control unit 19 (or control unit 41) that acquires information regarding the modes of travel of other surrounding vehicles 20. Therefore, even if the surrounding vehicles 20 change, for example, the imaging conditions can be set for the imaging element 100 based on the latest acquired information.

[0125] (7) The control unit 19 (or the control unit 41) is configured to acquire information through communication with other vehicles 20, and therefore can appropriately set the imaging conditions of the imaging element 100 based on new information acquired through communication.

[0126] (8) The control unit 19 (or the control unit 41) acquires information by capturing images of other vehicles 20 using the imaging unit 5 (or the imaging unit 50). Therefore, even in situations where communication is not possible, the control unit 19 can appropriately set the imaging conditions of the imaging element 100 based on the new information.

[0127] (9) The control unit 19 acquires information from a traffic light 40 different from that of the other vehicles 20. This allows the control unit 19 to appropriately set the imaging conditions of the imaging element 100 based on the new information even in a situation where communication with the other vehicles 20 is not possible.

[0128] (10) The control unit 19 (or control unit 41) acquires information indicating whether the other vehicle 20 is an autonomous vehicle or a manually driven vehicle, and sets different imaging conditions for the area for imaging the autonomous vehicle and the area for imaging the manually driven vehicle. This makes it possible to appropriately set imaging conditions for each unit area on the imaging surface of the image sensor 100 so as to, for example, increase the attention paid to manually driven vehicles compared to autonomous vehicles. In particular, by increasing the frame rate and decreasing the pixel thinning rate for manually driven vehicles, it is possible to quickly and accurately acquire information about the unpredictable behavior of manually driven vehicles.

[0129] (11) Since the automobile 10 is equipped with the imaging devices (1) to (10) above, imaging conditions for the imaging devices can be set appropriately in accordance with the movement patterns of other vehicles 20 around the automobile 10.

[0130] (12) The control unit 19 (or control unit 41) of the automobile 10 changes the imaging conditions of the image sensor 100 in response to the operation of the steering wheel, turn signal switch, accelerator, brake, etc. of the automobile 10. This allows the imaging conditions to be appropriately set for each unit area on the imaging surface of the image sensor 100 in response to changes in the course, speed, etc. of the automobile 10.

[0131] (13) The traffic light 40 includes an imaging unit 50 equipped with an imaging element 100 that can independently set imaging conditions for multiple areas, and a control unit 46 that sets imaging conditions for multiple areas based on information about the movement of the automobile 10 and other vehicles 20. This allows imaging conditions to be set for the imaging element 100 according to the traffic conditions of the automobile 10 and other vehicles 20 moving through an intersection, etc.

[0132] (14) The information regarding the movement of the automobile 10 and other vehicles 20 includes a signal that permits the automobile 10 and other vehicles 20 to move and a signal that does not permit the automobile 10 and other vehicles 20 to move, and the control unit 46 sets the imaging conditions based on each signal. This makes it possible to appropriately set the imaging conditions of the imaging element 100 for the cases where the automobile 10 and other vehicles 20 are moving and where the automobile 10 and other vehicles 20 are not moving.

[0133] (15) The signals permitting the movement of the automobile 10 and the other vehicle 20 include a signal permitting straight ahead travel, a signal permitting a left turn, and a signal permitting a right turn, and the control unit 46 sets the imaging conditions based on each of these signals. This allows the imaging conditions of the imaging element 100 to be appropriately set when the automobile 10 and the other vehicle 20 are going straight ahead, turning left, or turning right.

[0134] (16) The control unit 46 changes the imaging conditions in response to changes in information regarding the movement of the automobile 10 and other vehicles 20, so that the imaging conditions of the imaging element 100 can be appropriately changed at the timing when the above signal changes.

[0135] (17) The traffic light 40 is equipped with a control unit 46 that acquires information regarding the movement of surrounding automobiles 10 and other vehicles 20. Therefore, even if the surrounding other vehicles 20 change, for example, the imaging conditions for the imaging element 100 can be set based on the latest acquired information.

[0136] (18) The control unit 46 acquires information through communication with the automobile 10 and other vehicles 20, and can therefore appropriately set the imaging conditions of the imaging element 100 based on new information acquired through communication.

[0137] (19) The control unit 46 acquires information by capturing images of the automobile 10 and other vehicles 20 using the imaging unit 50, and therefore can appropriately set the imaging conditions of the imaging element 100 based on new information even in situations where communication is not possible.

[0138] (20) The control unit 46 acquires information indicating whether the automobile 10 and the other automobiles 20 are autonomous or manually driven automobiles as movement-related information, and sets different imaging conditions for the area capturing images of autonomously driven automobiles and the area capturing images of manually driven automobiles. This makes it possible to appropriately set imaging conditions for each unit area on the imaging surface of the image sensor 100 so as to, for example, increase the attention paid to manually driven automobiles compared to autonomously driven automobiles. In particular, by increasing the frame rate and decreasing the pixel thinning rate for manually driven automobiles, it is possible to quickly and accurately acquire information about the unpredictable behavior of manually driven automobiles.

[0139] (21) The control unit 46 acquires information indicating a change in the course of the automobile 10 and other vehicles 20, and changes the area for setting the imaging conditions based on the change in the course of the automobile 10 and other vehicles 20. This allows the imaging conditions of the image sensor 100 to be changed appropriately at the timing of the change in course.

[0140] (22) The automobile 10 is equipped with a communication unit 41 that communicates with other automobiles 10 and other cars 20 that are different from the other automobiles 10 and other cars 20, so that, for example, information about other cars 20 around a traffic light 40 can be transmitted to the automobile 10.

[0141] (23) The automobile 10 and the traffic light 40 measure the speed at which the object being imaged is moving, and change the imaging conditions of the image sensor 100 according to the magnitude and direction of the speed. Therefore, the imaging conditions can be appropriately set for each area on the imaging surface of the image sensor 100 according to the movement of the object.

[0142] (24) The automobile 10 acquires information such as the number of seconds until the traffic light changes and reflects this information in the driving situation of the automobile 10. Therefore, smooth driving can be achieved by taking the traffic light change into consideration in advance.

[0143] (25) For a certain period of time immediately after a traffic light changes, the automobile 10 or the traffic light 40 sets the imaging conditions of the image sensor 100 so that the area of ​​interest includes both the area of ​​interest set in the immediately preceding traffic light and the area of ​​interest to be set in the new traffic light. This allows the imaging conditions of the image sensor 100 to be set appropriately for the transient state of the traffic light change.

[0144] (26) The imaging system 1 including the automobile 10 and the traffic light 40 can realize a more organized traffic system based on accurate and efficient information acquisition through imaging and communication.

[0145] In the above-described embodiment, the imaging unit 5 and the imaging unit 50 are controlled by the control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40, respectively, but part of the control of the imaging unit 5 and the imaging unit 50 may be performed by a control circuit (CPU, etc.) within the imaging unit.

[0146] Furthermore, some of the processing performed by the control unit 46 of the traffic light 40 may be performed by the control unit 34 of the traffic light information generating device 30. The imaging unit 50 such as a camera does not necessarily have to be attached to the traffic light 40, and may be installed depending on the traffic signal or traffic conditions at an intersection, etc.

[0147] Furthermore, in the above embodiment, the display unit and audio playback unit of the navigation system 13 are used to display the message, but a separate display and playback device may also be used. Furthermore, a display and playback device consisting of a HUD (Head Up Display) that projects information onto the windshield of the automobile 10 and a speaker that plays back audio information may also be used.

[0148] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment.

[0149] (Variation 1) The control unit 46 of the traffic light 40 may obtain information about the proportion of autonomous vehicles among vehicles on a road or at an intersection through imaging by the imaging unit 50 or communication by the communication unit 41, and change the imaging conditions according to that proportion. For example, the control unit 46 may control the thinning rate to be higher during times when the proportion of autonomous vehicles is high than when the proportion of autonomous vehicles is low. This can save power consumption and enable more efficient imaging.

[0150] (Variation 2) The control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40 may identify whether the vehicle is an autonomous vehicle or a manually driven vehicle by capturing an image, and may also identify signs such as a novice driver sign or an elderly driver sign. Different imaging conditions are set for the unit areas of the image sensor 100 corresponding to the novice driver vehicle and the elderly driver vehicle. For example, a vehicle displaying a novice driver sign may be imaged at a frame rate that is higher than the frame rate for the unit areas of the image sensor 100 corresponding to the manually driven vehicle. This allows the imaging conditions to be set appropriately for each unit area on the imaging surface of the image sensor 100 depending on the object.

[0151] (Variation 3) In the above explanation, distance measurement and detection of surrounding moving objects and obstacles are performed by capturing images using the image sensor 100, but a radar (not shown) may also be used in combination. This makes it possible to take advantage of the characteristics of the image sensor 100 and the radar to obtain more reliable traffic information.

[0152] Although various embodiments and modifications have been described above, the present invention is not limited to these. Aspects in which the configurations shown in the embodiments and modifications are used in combination are also included within the scope of the present invention. Other aspects conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0153] 1. Imaging system 5, 50 (50-1, 50-2)...imaging unit 10. Automobiles 19, 46...Control section 20...Other cars 30…Traffic signal generation device 40...Traffic light 40a...traffic lights for automobiles 40b...Pedestrian traffic light 42...Display section 70...imaging area 71, 71A, 71B... Area of ​​interest 72, 72A, 72B, 73A, 73B, 76, 77…Vehicles 74A, 74B...Semi-attention area 75…Special attention area 90...Pedestrian 100...Image sensor 113...imaging chip

Claims

[Claim 1] an imaging unit including an imaging element capable of independently setting imaging conditions for a plurality of regions; a setting unit that sets imaging conditions for the plurality of areas based on a speed and a moving direction of at least one of a vehicle and a pedestrian captured by the imaging unit as a moving object; An imaging device comprising:

Images