Vehicle

Abstract

To appropriately set imaging conditions of an imaging unit.SOLUTION: A drive support device is mounted on a vehicle. The drive support device includes: an imaging unit configured to image the outside of the vehicle; a ranging unit configured to measure distance to an object present outside of the vehicle; a positioning unit configured to measure a current position of the vehicle; a control unit configured to generate data for performing drive support of the vehicle based on information from the imaging unit, the ranging unit, and the positioning unit; and a receiving unit configured to receive information transmitted from other vehicle imaged by the imaging unit. The control unit determines whether or not other vehicle is traveling by automatic driving based on information from the receiving unit to change a frame rate and a thinning-out rate that are imaging conditions of the imaging unit according to the determination results.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vehicle. [Background technology]

[0002] Technologies have been developed that detect the vehicle's driving environment based on images acquired by a camera mounted on the vehicle, and perform automatic driving control, such as following the vehicle ahead, and driving assistance, such as warnings, braking, and steering assistance, based on the detected driving environment data (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-79424 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is expected that in the future there will be a mixture of cars with automatic driving control and cars that are manually driven by a driver, but there were not many proposals regarding this point. [Means for solving the problem]

[0005] According to a first aspect of the present invention, a vehicle is equipped with a driving assistance device, the driving assistance device having an imaging unit that images the outside of the vehicle, a ranging unit that measures the distance to an object outside the vehicle, a positioning unit that measures the current position of the vehicle, a control unit that generates data for providing driving assistance to the vehicle based on information from the imaging unit, the ranging unit, and the positioning unit, and a receiving unit that receives information transmitted from another vehicle imaged by the imaging unit, and the control unit determines whether the other vehicle is a vehicle traveling in autonomous driving mode based on the information from the receiving unit, and changes the imaging conditions of the imaging unit, namely the frame rate and thinning rate, in accordance with the result of the determination. [Brief description of the drawings]

[0006] [Figure 1] 1 is a block diagram showing a configuration of an imaging system according to an embodiment; [Diagram 2] FIG. 1 is a diagram illustrating an example of the arrangement of traffic lights at an intersection. [Diagram 3] FIG. 1 illustrates an example of a traffic light for automobiles. [Figure 4] FIG. 2 is a cross-sectional view of a stacked imaging element. [Diagram 5] 2 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Figure 6] FIG. 2 is a diagram illustrating a circuit of a unit area. [Figure 7] FIG. 2 is a block diagram showing a functional configuration of an imaging element. [Figure 8] 4 is a diagram illustrating an example of the positions of focus detection pixels on an imaging surface; [Figure 9] FIG. 2 is an enlarged view of a region including a portion of a focus detection pixel line. [Figure 10] 4 is a flowchart illustrating control by a control unit of the automobile. [Figure 11] FIG. 11(a) is a diagram explaining the positional relationship of the automobile, FIG. 11(b) is a diagram showing a schematic view of a subject image captured by a front camera, and FIG. 11(c) is a diagram showing a schematic view of a subject image captured by a rear camera. [Figure 12] 4 is a flowchart illustrating control by a control unit of a traffic light. [Figure 13] FIG. 2 is a diagram illustrating the control of an imaging unit of a traffic light for automobiles. [Figure 14] FIG. 2 is a diagram illustrating the control of an imaging unit of a traffic light for automobiles. [Figure 15] 11 is a diagram illustrating the control of an imaging unit of a pedestrian traffic light. FIG. [Figure 16] FIG. 16(a) is a diagram illustrating an example of an image captured by an imaging unit of a pedestrian traffic light, and FIG. 16(b) is a diagram for explaining settings of imaging conditions. [Figure 17] 1A and 1B are diagrams illustrating examples of scenes captured by an imaging unit of a traffic light for automobiles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] 1 is a block diagram illustrating a configuration of an imaging system 1 including an imaging device according to an embodiment. The imaging system 1 uses an automobile 10, another vehicle 20, a traffic signal generating device 30, and a traffic light 40. Instead of the traffic light 40 or in combination with the traffic light 40, an information providing system installed on the road or VICS (registered trademark: Vehicle Information and Communication System) may be used.

[0008] (Car 10) The automobile 10 includes a vehicle operation unit 11, a GPS device 12, a navigation system 13, an optical system 14, a photoelectric conversion unit 15, a communication unit 16, a storage unit 17, a sensor 18, and a control unit 19. The automobile 10 has a basic configuration as an automobile, although a detailed description is omitted.

[0009] The vehicle operation unit 11 includes various operating members related to the operation of the vehicle, such as a steering wheel, a turn signal switch, a shift lever, an accelerator, a brake, and a switch for switching between an automatic driving mode and a manual driving mode.

[0010] The GPS device 12 calculates the position (longitude, latitude, etc.) of the automobile 10 based on signals obtained by receiving radio waves from GPS satellites. The position information calculated by the GPS device 12 is output to the navigation system 13 and the control unit 19.

[0011] The navigation system 13 is a system that detects the current position of the automobile 10 using the GPS device 12 etc., obtains map data corresponding to the current position from a storage medium or a network, displays it on a liquid crystal monitor, and provides guidance on a driving route to an input destination. This navigation system 13 has an operation unit that accepts operations from a user, the above-mentioned liquid crystal monitor, a speaker that provides voice guidance, a reading unit that reads map data, etc.

[0012] The optical system 14 is composed of multiple lenses, and forms a subject image on the photoelectric conversion unit 15. When the optical system 14 is directed toward the front of the automobile 10, the photoelectric conversion unit 15 acquires an image in the traveling direction of the automobile 10. When the optical system 14 is directed toward the rear of the automobile 10, the photoelectric conversion unit 15 acquires an image in the opposite direction to the traveling direction of the automobile 10. The optical system 14 has an angle of view that corresponds to multiple driving lanes (two or three lanes, etc.). A stereo camera may be formed by providing a plurality of optical systems 14.

[0013] The photoelectric conversion unit 15 includes an imaging element 100 configured by stacking an imaging chip that outputs a pixel signal corresponding to light incident from the optical system 14, a signal processing chip that processes the pixel signal, and a memory chip that stores the pixel signal. As will be described in detail later, the imaging element 100 can set imaging conditions (including the case where imaging is not performed) individually for each pixel or for each unit area consisting of multiple pixels (e.g., 16 pixels x 16 pixels).

[0014] In this embodiment, the optical system 14 and the photoelectric conversion unit 15 constitute the camera imaging unit 5, which captures images of objects (moving bodies, obstacles, etc.) around the automobile 10 and white lines (including lines of other colors such as yellow) on the road. The automobile 10 is equipped with a front imaging unit 5 that captures an image of the front of the automobile 10, and a rear imaging unit 5 that captures an image of the rear of the automobile 10. In this description, a white line refers to a line such as a white line drawn on a roadway. The term white line also includes solid and broken lines. Although not shown, a radar may be provided, and surrounding objects may be detected by this radar and the imaging unit 5 (optical system 14, photoelectric conversion unit 15).

[0015] The communication unit 16 performs wireless communication (including optical beacons, radio beacons, and visible light communication) with external devices such as other cars 20 and traffic lights 40. Any communication method may be used.

[0016] The storage unit 17 is configured, for example, with a non-volatile semiconductor memory such as a flash memory, and stores various programs and control parameters for driving the automobile 10 (including automatic driving).

[0017] The sensor 18 includes one or more various sensors such as a vehicle speed sensor, a yaw rate sensor, etc. The vehicle speed sensor detects the vehicle speed V of the automobile 10, and sends a detection signal to the control unit 19, etc. The yaw rate sensor detects the yaw rate of the automobile 10, and sends a detection signal to the control unit 19, etc. The yaw rate is the rate of change of the rotation angle in the turning direction of the vehicle, etc.

[0018] The control unit 19 controls the entire automobile 10, and includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. In this embodiment, the control unit 19 sets and controls the imaging conditions of each unit area of ​​the imaging element 100 of the photoelectric conversion unit 15. When the autonomous driving mode is set by the vehicle operation unit 11, the control unit 19 detects white lines on the road using the imaging unit 5 (optical system 14, photoelectric conversion unit 15), detects moving objects and obstacles around the automobile 10 using the imaging unit 5, and performs autonomous driving to a destination inputted to the navigation system 13 in cooperation with the navigation system 13.

[0019] In this embodiment, the automatic driving mode refers to the operation of the steering wheel, accelerator, brake, turn signal switch, shift lever, etc. being performed automatically under the control of the control unit 19. Manual driving refers to the operation of the steering wheel, accelerator, brake, turn signal switch, shift lever, etc. being performed by the driver, and there are cases where the transmission is an automatic transmission and a manual transmission. The automatic driving mode includes not only fully automatic driving in which driving is performed entirely under the control of the control unit 19, but also semi-automatic driving in which the control unit 19 stops or decelerates the automobile 10 to avoid collisions, etc., based on the output of the imaging unit 5, GPS 12, communication unit 16, sensor 18, etc. even when the user is operating the car operation unit 11. This allows the user to ensure safety while enjoying driving the automobile 10. The semi-automatic driving also includes a case where the control unit 19 controls some of the operations of the steering wheel, accelerator, brake, turn signal switch, shift lever, etc. on behalf of the driver.

[0020] (20 other cars) The other car 20 includes a communication unit 21, a vehicle operation unit 22, a storage unit 23, an imaging unit 24, a control unit 25, etc., and the functions of each unit are the same as those of the automobile 10. Although the other car 20 is omitted in FIG. 1, it has a basic configuration as an automobile. However, some of the other cars 20 may not include the communication unit 21. In addition, the other cars 20 include a mixture of automatic and manual driving cars. Among these, at least the vehicles with an automatic driving mode are capable of communicating with each other via the communication unit 21, and are configured to be able to transmit and receive information regarding whether the vehicle is in automatic or manual driving mode, and image data acquired by the imaging unit 24.

[0021] (Traffic signal generator 30) The traffic signal generating device 30 is a device that controls the signal lights displayed on the display unit 42 of the traffic light 40, and has a traffic light information generating unit 31, a storage unit 32, a communication unit 33, and a control unit 34. The traffic signal generating device 30 can be installed at each of the multiple traffic lights 40 provided at an intersection or the like, but a single traffic signal generating device 30 may control the multiple traffic lights 40.

[0022] The traffic light information generating unit 31 generates traffic signals based on the types and installation positions of multiple traffic lights 40 installed at intersections and the like, and traffic-related instructions from a traffic control center (not shown).

[0023] The storage unit 32 is configured, for example, by a non-volatile semiconductor memory such as a flash memory, and stores various programs and control parameters of the traffic signal generating device 30.

[0024] The communication unit 33 transmits, by wire or wirelessly, the traffic signals generated by the signal information generation unit 31 to one or more traffic lights 40. The communication unit 33 also transmits and receives information to and from a traffic control center.

[0025] The control unit 34 controls the entire traffic signal generating device 30, and includes a CPU, RAM, ROM, etc. The control unit 34 also analyzes traffic conditions based on traffic volume and the like, and can control the signal information generating unit 31 based on the results.

[0026] (Traffic Light 40) The traffic light 40 has a communication unit 41, a display unit 42, an optical system 43, a photoelectric conversion unit 44, a memory unit 45, and a control unit 46. Although only one traffic light 40 is shown in FIG. 1, a plurality of traffic lights 40 are usually provided. For example, in the case of an intersection, as shown in FIG. 2, four traffic lights 40a for automobiles and eight traffic lights 40b for pedestrians are provided. The traffic lights 40 receive traffic signals corresponding to their respective installation positions from the traffic signal generating device 30 and turn on or blink the indicator lights of the display unit 42.

[0027] The communication unit 41 receives, by wire or wirelessly, the traffic signals generated by the traffic signal information generation unit 31. In addition, the communication unit 41 transmits and receives various information, such as vehicle driving information and traffic information, between the automobile 10, other vehicles 20, and other traffic lights 40.

[0028] The display unit 42 has signal lights and displays the signal lights according to the traffic signals received by the communication unit 41. Specifically, the display unit 42 turns on, blinks, or turns off the signal lights to permit or restrict the movement, such as proceeding or stopping, of vehicles traveling on the road and pedestrians crossing the road. Note that the display unit 42 may turn on not only red lights, yellow lights, and blue lights, but also arrow lights indicating that it is OK to go straight, turn left, or turn right at an intersection.

[0029] In this embodiment, the optical system 43 and the photoelectric conversion unit 44 form an imaging unit 50 of the camera. The optical system 43 is composed of a plurality of lenses, and forms a subject image on the photoelectric conversion unit 44. When the imaging unit 50 is provided in a traffic light 40a for automobiles, the optical system 43 is mainly used to obtain images of vehicles. When the imaging unit 50 is provided in a traffic light 40b for pedestrians, the optical system 43 is mainly used to obtain images of pedestrians (including bicycles). The imaging unit 50 is provided near the display unit 42 (signal light).

[0030] The photoelectric conversion unit 44 includes an imaging element 100 configured by stacking an imaging chip that outputs pixel signals corresponding to light incident from the optical system 43, a signal processing chip that processes the pixel signals, and a memory chip that stores the pixel signals. This has the same configuration as the photoelectric conversion unit 15 in the automobile 10 described above. The imaging element 100 can set imaging conditions according to traffic signals for each pixel or for each unit area consisting of multiple pixels (e.g., 16 pixels x 16 pixels).

[0031] Fig. 3 is a diagram illustrating a traffic light 40a for automobiles arranged at an intersection. In Fig. 3, an imaging unit 50 is installed near the lower side of a display unit 42a of the traffic light 40a. The imaging unit 50 has, for example, a wide-angle lens as an optical system 43, and has an angle of view that includes four lanes on both sides of the multiple driving lanes (e.g., two lanes on each side) of the intersection illustrated in Fig. 2.

[0032] The storage unit 45 is configured, for example, by a non-volatile semiconductor memory such as a flash memory, and stores image data acquired by the photoelectric conversion unit 44 and the like.

[0033] The control unit 46 controls the entire traffic light 40, and includes a CPU, RAM, ROM, etc. In this embodiment, the control unit 46 controls the display of the signal lights of the display unit 42 in response to the traffic signal, and also controls imaging using the imaging element 100 of the photoelectric conversion unit 44.

[0034] In addition, if the shooting range of the imaging unit 50 of the traffic light 40a for automobiles includes the shooting range of the imaging unit 50 of the traffic light 40b for pedestrians, the imaging unit 50 (optical system 43, photoelectric conversion unit 44) of the traffic light 40b for pedestrians may be omitted.

[0035] <Explanation of stacked image sensor> The stacked imaging element 100 provided in the imaging section of the above-mentioned automobile 10, the other vehicle 20, and the traffic light 40 will be described. The stacked imaging element 100 is described in WO13 / 164915, which was previously filed by the applicant of the present application and published. FIG. 4 is a cross-sectional view of the stacked imaging element 100. The imaging element 100 includes a back-illuminated imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 such as Cu.

[0036] As shown in the figure, incident light is mainly incident in the Z-axis positive direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface (imaging surface). As shown in the coordinate axes, the left direction on the paper surface perpendicular to the Z-axis is the X-axis positive direction, and the front direction on the paper surface perpendicular to the Z-axis and the X-axis is the Y-axis positive direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 4.

[0037] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 includes a plurality of PDs (photodiodes) 104 that are two-dimensionally disposed and accumulate electric charges according to incident light, and transistors 105 that correspond to the PDs 104.

[0038] A color filter 102 is provided on the incident side of the PD layer 106 where the incident light is incident, via a passivation film 103. There are a plurality of types of color filters 102 that transmit different wavelength regions, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

[0039] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0040] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may include passive and active elements.

[0041] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed, etc., so that the aligned bumps 109 are joined and electrically connected to each other.

[0042] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, etc., so that the aligned bumps 109 are bonded to each other and electrically connected.

[0043] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may be adopted. Also, it is sufficient to provide about one bump 109 for each block described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in a peripheral region other than the pixel region where the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided.

[0044] The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0045] FIG. 5 is a diagram for explaining the pixel array of the imaging chip 113 and the unit area 131. In particular, the imaging chip 113 is shown as observed from the back side (imaging surface) side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In the example of FIG. 5, adjacent 4 pixels x 4 pixels, or 16 pixels, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be around 1000, for example, 32 pixels x 64 pixels, or may be more or less than that.

[0046] 5 includes four so-called Bayer arrays, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R, arranged vertically and horizontally. The green pixels Gb and Gr are pixels having a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel having a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter 102 and receives light in the red wavelength band.

[0047] In this embodiment, a plurality of blocks are defined so that each block includes at least one unit region 131, and each block can control the pixels included in each block with different control parameters. In other words, imaging signals with different imaging conditions can be acquired for a pixel group included in one block and a pixel group included in another block. Examples of the control parameters are frame rate, gain, thinning rate, number of summation rows or number of summation columns for adding pixel signals, charge accumulation time or number of accumulations, number of digitization bits (word length), etc. The imaging element 100 can freely perform thinning not only in the row direction (X-axis direction of the imaging chip 113) but also in the column direction (Y-axis direction of the imaging chip 113). Furthermore, the control parameters may be parameters for image processing after image signals are acquired from pixels.

[0048] Fig. 6 is a diagram for explaining a circuit in a unit area 131. In the example of Fig. 6, one unit area 131 is formed by 9 pixels, 3 pixels x 3 pixels adjacent to each other. As described above, the number of pixels included in the unit area 131 is not limited to this, and may be more or less than this. The two-dimensional position of the unit area 131 is indicated by symbols A to I.

[0049] The reset transistors of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 6, a reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 310 to turn on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 to turn on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. Dedicated reset wiring is also provided for the other pixels D to I to turn on and off their respective reset transistors.

[0050] The transfer transistors of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 6, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. Dedicated transfer wiring is also provided for the other pixels D to I to turn on and off the respective transfer transistors.

[0051] Furthermore, the selection transistors of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 6, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. Dedicated selection wiring is also provided for the other pixels D to I to turn on and off their respective selection transistors.

[0052] The power supply wiring 304 is commonly connected from pixel A to pixel I included in the unit area 131. Similarly, the output wiring 308 is commonly connected from pixel A to pixel I included in the unit area 131. The power supply wiring 304 is commonly connected between the multiple unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0053] By individually turning on and off the reset transistors and transfer transistors of the unit area 131, it is possible to control charge accumulation including the accumulation start time, accumulation end time, and transfer timing independently for pixels A to I included in the unit area 131. In addition, by individually turning on and off the selection transistors of the unit area 131, it is possible to output pixel signals of each of pixels A to I via a common output wiring 308.

[0054] Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order for rows and columns for pixels A to I included in the unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, pixel signals are output in the order of "ABCDEFGHI" in the example of FIG.

[0055] In this way, by configuring the circuit based on the unit area 131, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output pixel signals at different frame rates between the unit areas 131. Also, by resting the unit areas 131 included in a part of the imaging chip 113 while causing the unit areas 131 included in other areas to accumulate charge (imaging), it is possible to cause imaging to be performed only in a predetermined area of ​​the imaging chip 113 and output the pixel signals. Furthermore, it is also possible to switch the area (target area for accumulation control) in which charge accumulation (imaging) is performed between frames, and cause imaging to be performed sequentially in different areas of the imaging chip 113 and output pixel signals.

[0056] Fig. 7 is a block diagram showing a functional configuration of the image sensor 100 corresponding to the circuit shown in Fig. 6. An analog multiplexer 411 sequentially selects nine PDs 104 forming a unit area 131 and outputs the respective pixel signals to output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed in the image sensor chip 113 together with the PDs 104.

[0057] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 that performs CDS and A / D conversion and is formed in the signal processing chip 111. The A / D converted pixel signals are passed to a demultiplexer 413 and stored in pixel memories 414 corresponding to the respective pixels. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 112.

[0058] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that although Fig. 7 shows the connections for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 415 for each unit area 131. For example, one arithmetic circuit 415 may process sequentially while referring to values ​​of the pixel memories 414 corresponding to each unit area 131 in order.

[0059] As described above, output wiring 308 is provided corresponding to each unit area 131. Since image pickup element 100 has imaging chip 113, signal processing chip 111, and memory chip 112 stacked on top of each other, by using bumps 109 for electrical connection between the chips for output wiring 308, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0060] <Explanation of distance measurement> FIG. 8 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 100. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of FIG. 8, 15 focus detection pixel lines 60 are arranged at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 60 output image signals for distance measurement. In the imaging chip 113, normal imaging pixels are provided at pixel positions other than the focus detection pixel lines 60. The imaging pixels output image signals for monitoring moving objects, obstacles, etc.

[0061] Fig. 9 is an enlarged view of a region including a portion of one of the focus detection pixel lines 60. Fig. 9 illustrates red pixels R, green pixels G (Gb, Gr), and blue pixels B, as well as focus detection pixels P1 and P2. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0062] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B indicate the light receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system. That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and the light that passes through these mask openings reaches the light receiving section of the imaging pixel.

[0063] The shape of the light receiving regions (mask openings) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0064] The semicircular regions illustrated for the focus detection pixel P1 and the focus detection pixel P2 indicate the light receiving regions of the focus detection pixels. That is, the focus detection pixel P1 has a semicircular mask opening on the left side of the pixel position in FIG. 9, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel P1. On the other hand, the focus detection pixel P2 has a semicircular mask opening on the right side of the pixel position in FIG. 9, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel P2. In this way, the focus detection pixel P1 and the focus detection pixel P2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system.

[0065] The positions of the focus detection pixel lines in the imaging chip 113 are not limited to the positions exemplified in Fig. 8. The number of focus detection pixel lines is also not limited to the example in Fig. 8. Furthermore, the shape of the mask openings in the focus detection pixels P1 and P2 is not limited to a semicircle, and may be, for example, a rectangular shape obtained by horizontally dividing the square-shaped light receiving regions (mask openings) in the imaging pixels R, G, and B.

[0066] Furthermore, the focus detection pixel line in the imaging chip 113 may be an arrangement of focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 113. An imaging element in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Fig. 9 is well known, and detailed illustration and description of these pixels will be omitted.

[0067] 9, a so-called 1PD structure has been described in which each of the focus detection pixels P1 and P2 receives one of a pair of light beams for focus detection. Alternatively, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2007-282107, a so-called 2PD structure may be used in which each of the focus detection pixels receives both of a pair of light beams for focus detection. By using such a 2PD structure, it becomes possible to read image data from the focus detection pixels, and the focus detection pixels do not become defective pixels.

[0068] In this embodiment, the focus adjustment state (defocus amount) of the imaging optical system is calculated by detecting the amount of image shift (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system based on the distance measurement image signals output from the focus detection pixel P1 and the focus detection pixel P2.

[0069] In general, the pair of images approach each other in a so-called front focus state in which the imaging optical system forms a sharp image of an object (e.g., a preceding vehicle) in front of the intended focal plane, and move away from each other in a so-called back focus state in which the imaging optical system forms a sharp image of the object behind the intended focal plane. In a focused state in which a sharp image of the object is formed on the intended focal plane, the pair of images coincide relatively. Therefore, the amount of relative positional deviation between the pair of images corresponds to the distance to the object (depth information).

[0070] The defocus amount calculation based on the phase difference is well known in the field of cameras, and therefore a detailed description will be omitted. Here, since the defocus amount and the distance to the object correspond one-to-one, the distance from the camera to each object can be obtained by calculating the defocus amount for each object. That is, the distance measurement (distance measurement) to the object can be performed at each of a plurality of positions on the shooting screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a formula or a lookup table, and stored in a non-volatile memory or the like.

[0071] <Automobile control> The control executed by the control unit 19 of the automobile 10 will be described below with reference to the flowchart of Fig. 10. Note that this flowchart is started by starting the automobile 10, for example, by starting the engine or the driving system. A program for executing the processing according to the flowchart of Fig. 10 is stored in a storage medium such as a ROM in the control unit 19, or in the storage unit 17 of the automobile 10.

[0072] In step S1, the control unit 19 starts imaging by the imaging element 100 of the photoelectric conversion unit 15. As described above, the imaging unit 5 (optical system 14, photoelectric conversion unit 15) of the automobile 10 obtains images of the front and rear of the automobile 10, respectively.

[0073] In step S2, the control unit 19 communicates with other vehicles 20 (vehicles traveling in the same lane as the automobile 10, as well as vehicles traveling in a different driving lane in the same traveling direction as the automobile 10) via the communication unit 16. In this embodiment, as shown in FIG. 11(a), it is assumed that the preceding vehicle 73A traveling in the same lane as the automobile 10 is an autonomous vehicle, and the preceding vehicle 72A traveling in the adjacent driving lane (same traveling direction) is a manually driven vehicle. It is also assumed that the following vehicle 73B traveling in the same lane as the automobile 10 is an autonomous vehicle, and the following vehicle 72B traveling in the adjacent driving lane (same traveling direction) is a manually driven vehicle.

[0074] Whether a vehicle around the automobile 10 is an autonomous vehicle or a manually driven vehicle is determined based on the communication results of known vehicle-to-vehicle communication by the communication unit 16. In addition, if an identification mark or the like is displayed on the other vehicle 20, the determination may be based on the imaging results by the imaging unit 5 (optical system 14, photoelectric conversion unit 15). The identification mark may be a predetermined mark or code displayed on the body of the vehicle, or identification information may be displayed on a display unit (not shown) provided on the roof of the vehicle or the like. Note that, for other vehicles 20 that cannot be determined to be autonomous or manually driven based on the communication results (communication not possible) or imaging results, the control unit 19 presumes that the other vehicle is a manually driven vehicle.

[0075] In step S3, the control unit 19 sets the imaging conditions individually for each of the unit regions 131 (FIG. 5) of the imaging element 100 of the photoelectric conversion unit 15. FIG. 11(b) and FIG. 11(c) are diagrams showing schematic images of a subject (object) formed on the imaging element 100 which images the front and rear of the automobile 10, respectively. In reality, an inverted reverse image is formed, but for ease of understanding, an upright normal image is shown. The white line 80a indicates the dividing line on the left side of the road in the direction of travel, the white line 80b indicates the boundary line of the travel lane (traffic lane), and the white line 80c indicates the dividing line on the right side of the road.

[0076] As described above, since the vehicles 72A and 72B traveling in the lane next to the automobile 10 are manually driven vehicles, the control unit 19 sets the area including the vehicle 72A in Fig. 11(b) as the attention area 71A. Then, the frame rate of the unit area of ​​the image sensor 100 corresponding to the attention area 71A is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%, lower than that of the normal area.

[0077] 11(c), the control unit 19 sets the area including the vehicle 72B as the attention area 71B.Then, the frame rate of the unit area of ​​the image sensor 100 corresponding to the attention area 71B is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%, which is lower than that of the normal area. The control unit 19 further changes the setting of this thinning rate in accordance with the moving speed of the automobile 10 or the relative moving speed between the automobile 10 and other vehicles 20. For example, the thinning rate is changed to a lower value as the relative moving speed becomes faster.

[0078] FIG. 11(b) is an example in which all areas except the attention area 71A are normal areas, but the area surrounding the vehicle 73A, which is an autonomous vehicle, may be a quasi-attention area 74A, and the areas except the attention area 71A and the quasi-attention area 74A may be normal areas. FIG. 11(c) is an example in which all areas except the attention area 71B are normal areas, but the area surrounding the vehicle 73B, which is an autonomous vehicle, may be a quasi-attention area 74B, and the areas except the attention area 71B and the quasi-attention area 74B may be normal areas. The control unit 19 may set the imaging conditions of the quasi-attention areas 74A and 74B differently for fully autonomous driving and semi-autonomous driving. In this case, the control unit 19 may set the frame rate of the image sensor 100 in the semi-autonomous driving to be higher than the frame rate of the image sensor 100 in the fully autonomous driving. The control unit 19 may set the imaging area as a normal area in the fully autonomous driving.

[0079] The control unit 19 sets the frame rate of the unit areas of the imaging element 100 corresponding to the quasi-attention areas 74A and 74B, which respectively capture images of the front and rear of the automobile 10, to a lower frame rate than the frame rate of the normal area (e.g., 30 fps), and sets the thinning rate to approximately 30 to 60%.

[0080] Furthermore, the control unit 19 may set an area including a white line on the road as an attention area in addition to the attention areas 71A and 71B. Then, for the image sensor 100 capturing images of the front and rear of the automobile 10, the frame rate of the unit area of ​​the image sensor 100 corresponding to the attention area is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%. In this way, by varying the imaging conditions for each unit area of ​​the image sensor 100 when capturing images of a manually driven vehicle and an autonomously driven vehicle, the image sensor 100 can be used efficiently, and power consumption and heat generation can be reduced.

[0081] 10, the control unit 19 judges whether or not communication with the traffic light 40 is possible via the communication unit 16, i.e., whether or not the control unit 19 is approaching the traffic light 40 (intersection). If the control unit 19 is unable to communicate with the traffic light 40 (outside the communication area), it makes a negative judgment at step S4 and proceeds to step S7. On the other hand, if the control unit 19 is able to communicate with the traffic light 40 (within the communication area), it makes a positive judgment at step S4 and proceeds to step S5.

[0082] In step S5, the control unit 19 receives information based on the image acquired by the photoelectric conversion unit 44 of the traffic light 40 (traffic light 40a for automobiles or traffic light 40b for pedestrians). For example, when the automobile 10 turns left (when turning right in areas such as the United States where vehicles drive on the right side of the road), the control unit 19 receives information about a person from the traffic light 40b for pedestrians. In this case, the control unit 46 of the traffic light 40 determines the presence or absence of a pedestrian based on the image acquired by the photoelectric conversion unit 44, and the control unit 19 receives the information about the pedestrian determined by the control unit 46. The control unit 19 of the automobile 10 may receive image data acquired by the photoelectric conversion unit 44 of the traffic light 40, and determine the presence or absence of a pedestrian based on the received image data.

[0083] When the automobile 10 makes a right turn (when turning left in an area where vehicles drive on the right side of the road), the control unit 19 receives information from the automobile traffic light 40a, such as whether a vehicle moving straight in the oncoming lane is an autonomous vehicle or a manually driven vehicle. Furthermore, the control unit 19 receives information about the switching of traffic signals (for example, signal switching information such as a change from a green light to a red light in a few seconds) from the automobile traffic light 40a.

[0084] In step S6, the control unit 19 sets the imaging conditions of the image sensor 100 of the photoelectric conversion unit 15 based on the information obtained in step S5. For example, when the automobile 10 turns left at an intersection, the control unit 19 makes the frame rate of the unit area of ​​the image sensor 100 corresponding to the left side of the shooting screen higher than the frame rate of the unit area corresponding to the right side of the shooting screen, and changes the frame rate according to the speed of the automobile 10 or the moving speed of pedestrians (for example, 4 km / h).

[0085] For example, when the automobile 10 moving at 50 km / h decelerates to about 10 km / h to make a left turn, the frame rate of the unit area corresponding to the left side of the shooting screen is lowered compared to that before deceleration. Also, when the automobile 10 turns left, the control unit 19 sets the imaging conditions of the image sensor 100 depending on whether a pedestrian crossing the sidewalk is approaching the automobile 10 or moving away from the automobile 10. That is, the control unit 19 increases the frame rate and decreases the thinning rate of the unit area of ​​the image sensor 100 corresponding to a person approaching the automobile 10, and decreases the frame rate and increases the thinning rate of the unit area of ​​the image sensor 100 corresponding to a pedestrian moving away from the automobile 10 (especially a person who has already crossed the crosswalk where the automobile 10 is scheduled to pass).

[0086] Furthermore, when the automobile 10 turns right at an intersection, if another vehicle 20 coming straight from the oncoming lane is a manually driven vehicle, the control unit 19 makes the frame rate of the unit area corresponding to the right side of the photographed screen relatively higher than the frame rate of the unit area corresponding to the left side of the photographed screen, thereby lowering the thinning rate. Furthermore, if a pedestrian crossing the crosswalk ahead of the right turn is approaching the automobile 10, the control unit 19 further increases the frame rate of the unit area of ​​the image sensor 100 corresponding to the approaching pedestrian, thereby further lowering the thinning rate.

[0087] When the automobile 10 is turning right or left, the control unit 19 may predict the imaging area that requires attention (predict the imaging area in which pedestrians, etc. may be captured) and change the imaging conditions depending on the operation state of the turn signal switch and the amount of steering wheel operation.

[0088] The control unit 19 also receives information about the change in traffic light from the traffic light 40a for automobiles, and changes the imaging conditions of the image sensor 100 based on the information. For example, the control unit 19 receives information about the change in traffic light, that is, the green light will change to red light in a few seconds, from the traffic light 40a for automobiles, and when the automobile 10 decelerates, the control unit 19 controls the frame rate of the image sensor 100 for imaging the front to be lower than that before deceleration, or the thinning rate to be higher. On the other hand, the control unit 19 controls the imaging conditions of the image sensor 100 for imaging the rear to be maintained as they are. In addition, since it is expected that a following vehicle will approach the automobile 10 when the automobile 10 decelerates, the frame rate of the image sensor 100 capturing images of the rear may be controlled to be higher than before the deceleration, or the thinning rate may be controlled to be lower. When the speed of the automobile 10 is changed, the control unit 19 may predict the change in speed according to the amount of operation of the brake or accelerator (amount of depression of the pedal) and change the settings of the imaging conditions.

[0089] In step S7, the control unit 19 determines whether the engine (or the driving system) is on. If the engine (or the driving system) is on, the control unit 19 makes a positive determination in step S7 and repeats the processing in step S2 and thereafter. If the engine (or the driving system) is off, the control unit 19 makes a negative determination in step S7 and ends the processing according to this flowchart.

[0090] <Traffic signal control> Next, the control executed by the control unit 46 of the traffic light 40 will be described with reference to the flowchart of Fig. 12. A program for executing the processing according to the flowchart of Fig. 12 is stored in a storage medium such as a ROM in the control unit 46 or in the storage unit 45.

[0091] In step S10, the control unit 46 determines whether or not a traffic signal has been received from the traffic signal generating device 30. When the control unit 46 receives a traffic signal from the traffic signal generating device 30, it makes an affirmative decision in step S10 and proceeds to step S11. When the control unit 46 has not received a traffic signal from the traffic signal generating device 30, it makes a negative decision in step S10 and waits for reception.

[0092] In step S11, the control unit 46 performs display control of the display unit 42. For example, the control unit 46 performs control to switch the signal light display of the display unit 42 from red to green in response to the traffic signal received from the traffic signal generating device 30.

[0093] In step S12, the control unit 46 communicates with one or more vehicles or other traffic lights 40. The vehicles with which the control unit 46 communicates may include the automobile 10 and other cars 20 equipped with a communication unit 21. The communication target vehicle may be a vehicle within a predetermined range from the intersection or traffic light 40, or may be a vehicle with which communication can be made via an information providing system (not shown) installed on the road.

[0094] The control unit 46 acquires information from the communication target vehicle about the travel mode of the vehicle or the vehicles around the vehicle, such as whether the vehicle or the vehicles around the vehicle are autonomous or manually driven vehicles. Furthermore, the control unit 46 acquires information about the driving conditions of the vehicle or the vehicles around the vehicle from the communication target vehicle. For example, the control unit 19 or the control unit 25 of the communication target vehicle predicts a course change (right turn or left turn) at an intersection from the state of a turn signal switch for activating a blinker (directional indicator). The control unit 46 acquires right turn prediction information or left turn prediction information predicted by the communication target vehicle from the communication target vehicle as information about the driving conditions.

[0095] The control unit 46 may determine whether a vehicle is an autonomous vehicle or a manually driven vehicle based on the results of communication with the communication target vehicle. In addition, if an identification mark or the like is displayed on the vehicle, the control unit 46 may determine based on the imaging results by the imaging unit (optical system 43, photoelectric conversion unit 44). Furthermore, regarding a change of course (right or left turn) of the vehicle, the control unit 46 may determine from the operating state of the turn signal of the vehicle based on the imaging results by the imaging unit (optical system 43, photoelectric conversion unit 44), or may determine based on whether the vehicle is in a left turn lane or a right turn lane.

[0096] The control unit 46 also communicates with other traffic lights 40, including the traffic light 40a for automobiles or the traffic light 40b for pedestrians, to obtain information on traffic conditions at intersections, etc. The control unit 46 further communicates with the traffic signal generating device 30 involved in generating the traffic signals for each traffic light, as necessary, to obtain information on the display status of the traffic signal.

[0097] In step S13, the control unit 46 sets imaging conditions for the imaging element 100 of the photoelectric conversion unit 44 based on the display status of the signal lamp on the display unit 42 and the information acquired in step S12. Details of setting the imaging conditions for the imaging element 100 of the photoelectric conversion unit 44 will be described later.

[0098] In step S14, the control unit 46 causes the imaging unit (optical system 43, photoelectric conversion unit 44) to capture an image under the imaging conditions set in step S13.

[0099] In step S15, the control unit 46 transmits the image data acquired in step S14 to the automobile 10, another vehicle 20, another traffic light 40, or the like, via the communication unit 41. The control unit 46 also transmits various pieces of information extracted by performing image processing or image analysis based on the image data, such as data on the direction and speed of the vehicle, and information on the driving lane of the vehicle estimated from identification of the operating state of the blinkers, etc.

[0100] Furthermore, the control unit 46 may recognize the estimated driving lane of the communication target vehicle (the automobile 10 or another vehicle 20) and obstacles (including vehicles and pedestrians) based on the analyzed information, generate a message based on the recognition result, and transmit the message from the communication unit 41 to the communication target vehicle. Examples of messages include "A motorcycle is approaching from behind," "A pedestrian is crossing," and "An oncoming vehicle is proceeding straight ahead." The control unit 46 repeatedly executes the processes from step S10 to step S15.

[0101] <Setting imaging conditions for traffic signals> FIG. 13 is a diagram showing an example of control of imaging conditions in the imaging element 100 of the imaging unit 50-1 that is installed integrally with or in the vicinity of a traffic light 40a for automobiles when the traffic light 40a for automobiles is displaying a green light.

[0102] 13, a traffic light 40a for automobiles for driving lane A, an imaging unit 50-1, and a traffic signal generating device 30-1 are shown. Other traffic lights and the like at the intersection are not shown. A control unit 46 of the traffic light 40a for automobiles sets the range of the driving lane in which the vehicle is moving due to a green light within the imaging area 70 of the imaging unit 50-1 as an attention area 71 (shaded area).

[0103] The control unit 46 controls the unit area of ​​the image sensor 100 corresponding to the attention area 71 by making the frame rate higher or lowering the thinning rate than those of unit areas other than the attention area 71. Furthermore, when a vehicle 72 that is a non-automated manually driven vehicle is present in the attention area 71 in FIG. 13, the control unit 46 sets the area surrounding the vehicle 72 as a special attention area 75. Then, the control unit 46 makes the frame rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of ​​the image sensor 100 corresponding to the special attention area 75 to be lower than the thinning rate of the unit area corresponding to the attention area 71.

[0104] In addition, the control unit 46 can also capture high-resolution images of areas within the attention area 71 that include vehicles 76 and 77 that have stopped temporarily to turn right or left by setting a thinning rate lower than that of the unit area corresponding to the attention area 71.

[0105] FIG. 14 is a diagram showing an example of control of imaging conditions in the imaging element 100 of the imaging unit 50-1 that is installed integrally with or in the vicinity of a traffic light 40a for automobiles when the traffic light 40a for automobiles is displaying a red light.

[0106] 14, a traffic light 40a for automobiles for driving lane A, an imaging unit 50-1, and a traffic signal generating device 30-1 are shown. Other traffic lights and the like at the intersection are not shown. The control unit 46 sets a crosswalk where a pedestrian 90 can enter and its vicinity as an attention area 71 within the imaging area 70 of the imaging unit 50-1 (shaded area).

[0107] The control unit 46 controls the unit region of the image sensor 100 corresponding to the attention region 71 by making the frame rate higher or lowering the thinning rate than those of unit regions other than the attention region 71. Furthermore, when the control unit 46 recognizes a pedestrian 90, it sets an area of ​​a predetermined range including the pedestrian 90 as a special attention region 75. Then, it makes the frame rate of the unit region of the image sensor 100 corresponding to the special attention region 75 higher than that of the unit region corresponding to the attention region 71. The control unit 46 also controls the thinning rate of the unit region of the image sensor 100 corresponding to the special attention region 75 to be lower than the thinning rate of the unit region corresponding to the attention region 71.

[0108] Fig. 15 is a diagram showing an example of control of imaging conditions of an imaging unit 50-2 installed integrally with or adjacent to a pedestrian traffic light 40b. In Fig. 15, a pedestrian traffic light 40b, an imaging unit 50-2, and a traffic signal generating device 30-2 are shown. Other traffic lights and the like at the intersection are not shown. A control unit 46 of the pedestrian traffic light 40b sets a crosswalk where a pedestrian 90 can enter and its vicinity as an attention area 71 within the imaging area 70 of the imaging unit 50-2 (shaded area).

[0109] The control unit 46 controls the unit region of the image sensor 100 corresponding to the attention region 71 by making the frame rate higher or lowering the thinning rate than those of unit regions other than the attention region 71. Furthermore, when the control unit 46 recognizes a pedestrian 90, it sets an area of ​​a predetermined range including the pedestrian 90 as a special attention region 75. Then, it makes the frame rate of the unit region of the image sensor 100 corresponding to the special attention region 75 higher than that of the unit region corresponding to the attention region 71. The control unit 46 also controls the thinning rate of the unit region of the image sensor 100 corresponding to the special attention region 75 to be lower than the thinning rate of the unit region corresponding to the attention region 71.

[0110] FIG. 16(a) is a diagram illustrating a scene captured by an imaging unit 50-2 installed in a pedestrian traffic light 40b. FIG. 16(b) is a diagram illustrating the setting of imaging conditions based on a subject recognition result using image data acquired by the imaging unit 50-2. In FIG. 16(a), an imaging unit 50-2 having the above-mentioned imaging element 100 is installed in a pedestrian traffic light 40b. The imaging element 100 according to this embodiment can measure movement not only in the up-down and left-right directions but also in the depth direction, so that it is possible to measure the speed Vo at which a pedestrian 90, who is a subject, moves.

[0111] 16(b), the control unit 46 sets an area including a pedestrian 90 crossing the crosswalk within the imaging area 70 of the imaging unit 50-2 as an attention area 71. Then, the control unit 46 makes the imaging conditions for the unit area of ​​the imaging element 100 corresponding to the attention area 71 different from those for the unit areas other than the attention area 71. At this time, the control unit 46 makes the imaging conditions different depending on the speed Vo of the pedestrian 90.

[0112] For example, when the absolute value of the speed Vo of the pedestrian 90 is large, the control unit 46 controls the unit area of ​​the image sensor 100 corresponding to the attention area 71 including the pedestrian 90 by increasing the frame rate or decreasing the thinning rate compared to unit areas other than the attention area 71.

[0113] Furthermore, the control unit 46 may change the imaging conditions for the attention area 71 based on the positional relationship between nearby vehicles or buildings and the pedestrian 90. For example, when a vehicle operating a blinker to turn right or left at an intersection is present in the imaging area 70, there is a possibility that the vehicle will enter the crosswalk, so the control unit 46 sets an area of ​​the attention area 71 that is closer to the vehicle as a special attention area. Then, the control unit 46 sets the frame rate of the unit area of ​​the imaging element 100 corresponding to the special attention area to be higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of ​​the imaging element 100 corresponding to the special attention area to be lower than that of the unit area corresponding to the attention area 71.

[0114] Furthermore, the control unit 46 may also set different imaging conditions for a plurality of pixels or regions 78P indicated by hatching and a plurality of pixels or regions 78S not indicated by hatching in the attention region 71. In the example of Fig. 16(b), different imaging conditions are set for pixels or regions adjacent to each other vertically and horizontally in a checkerboard pattern, but the present invention is not limited to this.

[0115] Furthermore, when control unit 46 recognizes objects such as other pedestrians or bicycles within the range of imaging region 70, control unit 46 may add regions including each of these objects to attention region 71. In the plurality of attention regions 71, imaging conditions may be made different between a plurality of pixels or regions 78P indicated by hatching and a plurality of pixels or regions 78S not indicated by hatching.

[0116] Fig. 17 is a diagram illustrating an example of a scene captured by an imaging unit 50-1 installed in a traffic light 40a for automobiles at an intersection. In Fig. 17, the traffic light 40a for automobiles is equipped with a display unit 42a having indicator lights indicating whether to go straight, turn left, or turn right. An example of setting imaging conditions based on a subject recognition result using image data captured by the imaging unit 50-1 will be described.

[0117] 17 shows a state in which the display unit 42a of the traffic light 40a lights up the indicator lights indicating that going straight and turning left are permitted, and the vehicle is waiting to turn right. The control unit 46 of the traffic light 40a for automobiles controls the frame rate of the unit area of ​​the image sensor 100 corresponding to the driving lane A, through which vehicles going straight or turning left pass, and the driving lane B, through which vehicles going straight pass, by making the frame rate higher than the frame rate of the unit area corresponding to the other driving lane C, or by lowering the thinning rate. In other words, the control unit 46 controls the frame rate of the unit area corresponding to the driving lane C by lowering it or by setting the thinning rate higher. Note that lowering the frame rate also includes setting the unit area so that no imaging is performed.

[0118] In controlling the imaging conditions of the image sensor 100 in the automobile 10 or traffic light 40 (traffic light 40a for automobiles, traffic light 40b for pedestrians) according to the embodiment described above, the imaging conditions may be changed almost simultaneously with the timing of switching the display on the display unit 42, or the imaging conditions may be changed at a fixed time interval from the timing of switching the display. Alternatively, for a fixed time immediately after the display is switched, the imaging conditions may be changed by setting both the attention area set before the display is switched and the attention area to be set after the display is switched as the attention area.

[0119] According to the above-described embodiment, the following advantageous effects can be obtained. (1) The imaging device of the automobile 10 (or traffic light 40) includes an imaging unit 5 (or imaging unit 50) having an imaging element 100 capable of setting imaging conditions for a plurality of areas, and a control unit 19 (or control unit 41) that sets imaging conditions for a plurality of unit areas based on the driving manner of other surrounding vehicles 20. This makes it possible to set imaging conditions suitable for the driving manner of the surrounding vehicles 20 for the imaging element 100.

[0120] (2) The control unit 19 (or the control unit 41) sets different imaging conditions for each area in which other vehicles 20 with different driving methods are imaged, and therefore different imaging conditions can be set for the imaging element 100 between areas in which vehicles 20 with different driving methods are imaged.

[0121] (3) The driving methods of the other vehicles 20 are an automatic driving method and a manual driving method, and the control unit 19 (or control unit 41) sets different imaging conditions for the area for imaging the vehicle 20 using the automatic driving method and the area for imaging the vehicle 20 using the manual driving method, so that different imaging conditions can be set for the imaging element 100 between the area for imaging the vehicle 20 using the automatic driving method and the area for imaging the vehicle 20 using the manual driving method.

[0122] (4) The control unit 19 (or the control unit 41) sets the frame rate of the area for imaging the manually driven vehicle 20 higher than the frame rate of the area for imaging the automatically driven vehicle 20. This makes it possible to increase the attention given to the manually driven vehicle 20, since the manually driven vehicle 20 is imaged more frequently than the automatically driven vehicle 20. In other words, it becomes possible to quickly and accurately obtain information about the unpredictable behavior of the manually driven vehicle 20.

[0123] (5) The control unit 19 (or the control unit 41) sets the pixel thinning rate of the area capturing an image of the manually driven vehicle 20 to be lower than the pixel thinning rate of the area capturing an image of the automatically driven vehicle 20. This makes it possible to obtain a larger amount of information about the manually driven vehicle 20 than about the automatically driven vehicle 20. In other words, it becomes possible to obtain more accurate information about the unpredictable behavior of the manually driven vehicle 20.

[0124] (6) The vehicle is provided with a control unit 19 (or control unit 41) that acquires information regarding the movement methods of other surrounding vehicles 20. Therefore, even if the surrounding vehicles 20 change, for example, the imaging conditions can be set for the imaging element 100 based on the latest acquired information.

[0125] (7) The control unit 19 (or the control unit 41) is configured to obtain information by communicating with other vehicles 20, and therefore can appropriately set the imaging conditions of the imaging element 100 based on new information obtained through communication.

[0126] (8) The control unit 19 (or the control unit 41) acquires information by capturing images of other vehicles 20 using the imaging unit 5 (or the imaging unit 50), and therefore can appropriately set the imaging conditions of the imaging element 100 based on new information even in a situation where communication is not possible.

[0127] (9) The control unit 19 acquires information from a traffic light 40 different from that of the other vehicles 20. As a result, even in a situation where communication with the other vehicles 20 is not possible, the imaging conditions of the imaging element 100 can be appropriately set based on the new information.

[0128] (10) The control unit 19 (or the control unit 41) acquires information indicating whether the other vehicle 20 is an autonomous vehicle or a manually driven vehicle, and sets different imaging conditions for an area for imaging an autonomous vehicle and an area for imaging a manually driven vehicle. This makes it possible to appropriately set imaging conditions for each unit area on the imaging surface of the image sensor 100 so as to, for example, increase the attention given to manually driven vehicles compared to autonomous vehicles. In particular, by increasing the frame rate and decreasing the pixel thinning rate for manually driven vehicles, it is possible to quickly and accurately acquire information about the unpredictable behavior of manually driven vehicles.

[0129] (11) Since the automobile 10 is equipped with the imaging devices described above in (1) to (10), the imaging conditions for the imaging devices can be appropriately set in accordance with the movement patterns of the other vehicles 20 around the automobile 10.

[0130] (12) The control unit 19 (or control unit 41) of the automobile 10 changes the imaging conditions of the image sensor 100 in response to the operation of the steering wheel, turn signal switch, accelerator, brake, etc. of the automobile 10. This makes it possible to appropriately set imaging conditions for each unit area on the imaging surface of the image sensor 100 in response to changes in course, speed, etc. of the automobile 10.

[0131] (13) The traffic light 40 includes an imaging unit 50 having an imaging element 100 that can set imaging conditions for multiple areas independently, and a control unit 46 that sets the imaging conditions for the multiple areas based on information related to the movement of the automobile 10 and other vehicles 20. This makes it possible to set imaging conditions for the imaging element 100 according to the traffic conditions of the automobile 10 and other vehicles 20 moving through an intersection or the like.

[0132] (14) The information regarding the movement of the automobile 10 and other cars 20 includes a signal that permits the automobile 10 and other cars 20 to move and a signal that does not permit the automobile 10 and other cars 20 to move, and the control unit 46 sets the imaging conditions based on each signal. This makes it possible to appropriately set the imaging conditions of the imaging element 100 for the cases where the automobile 10 and other cars 20 are moving and where the automobile 10 and other cars 20 are not moving.

[0133] (15) The signals permitting the movement of the automobile 10 and the other vehicle 20 include a signal permitting going straight, a signal permitting a left turn, and a signal permitting a right turn, and the control unit 46 sets the imaging conditions based on each of these signals. This makes it possible to appropriately set the imaging conditions of the imaging element 100 when the automobile 10 and the other vehicle 20 go straight, when they turn left, and when they turn right.

[0134] (16) The control unit 46 changes the imaging conditions in response to switching of information related to the movement of the automobile 10 and other vehicles 20, so that the imaging conditions of the imaging element 100 can be appropriately changed at the timing of the switching of the above-mentioned signal.

[0135] (17) The traffic light 40 is equipped with a control unit 46 that acquires information regarding the movement of the surrounding automobiles 10 and other vehicles 20. Therefore, even if the surrounding other vehicles 20 change, the imaging conditions can be set for the imaging element 100 based on the latest acquired information.

[0136] (18) The control unit 46 obtains information through communication with the automobile 10 and other vehicles 20, and can therefore appropriately set the imaging conditions of the imaging element 100 based on new information obtained through communication.

[0137] (19) The control unit 46 obtains information by capturing images of the automobile 10 and other vehicles 20 using the imaging unit 50. Therefore, even in a situation where communication is not possible, the control unit 46 can appropriately set the imaging conditions of the imaging element 100 based on new information.

[0138] (20) The control unit 46 acquires information indicating whether the automobile 10 and the other automobiles 20 are autonomous or manual vehicles as information related to movement, and sets different imaging conditions for an area for imaging autonomous vehicles and an area for imaging manually driven vehicles. This makes it possible to appropriately set imaging conditions for each unit area on the imaging surface of the imaging element 100 so as to, for example, increase the attention given to manually driven vehicles compared to autonomous vehicles. In particular, by increasing the frame rate and decreasing the pixel thinning rate for manually driven vehicles, it is possible to quickly and accurately acquire information about the unpredictable behavior of manually driven vehicles.

[0139] (21) The control unit 46 acquires information indicating lane changes of the automobile 10 and other vehicles 20, and changes the area for setting the imaging conditions based on the lane changes of the automobile 10 and other vehicles 20. This makes it possible to appropriately change the imaging conditions of the image sensor 100 at the timing of lane changes.

[0140] (22) The automobile 10 is equipped with a communication unit 41 that communicates with other automobiles 10 and other cars 20 that are different from the other automobiles 10 and other cars 20. Therefore, for example, information about other cars 20 around a traffic light 40 can be transmitted to the automobile 10.

[0141] (23) The automobile 10 and the traffic light 40 measure the speed at which the object to be imaged moves, and change the imaging conditions of the image sensor 100 according to the magnitude and direction of the speed. Therefore, the imaging conditions can be appropriately set for each area on the imaging surface of the image sensor 100 according to the movement state of the object.

[0142] (24) The automobile 10 obtains information such as how many seconds it will take for the traffic light to change and reflects this information in the driving conditions of the automobile 10. This allows for smooth driving that takes traffic light changes into consideration in advance.

[0143] (25) The automobile 10 or the traffic light 40 sets the imaging conditions of the image sensor 100 so that the area of ​​interest includes both the area of ​​interest set in the immediately preceding traffic light and the area of ​​interest to be set in the new traffic light for a certain period of time immediately after the traffic light changes. This allows the imaging conditions of the image sensor 100 to be set appropriately for the transitional state of the traffic light change.

[0144] (26) The imaging system 1 including the automobile 10 and the traffic light 40 can realize a more organized traffic system based on accurate and efficient information acquisition through imaging and communication.

[0145] In the above-described embodiment, the imaging unit 5 and the imaging unit 50 are controlled by the control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40, respectively. However, part of the control of the imaging unit 5 and the imaging unit 50 may be performed by a control circuit (CPU, etc.) within the imaging unit.

[0146] Furthermore, a part of the processing performed by the control unit 46 of the traffic light 40 may be performed by the control unit 34 of the traffic light information generating device 30. The imaging unit 50 such as a camera does not necessarily have to be attached to the traffic light 40, and may be installed according to the traffic signal or traffic conditions at an intersection or the like.

[0147] Furthermore, in the above embodiment, the display unit and audio playback unit of the navigation system 13 are used to display the message, but a separate display and playback device may be used. Furthermore, a display and playback device consisting of a HUD (Head Up Display) that projects information on the windshield of the automobile 10 and a speaker that plays back audio information may be used.

[0148] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment.

[0149] (Variation 1) The control unit 46 of the traffic light 40 may obtain information regarding the proportion of autonomous vehicles among the vehicles on the road or at the intersection through imaging by the imaging unit 50 or communication by the communication unit 41, and change the imaging conditions according to the proportion. For example, the control unit 46 controls the thinning rate to be higher during a time period when the proportion of autonomous vehicles is high than when the proportion of autonomous vehicles is low. This makes it possible to save power consumption and perform imaging more efficiently.

[0150] (Variation 2) The control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40 may identify signs such as a novice driver sign or an elderly driver sign by imaging, in addition to identifying whether the vehicle is an autonomous vehicle or a manually driven vehicle. Different imaging conditions are set for the unit areas of the image sensor 100 corresponding to the novice driver vehicle and the elderly driver vehicle. For example, a vehicle displaying a novice driver sign may be imaged at a frame rate that is higher than the frame rate for the unit areas of the image sensor 100 corresponding to the manually driven vehicle. This allows the imaging conditions to be appropriately set for each unit area on the imaging surface of the image sensor 100 according to the object.

[0151] (Variation 3) In the above description, distance measurement and detection of surrounding moving objects and obstacles are performed by capturing images using the image sensor 100, but a radar (not shown) may be used in combination. This makes it possible to obtain more reliable traffic information by taking advantage of the characteristics of the image sensor 100 and the radar.

[0152] Although various embodiments and modifications have been described above, the present invention is not limited to these. Aspects in which the configurations shown in the embodiments and modifications are used in combination are also included within the scope of the present invention. Other aspects that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0153] 1. Imaging system 5, 50 (50-1, 50-2)…imaging section 10. Automobiles 19, 46…Control section 20…Other cars 30…Traffic signal generation device 40…Traffic light 40a…Traffic signals for automobiles 40b…Pedestrian traffic light 42...Display section 70…Imaging area 71, 71A, 71B…Area of ​​interest 72, 72A, 72B, 73A, 73B, 76, 77…Vehicles 74A, 74B...quasi-attention area 75…Special attention area 90…Pedestrian 100...Image sensor 113...Imaging chip

Claims

[Claim 1] A vehicle equipped with a driving assistance device, The driving assistance device includes: An imaging unit that images the outside of the vehicle; A distance measuring unit for measuring a distance to an object present outside the vehicle; A positioning unit that measures a current position of the vehicle; a control unit that generates data for performing driving assistance for the vehicle based on information from the imaging unit, the distance measuring unit, and the positioning unit; a receiving unit that receives information transmitted from another vehicle captured by the imaging unit; having The control unit determines whether the other vehicle is a vehicle traveling in an autonomous driving mode based on the information from the receiving unit, and changes a frame rate and a thinning rate, which are imaging conditions of the imaging unit, according to a result of the determination. vehicle.

Images