AUTONOMOUS DRIVING CONTROL DEVICE AND PROGRAM PRODUCT

Abstract

Autonomous driving control device (1) that can be installed in a vehicle comprising: at least one wide-angle camera (32w) and one narrow-angle camera (32n) designed to capture at least one wide-angle image and one narrow-angle image of at least one directed wide-angle area and one directed narrow-angle area of ​​the vehicle, which are at least partially different from each other;and a vehicle control unit (24, 26, 28) designed to control a driving state of the vehicle, wherein the wide-angle camera has a predetermined wide-angle field of view, wherein the narrow-angle camera has a predetermined narrow-angle field of view that is narrower than the wide-angle field of view, wherein the directed narrow-angle area of ​​the narrow-angle camera is further away from the vehicle than the directed wide-angle area of ​​the wide-angle camera, wherein the autonomous driving control device comprises: a drive control unit (6) designed to determine a value of at least one controlled variable for autonomous driving of the vehicle based on the wide-angle and narrow-angle images and an autonomous driving condition; and to output the value of the at least one controlled variable to the vehicle control unit in order to cause the vehicle control unit to perform a task for autonomous driving of the vehicle;and a camera monitoring unit (8) designed to determine whether a camera from the wide-angle camera and the narrow-angle camera is malfunctioning;and if it is determined that the wide-angle camera is malfunctioning, to limit the autonomous driving condition on the basis of the directed wide-angle area corresponding to the malfunctioning wide-angle camera, wherein the drive control unit is designed to determine the value of the at least one controlled variable according to the limited autonomous driving condition in order to cause the vehicle control unit to continuously perform the task of autonomously driving the vehicle according to the determined value of the at least one controlled variable, wherein the camera monitoring unit is designed to perform at least one of the following tasks: a first task to cause a driver of the vehicle to interrupt the task of autonomously driving the vehicle when it is determined that the narrow-angle camera is malfunctioning;and a second task to force the interruption of the autonomous driving task of the vehicle if it is determined that the narrow-angle camera is malfunctioning.

Description

Cross-reference to relevant registrations

[0001] This application is based on and claims priority from Japanese Patent Application 2018-104690 filed on May 31, 2018, the disclosure of which is hereby incorporated by reference. technical field

[0002] The present invention relates to autonomous driving control devices and program products. More specifically, the present invention relates to those devices and programs each capable of autonomously driving a vehicle based on images of a front area in its traveling direction in front of the vehicle. background

[0003] A known autonomous driving control device disclosed in Japanese Patent Laid-Open Publication No. 2017-157067, for example, includes an autonomous driving system installed in a host vehicle for driving the host vehicle autonomously when the host vehicle is set to an autonomous driving mode. In contrast, when the host vehicle is set to a manual driving mode, manual driving of the host vehicle is performed by the driver.

[0004] The published autonomous driving system installed in a host vehicle includes an autonomous driving system composed of an individual computer and capable of autonomous driving of the host vehicle in one stage 3 from previously defined six levels of autonomous driving.

[0005] The published autonomous driving control device also includes a preventive safety system composed of an individual computer and capable of autonomous driving of the host vehicle in one stage 2 of the previously defined levels. The autonomous driving system controls the acceleration, steering and braking of the host vehicle in the stage 3 of the six autonomous drive levels. The predictive safety system controls some of the host vehicle's acceleration, steering and braking in the stage 2 of the six autonomous drive levels.

[0006] In particular, the autonomous driving control device disclosed in the published patent document is designed such that the predictive safety system performs autonomous driving of the host vehicle in the stage 2 which carries out six levels of autonomous driving in response to an autonomous request or query. Thereafter, the autonomous driving control device is designed such that the autonomous driving system performs autonomous driving of the host vehicle in the stage 3 which performs six tiers of autonomous driving when it is determined that a predetermined authorization condition of the tier 3 is satisfied.

[0007] The safety predictive system and the autonomous driving system are each configured to perform an emergency limp-home function mode for sidestepping the host vehicle to a safe place when it is determined that there is a malfunction in the other of the safety predictive system and the autonomous driving system. While the emergency limp home function mode is being performed, the predictive safety system and the autonomous driving system are each configured in response to the driver's intention to drive the host vehicle, autonomous driving of the host vehicle into the stage 1 to switch between the six autonomous drive levels. Note that the stage 1 of the six levels of autonomous driving represents a driver assistance mode for assisting the driver in accelerating, steering and braking the host vehicle. summary

[0008] The autonomous driving control device disclosed in the published patent document is configured as a redundant vehicle control system, that is, a duplicated vehicle control system composed of the individual computer of the autonomous driving system and the individual computer of the predictive safety system. This redundant vehicle control system of the autonomous driving control device enables autonomous driving of the host vehicle to be performed continuously even when one of the autonomous driving system and the predictive safety system malfunctions. In addition, the published autonomous driving control device resulting from the stage autonomous driving system 3 and the predictive safety system of the stage 2 exists, at a lower cost than the autonomous driving control devices, the respective duplicated level autonomous systems 3 exhibit.

[0009] A malfunction in one of the safety predictive system and the autonomous driving system of the published autonomous driving control device causes the other of the safety predictive system and the autonomous driving system to perform the emergency limp-home function mode to drive the host vehicle to a safe place. This may complicate continuous execution of the host vehicle's autonomous driving.

[0010] It is now assumed that the published autonomous travel control apparatus is configured to determine a travel route of the host vehicle using images picked up by an on-board camera and perform autonomous steering of the host vehicle. In this case, a malfunction in the on-vehicle camera may make it difficult for the autonomous driving control device to drive the host vehicle to a safe place even if the device includes the duplicated vehicle control system.

[0011] Such a malfunction of the in-vehicle camera may therefore cause the operation mode of the host vehicle to be switched from the autonomous driving mode to the manual driving mode. This makes it difficult for the driver to perform autonomous travel of the host vehicle using the published autonomous travel control device, resulting in the published autonomous travel control device having lower usability.

[0012] In view of the above circumstances, a first aspect of the present invention seeks to provide autonomous travel control devices each configured and capable of performing autonomous travel of a host vehicle using an image picked up by an on-vehicle camera to perform autonomous driving of the host vehicle even if there is a malfunction in the in-vehicle camera.

[0013] A second aspect of the present invention seeks to provide program products each causing a processor to perform autonomous driving of a host vehicle using an image captured by an on-board camera and capable of causing that a processor continuously performs autonomous driving of the host vehicle even when there is a malfunction in the on-vehicle camera.

[0014] According to a first exemplary aspect of the present invention, there is provided an autonomous driving control device installable in a vehicle that includes: at least first and second cameras configured to respectively capture at least first and second images of at least first and second directed areas that are differ at least in part with respect to the environment of the vehicle; and a vehicle control unit configured to control a driving state of the vehicle. The autonomous driving control device includes a driving control unit configured to determine a value of at least one controlled variable for autonomous driving of the vehicle based on the first and second images and an autonomous driving condition, and to output the value of the at least one controlled variable to the vehicle control unit to effect that the vehicle control unit performs a task of autonomous driving of the vehicle. The autonomous driving control device includes a camera monitoring unit configured to determine whether at least one of the first and second cameras is malfunctioning, and when it is determined that one of the first and second cameras is malfunctioning, the autonomous driving condition based on one of the first and second cameras to limit second directional areas corresponding to the one of the first and second cameras that is malfunctioning. The driving control unit is configured to determine the value of the at least one controlled variable according to the limited autonomous driving condition to cause the vehicle control unit to continuously perform the task of autonomous driving of the vehicle according to the determined value of the at least one controlled variable.

[0015] According to a second exemplary aspect of the present invention, there is provided a program product for a vehicle that includes at least first and second cameras configured to capture, respectively, at least first and second images of at least first and second directional areas of the vehicle that are at least partially distinguish from each other. The vehicle includes a vehicle control unit configured to control a driving state of the vehicle and a driving control unit. The driving control unit is configured, based on the first and second images and an autonomous driving condition, to determine a value of at least one controlled variable for autonomous driving of the vehicle and to output the value of the at least one controlled variable to the vehicle control unit to cause the vehicle control unit performs a task of autonomous driving of the vehicle. The computer program product includes a non-transitory computer-readable storage medium and a set of computer program instructions stored on the computer-readable storage medium. The instructions cause a computer to perform: 1. a first step of determining whether one of the first and second cameras is malfunctioning, 2. a second step of, when it is determined that one of the first and second cameras is malfunctioning, limiting the autonomous driving condition based on one of the first and second directed ranges that corresponds to the one of the first and second cameras that is malfunctioning .

[0016] Therefore, this configuration of the respective first and second example aspects enables autonomous driving of the vehicle to be performed based on the limited autonomous driving conditions even if one of the first and second cameras malfunctions. This eliminates the need to suspend the autonomous driving of the vehicle, making the autonomous driving device more convenient for drivers of the vehicle. character list

[0017] Further aspects of the present invention will become clear from the following description of the embodiments with reference to the accompanying drawings, which show: figure1 is a block diagram schematically showing an example of the overall structure of an autonomous travel control system according to an exemplary embodiment of the present invention; figure 2 is a perspective view schematically showing an example of the external appearance of a camera module used in FIG figure 1; figure 3 is a longitudinal cross-sectional view of the camera module to show an example of the internal structure of the camera module; figure 4 is a view schematically showing an example of the relationship between (1) a set of an angular field of view and a depth of field of a wide-angle camera of the camera module, (2) a set of an angular field of view and a depth of field of a narrow-angle camera (narrow-angle camera) of the camera module, (3) a set of an angular field of view and a depth of field of a telephoto camera (telephoto camera) of the camera module; and figure 5 is a flowchart that schematically illustrates a camera monitoring routine in accordance with the exemplary embodiment of the present invention. Detailed description of the embodiment

[0018] An exemplary embodiment of the present invention will be described below with reference to the accompanying drawings.

[0019] The following is an example of the configuration of an autonomous driving system 1 according to the exemplary embodiment of the present invention with reference to FIG figure 1 described.

[0020] According to figure 1 is the autonomous driving system 1 installed and designed, for example, in an autonomous vehicle V such as a passenger car, the autonomous vehicle V , hereinafter simply referred to as vehicle V referred to as driving autonomously.

[0021] The autonomous driving system 1 contains, for example, an electronic autonomous driving control unit (ECU) 10 as the main component. In addition to the autonomous driving control ECU 10 which is simply called an ECU 10 is called, contains the autonomous driving system 1 for example an on-board position sensor unit 12 , a surrounding situation sensor unit 13 , a communication module 14 , a street information repository 16 , an output unit 17 , an input device 18 and a camera module 30 . These components 12 until 18 and 30 can be communicated with the ECU 10 connectable.

[0022] In the vehicle V is a navigation system 22 , a driving force control ECU 24 , a braking force control ECU 26 and a steering control ECU 28 Installed. These components 20 until 28 are via an in-vehicle network 20 installed in the vehicle V, communicably connectable to each other.

[0023] The vehicle's position sensor unit 12 is able to measure the current position of the vehicle V. The vehicle's position sensor unit 12 contains, for example, a GPS receiver (GPS: Global Positioning System) and a gyro sensor. The GPS receiver is configured to receive GPS signals transmitted from GPS satellites via a GPS antenna to determine, for example, the latitude and longitude of the position of the GPS antenna of the vehicle V based on the received GPS signals as position data of the GPS antenna.

[0024] The gyro sensor is configured to measure a value of angular velocity about at least one of a predetermined pitch axis, roll axis, and yaw axis of the vehicle V . These pitch, roll and yaw axes pass through the center of gravity of the vehicle V. The pitch axis represents a horizontal axis parallel to the width direction of the vehicle V, the yaw axis represents a vertical axis parallel to the height direction of the vehicle V, and the roll axis represents a longitudinal axis parallel to the longitudinal direction of the vehicle V.

[0025] The vehicle's position sensor unit 12is designed to convert, as a current position of the vehicle V, a current position of the center of gravity of the vehicle V based on a predetermined positional relationship between the GPS antenna and the center of gravity of the vehicle V, the measured position data of the GPS antenna and the measured value of the angular velocity calculate at least one of the vehicle V's pitch, roll and yaw axes. Then the onboard position sensor unit 12 designed, a measurement signal indicative of the current position of the vehicle V to the ECU 10 to send.

[0026] The surrounding situation sensor unit 13 is able to measure a current surrounding situation around the vehicle V; the current environmental situation around the vehicle V can be controlled by the ECU 10 be used.

[0027] The surrounding situation sensor unit 13 includes, for example, at least one of a laser radar sensor, a millimeter-wave radar sensor, an ultrasonic wave sensor. The surrounding situation sensor unit 13 Is designed, (1) to emit test waves to a predetermined surrounding area around the vehicle V, (2) to receive reflection waves, i.e. echoes generated based on a reflection of the transmitted test waves by objects arranged in the vicinity of the vehicle V, (3) detect the presence, location, size, and / or distance of the respective objects based on the received reflection waves.

[0028] For example, the objects contain 1. one or more vehicles driving in the vicinity driving in the vicinity of vehicle V, 2. one or more environmental obstacles that are arranged in the vicinity of the vehicle V and interfere with the vehicle V traveling.

[0029] Which include one or more vehicles driving in the area 1. a preceding vehicle traveling in the same lane as vehicle V and located in front of vehicle V, 2. a preceding vehicle running in a lane adjacent to the lane of the vehicle V and located on the front of the vehicle V, 3. one or more coming vehicles, 4. one or more oncoming vehicles.

[0030] Containing one or more environmental obstacles 1. one or more stopping vehicles, 2. one or more fallen objects, 3. one or more stopping objects, 4. one or more pedestrians.

[0031] The surrounding situation sensor unit 13 is able to send a measurement signal to the ECU 10 to be sent indicating the current environmental situation.

[0032] The communication module 14 allows the ECU 10 to communicate via radio with traffic servers installed outside the vehicle V, thereby (1) traffic condition information, (2) obtain weather condition information.

[0033] The weather condition information represents the weather condition such as a bright condition, a rainy condition, a cloudy condition, a snowy condition, a foggy condition, or a sandstorm condition around the vehicle V, which can be collected by at least one of the traffic servers.

[0034] The traffic condition information includes various kinds of traffic information about each road on which the vehicle V can travel. The traffic condition information may include, for example, a speed limit of each passable road, information as to whether passing is permitted on each passable road, information as to whether there are traffic rules for each passable road.

[0035] The Street Information Store 16composed of, for example, a rewritable storage medium such as a flash ROM rewritably stores road information on one or more roads on which the vehicle V is scheduled to travel; where the road information about the respective planned roads includes: 1. the type of road, i.e. whether the road is an urban road or an expressway, how many lanes the road has, and whether there are one or more oncoming lanes on the road, 2. the width of each lane on the road, 3. the center line of each lane on the road, 4. the curvature of the road if the road is a curved road, 5. the positions of one or more stop lines marking the road, 6. the positions of one or more traffic lights if they are present on the road.

[0036] Note that the navigation system described later 22 For example, the road information is designed for the ECU 10 to provide.

[0037] The output unit 17 includes, for example, an image display and a speaker and is capable of visually and / or audibly outputting various messages to a driver of the vehicle V using the image display and / or the speaker.

[0038] The input unit 18 contains, for example, operating switches and / or operating levers and enables a driver of the vehicle V to enter the ECU 10 enter various instructions for autonomously driving the vehicle V using the operating switches and / or operating levers.

[0039] The camera module 30 is capable of capturing images of the surrounding area around the vehicle V.

[0040] The camera module 30 consists, for example, of three cameras each having different angular fields of view, i.e. a wide-angle camera 32w , a narrow-angle camera 32n , whose angular field of view is narrower than that of the wide-angle camera 32w is, and a telephoto camera 32t , whose angular field of view is narrower than that of the narrow-angle camera 32n is.

[0041] Note that the angular field of view of the respective lenses 33w , 33n and 33t at least one is out 1. a diagonal angular field of view corresponding to a diagonal direction of a picked-up image of the corresponding camera, i.e. a diagonal line of the light-receiving surface of the corresponding imaging device, 2. a horizontal angular field of view corresponding to a horizontal direction of a picked-up image of the corresponding camera, i.e. a horizontal direction of the light-receiving surface of the corresponding imaging device, 3. A vertical angular field of view corresponding to a vertical direction of a picked-up image of the respective camera, i.e., a vertical direction of the light-receiving surface of the respective imaging device.

[0042] This exemplary embodiment uses the horizontal angular field of view of each lens 33w , 33n , 33t than the angular field of view of the corresponding lens.

[0043] Images such as two-dimensional frame images taken from the respective cameras 32w , 32n and 32t are used for recognizing lane markers on a planned road on which the vehicle V is planned to travel and for recognizing objects present in the surrounding area around the vehicle V. The structure of the camera module 30 according to the exemplary embodiment will be described in detail later.

[0044] The navigation system 22 is designed to store various pieces of road information and pieces of map information about a plurality of roads that the vehicle V can travel.

[0045] In particular, the navigation system 22 designed, a road map on the image display of the output unit 17display around the current position of the vehicle V and on the road map a selected route from the current position of the vehicle V to a destination in response to a driver of the vehicle V entering the destination using the input unit 17 enters to display. Then there is the navigation system 22 designed to obtain the road information about one or more roads included in the selected route and the road information about the one or more planned roads for the ECU 10 to provide. The navigation system 22 is also configured to provide visual and audible guidance to a driver of the vehicle V using the image display and the speaker to allow the vehicle V to travel along the selected route.

[0046] The driving force control ECU 24 is configured to control at least one drive actuator such as an internal combustion engine and / or an electric motor; the at least one drive actuator outputs a controlled drive force to thereby rotatably drive drive wheels of the vehicle V .

[0047] In particular, the driving force control ECU 24 configured to control the at least one drive actuator to cause the at least one drive actuator to generate a drive force based on an operation amount of an accelerator pedal operated by a driver of the vehicle V when the operation mode of the vehicle V is set to a manual drive mode is, whereby the driving wheels are rotatably driven based on the generated driving force.

[0048] The driving force control ECU 24 is also designed to receive a required driving force from the ECU 10 to receive when the operating mode of the vehicle V is set to an autonomous driving mode, and to control the at least one drive actuator to cause the at least one drive actuator to generate a driving force that satisfies the required driving force, thereby driving the drive wheels based on the generated Driving force are rotatably driven.

[0049] Note that the operation mode of the vehicle V is determined by the ECU, for example 10 can be changed according to an instruction given by a driver by operating the input unit 17 is sent.

[0050] The brake force control ECU 26 is configured to control at least one brake actuator; the at least one brake actuator outputs a controlled braking force to brake the vehicle V.

[0051] In particular, the brake force control ECU 26 configured to control the at least one brake actuator to cause the at least one brake actuator to generate a braking force based on an operation amount of a brake pedal operated by a driver of the vehicle V when the operation mode of the vehicle V is set to the manual drive mode is, whereby the vehicle V is decelerated based on the generated braking force.

[0052] The brake force control ECU 26 is also designed to receive a required or required braking force from the ECU 10 to receive when the operating mode of the vehicle V is set to the autonomous driving mode, and to control the at least one brake actuator to cause the at least one brake actuator to generate a braking force that meets the required braking force, whereby the vehicle V based on the generated braking force is slowed down.

[0053] The steering control ECU 28 is configured to control an electric motor included in a steering mechanism of the vehicle V; the electric motor of the steering mechanism outputs a controlled steering torque that controls the vehicle V steering wheel.

[0054] In particular, the steering control ECU 28configured to control the electric motor of the steering mechanism to cause the electric motor to generate, as steering torque, a controlled assist torque based on an operation amount of the steering wheel operated by a driver of the vehicle V when the operation mode of the vehicle V is set to the manual drive mode is set. This controlled support torque supports the steering operation of the steering wheel by the driver.

[0055] The steering control ECU 28 is also designed to receive a required or required steering angle from the ECU 10 to receive when the running mode of the vehicle V is set to the autonomous driving mode, and to control the electric motor to cause the electric motor to generate the steering torque that satisfies the required steering angle.

[0056] The following is an example of the configuration of the ECU 10 described.

[0057] According to figure 1 serves the ECU 10 for example, as an autonomous driving control device and consists of, for example, a processing unit such as a central processing unit (CPU) 2 . The ECUs 10 may consist of another type of processing unit such as an application specific integrated circuit (ASIC). The ECUs 10 also consists of a memory 4 that includes, for example, a non-volatile storage medium that includes, for example, random access memory (RAM) and read only memory (ROM).

[0058] Various programs including control programs for causing the CPU 2 various tasks, i.e. routines, are stored in the memory 4 saved. Also, various pieces of data used by the CPU 2 can be used in memory 4 saved. The CPU 2 reads at least one of the control programs from memory 4 and executes the at least one control program to execute the routine corresponding to the at least one control program. In other words, the CPU 2 executes the at least one control program to generate predetermined functional modules such as a drive control unit 6 and a camera surveillance unit 8 (see dashed blocks in figure 1) to implement based on the at least one control program. In addition, the CPU 2 designed, the entire operations of the ECU 10 to control.

[0059] The drive control unit 6 performs an autonomous driving control task that causes at least one of the ECUs 24 , 26 and 28 the vehicle V drives autonomously when the operation mode of the vehicle V is set to the autonomous drive mode.

[0060] In particular, the ECU 10 sets the operating mode of the vehicle V to the autonomous driving mode when an instruction indicating autonomous driving is received from a driver of the vehicle V via the input unit 18 is entered.

[0061] For example, the instruction indicating the autonomous driving includes execution information for the autonomous driving to be executed; the performance information includes automatic cruise control (ACC), autonomous vehicle V steering, and / or autonomous vehicle V driving in a predetermined level of the previously defined levels.

[0062] The ACC is configured to adjust the speed of the vehicle V to cause the vehicle V to follow a preceding vehicle in front of the vehicle V . The autonomous steering is configured to autonomously control the steering wheel of the vehicle V to cause the vehicle V to travel within the appropriate lane on one or more planned roads.

[0063] The autonomous driving is designed to drive the vehicle V autonomously in the predetermined level of the previously defined levels instructed by the driver of the vehicle V.

[0064] Also receives the drive control unit 6 from the navigation system 22 (1) the route selected by the driver from the current position of the vehicle V to the destination, (2) the road information about the one or more roads included in the selected route.

[0065] Then the drive control unit adjusts 6 based on the autonomous driving execution information and the road information, autonomous driving conditions set for at least one of the ECUs 24 , 26 and 28 are required to drive the vehicle V autonomously. The autonomous driving conditions include, for example, a value of vehicle speed V, a vehicle speed limit V, i.e., an upper speed limit, during autonomous driving, a value of steering wheel steering angle, and a non-autonomous driving zone on one or more planned roads.

[0066] Also receives the drive control unit 6 the measurement signals from the respective sensor units 12 and 13 , receives the images from the respective cameras 32w , 32n and 32t and also obtains the traffic condition information and weather condition information through the communication module 14 . The above road information, the measurement signals, the images, the traffic condition information, and the weather condition information, which are collectively referred to as pieces of autonomous travel request information, are obtained from the respective devices.

[0067] Then the drive control unit determines 6 a value of at least one of the controlled variables for each of the at least one drive actuator, the at least one brake actuator, and the electric motor of the steering mechanism according to the autonomous travel conditions and the autonomous travel request information pieces each time the autonomous travel request information pieces are received.

[0068] Based on the calculated value of the at least one controlled variable for each of the at least one drive actuator, the at least one brake actuator and the electric motor of the steering mechanism, the drive control unit obtains 6 at least one out (1) a required driving force for the at least one driving actuator, (2) a required braking force for the at least one brake actuator, (3) a required steering angle for the electric motor.

[0069] Then there is the drive control unit 6 to the at least one of the driving force control ECU 24 , Brake Force Control ECU 26 and steering control ECU 28 the corresponding at least one of the required driving force for the at least one drive actuator, required braking force for the at least one brake actuator and required steering angle. This causes at least one of the driving force control ECU 24 , Brake force control ECU 26 and steering control ECU 28 performs a corresponding autonomous driving task instructed by the driver of the vehicle V .

[0070] Note that at least one of the ECUs 24 , 26 and 28 for example, serves as a vehicle control unit configured based on frame images obtained from the respective cameras 32w , 32n and 32t are recorded, and the autonomous driving conditions to determine a value of at least one controlled variable needed to cause the vehicle V to drive autonomously.

[0071] The above autonomous driving control task of the drive control unit 6 is continuously, i.e., repeatedly performed while the running mode of the vehicle V is set to the autonomous driving mode.

[0072] On the other hand is the camera surveillance unit 8 designed, 1. the operating status of the wide-angle camera 32w , the narrow-angle camera 32n and the telephoto camera 32t to monitor while the drive control unit 6 executes the autonomous driving control task as described above, 2. According to the monitoring result of the respective three cameras 32w , 32n and 32t to determine whether any of the three cameras 32w , 32n and 32t has a malfunction 3. To limit a part of the autonomous driving control task when it is determined that one of the three cameras 32w , 32n and 32t malfunctions to accordingly allow the autonomous driving of the vehicle V to be continuously performed.

[0073] The following is an example of the configuration of the camera module 30 and an example of functions of the camera module 30 with reference to the figure 2 and figure 3 described.

[0074] According to the figure 2 and figure 3 consists of the camera module 30 from the three cameras 32w , 32n and 32t , a camera body 40 and a bracket assembly 50 . The three cameras 32w , 32n and 32t are in the camera body 40 Installed.

[0075] Each of the cameras 32w , 32n and 32t contains, for example, a corresponding lens barrel 34w , 34n and 34t , having opposite first and second open ends, and a corresponding lens 33w , 33n and 33t , which is arranged coaxially in the corresponding lens barrel closer to the first open end of the corresponding lens barrel. This allows outside light to enter the lens barrel of the respective camera 32w , 32n and 32t through the first open end, so that this onto the corresponding lens 33w , 33n , 33t occurs.

[0076] Each of the cameras 32w , 32n and 32t also includes, for example, a lens, not shown, adjusted to be located in the corresponding lens barrel closer to the second open end of the corresponding lens barrel. The lens, not shown, for a respective camera 32w , 32n and 32t is set or determined is designed, for example, light that the corresponding lens 33w , 33n , 33t has happened in terms of an optical aberration such as a chromatic aberration of the respective lens 33w , 33n , 33t to correct.

[0077] Each of the cameras 32w , 32n and 32t also contains a corresponding imaging device 35w , 35n and 35t and a corresponding rectangular plate-shaped imaging plate 36w , 36n and 36t .

[0078] Each of the imaging plates 36w , 36n and 36t has opposing first and second major surfaces, and the respective lens barrel 34w , 34n and 34t is on the first major surface of the corresponding imaging plate 36w , 36n and 36t assembled. Any imaging device 35w , 35n and 35t is on the first major surface of the corresponding imaging plate 36w , 36n and 36t realized.

[0079] In particular, each lens barrel 34w , 34n and 34t at the perimeter of its second open end on the first major surface of the corresponding imaging plate 36w , 36n and 36t mounted in such a way that the corresponding imaging device 35w , 35n and 35t coaxial to the optical axis of the corresponding lens 33w , 33n and 33t is.

[0080] Any of the imaging devices 35w , 35n and 35t is formed by, for example, a color / monochrome CCD (CCD: Charge Coupled Device) image sensor or a color / monochrome CMOS (CMOS: Complementary Metal Oxide Semiconductor) image sensor. Any imaging device 35w , 35n and 35tconsists of a plurality of light receiving elements corresponding to respective pixels arranged two-dimensionally in a vertical direction corresponding to the height direction of the vehicle V and a horizontal direction corresponding to the width direction of the vehicle V. The light receiving elements of each imaging device 35w , 35n , 35t form a light-receiving surface, and the light-receiving surface of each imaging device 35w , 35n , 35t is directed toward the front end of the vehicle V, for example.

[0081] The Lens 33w , 33n , 33t the respective camera 32w , 32n , 32t is designed light entering the appropriate lens barrel 34w , 34n , 34t enters onto the light receiving surface of the corresponding imaging device 35w , 35n , 35t to focus.

[0082] The camera body 40 consists of a first housing segment 41 and a second housing segment 42 that are assembled to the camera body 40 to build. The first and second housing segments 41 and 42 consist, for example, each of a hard material with a relatively high heat dissipation capacity, such as aluminum.

[0083] The first housing segment 41 has a generally rectangular goblet shape having a bottom wall 41a , an open wall 41b opposite the bottom wall 41a , a first side wall 41c and a second side wall 41d opposite the first open side wall 41c having. The first side wall 41c consists of a peripheral edge and an opening defined around the peripheral edge. The peripheral edge has a first end that connects with a corresponding edge of the bottom wall 41a is connected so that the first side wall 41c perpendicular to the bottom wall 41 extends. Similarly, the second side wall 41d a first end mating with a corresponding edge of the bottom wall 41a is connected, so that the second side wall 41d perpendicular to the bottom wall 41a extends.

[0084] The first housing segment 41 has a flange 41e extending from a second end of the first side wall 41a , opposite the first end, perpendicular to the first sidewall 41a stretches away.

[0085] The second housing segment 42 has a substantially rectangular dish shape having a bottom wall 42a , an open wall 42b opposite the bottom wall 42a , a first side wall 42c and a second side wall 42d opposite the first side wall 42c having. The first side wall 42c has a first end connected to a corresponding edge of the bottom wall 42a is connected so that the first side wall 42c perpendicular to the bottom wall 42a extends. Similarly, the second side wall 42d a first end mating with a corresponding edge of the bottom wall 42a is connected, so that the second side wall 42d perpendicular to the bottom wall 42a extends.

[0086] The second housing segment 42 is the first housing segment 41 arranged facing such that a second end of each of the first and second side walls 42c and 42d , opposite the first end of the corresponding side wall, with the second end of the corresponding one of the first and second side walls 41c and 41d of the first housing segment 41 is connected by means of screws, for example, with which the camera body 40 is formed in which an installation space between the first and second housing segments 41 and 42 is defined.

[0087] The cameras 32w , 32n and 32t are through the opening of the first side wall 41c of the first housing element 41 so tightly mounted that (1) these are aligned in the vertical direction with spaces therebetween, (2) the first open ends of the respective lens barrels 34w , 34n and 34t from the first housing element 41 exposed.

[0088] As is the case, for example, in the figure 2 and figure 3 is the wide-angle camera 32w at the lowest portion of the first side wall 41c arranged, the telephoto camera 32t is at the top portion of the first side wall 41c arranged, and the narrow-angle camera 32n is higher than the wide-angle camera 32w and lower than the telephoto camera 32t arranged.

[0089] The camera body 40 , which consists of the first and second housing segments 41 and 42 consists is arranged in an interior of the vehicle V such that the bottom wall 41 and the first side wall 41c of the first housing segment 41 close to an interior surface 52a a front windshield 52 of the vehicle V are arranged while the open wall 41b of the first housing segment 41 directed downward from the vehicle V and the extending direction of the flange 41e is directed to the front end of the vehicle V, for example.

[0090] The bracket assembly is arranged, the camera body 40 , i.e. the camera module 30 , on a section of the inner surface 52a the front windshield 52 to assemble; the section of the inner surface 52a the front windshield 52 is determined to be out of the driver's line of sight. The bracket assembly 50 consists, for example, of a holder 54 and several mounting blocks 56 .

[0091] The bracket 54 used for mounting on the inner surface 52a the front windshield 52 .

[0092] The bracket 54 has a substantially rectangular plate-like shape with a substantially trapezoidal concave depression 54a in their front 54b is trained. The bracket 54 has opposite first and second main surfaces, and is arranged in the interior of the vehicle V such that its first main surface along the inner surface 52a the front windshield 52 is arranged. The mounting blocks 56 are distributed between the inner surface 52a the front windshield 52 and the first major surface of the bracket 54 arranged so that the bracket 54 over the mounting blocks 56 firmly to the front windshield 52 is mounted.

[0093] As described above, the cameras 32w , 32n and 32t firmly through the opening of the first side wall 41c of the first housing element 41 mounted in such a way that (1) the telephoto camera 32t , the narrow-angle camera 32n and the wide-angle camera 32w aligned vertically in this order from the top, (2) the first open ends of the respective lens barrels 34w , 34n and 34t from the first housing element 41 exposed.

[0094] This will cause when the bracket 54 , on which the camera module 30 is mounted, on the inner surface 52 the front windshield 52 attached, optical axes At , On and ugh of the respective cameras 32t , 32n and 32w (1) vertically aligned in this order from the top, (2) extend toward the front end of the vehicle V.

[0095] The concave depression 54a consists of a centered inner peripheral edge, which is a shorter side of the trapezoidal concave depression 54a corresponds, such that the first side wall 41c of the first housing segment 41located below the centered inner peripheral edge.

[0096] The concave depression 54a also consists of a pair of slanting edges each extending obliquely outward from both ends of the centered inner peripheral edges toward the front end of the vehicle V, for example.

[0097] The bracket assembly 50 also consists, for example, of a mounting wall 58 integrally extending from the centered inner peripheral edge of the concave recess 54a extends downward; the mounting wall 58 , having opposite shorter top and bottom and opposite longer vertical sides, is in the opening of the first side wall 41c arranged around the lens barrels 34w , 34n and 34t on the peripheral wall of the first side wall 41c to fix, bringing the cameras 32w , 32n and 32t be positioned while their lenses 33w , 33n and 33t are directed to the front of the vehicle V, for example.

[0098] The bracket assembly 50 also consists, for example, of a pair of inclined side walls 62 , extending integrally from the respective sloping edges of the concave recess 54a extend downwards. In other words, each of the sloping side walls 62 , having opposite top and bottom ends, is arranged such that the top of the respective sloping side walls 62 with the corresponding sloping edge of the concave recess 54 connected is.

[0099] The sloping side walls 62 extend obliquely outwards along the respective oblique edges of the concave depression 54a from the longer vertical sides of the mounting wall 58 toward, for example, the front end of the vehicle V while rotating around the optical axes At , On and ugh of the respective cameras 32t , 32n and 32w are centered and tilted.

[0100] In addition, there is the mounting arrangement 50 from a base wall, for example 64 , which has a substantially trapezoidal shape sloping outwards from the bottom of the mounting wall 58 toward, for example, the front of the vehicle V below the optical axes At , On and ugh of the respective cameras 32t , 32n and 32w extends. In other words, the base wall 64 has opposite non-parallel sides connected to the bottoms of the respective sloping side walls.

[0101] I.e. the sloping side walls 62 and the base wall 64 , which is associated with it, surround the optical axes At , On and ugh of the respective cameras 32t , 32n and 32w , which leads to the arrangement of the sloping side walls 62 and the base wall 64 a hood 60 forms to prevent light from outside the walls 62 and 64 entry.

[0102] This, therefore, allows light from the direction ahead of the vehicle V to pass through the front windshield 52 on the cameras 32w , 32n and 32t incident, so that the incident light through the respective cameras 32w , 32n and 32 Will be received.

[0103] The light coming from the respective cameras 32w , 32n and 32t is received is through the appropriate lens 33w , 33n , 33t and the lens, not shown, mounted on the light-receiving surface of the corresponding imaging device 35w , 35n , 35tis fixed, so that the respective two-dimensionally arranged light-sensitive elements (pixels) receive a corresponding light component during a controllable shutter time, i.e. an exposure period. Then, the respective photosensitive elements convert a corresponding received light component into an electric charge, i.e., an electric signal corresponding to the intensity of the received light component, thereby generating the electric signals as received light data, i.e., picking up a two-dimensional frame image.

[0104] Also includes the camera module 30 a control board 44 which consists of a rigid substrate such as a glass epoxy substrate and which has a substantially rectangular plate-like shape. The control circuit board 44 is installed in the installation space that is between the first and second housing segments 41 and 42 is defined. The control circuit board 44 contains a control circuit 46 , which consists of many electrical and / or electronic elements.

[0105] Each of the cameras 32w , 32n and 32t also includes appropriate mapping circuitry 37w , 37n and 37t , attached to the corresponding imaging plate 36w , 36n and 36t is installed.

[0106] Each of the imaging circuits 37w , 37n and 37t is communicable with the corresponding imaging device 35w , 35n and 35t tied together. Each of the imaging circuits 37w , 37n and 37t is also communicable with the control circuit 46 connected to the control board 44 by means of, for example, a corresponding flexible printed circuit board (FPC) 38w , 38n and 38t is installed.

[0107] In particular, the control circuit 46 designed, the corresponding imaging device 35w , 35n and 35t the respective camera 32w , 32n , 32t corresponding, for example, to a predetermined value of exposure duration, i.e. shutter time, and a predetermined value of frame rate in cooperation with the corresponding imaging circuit 37w , 37n , 37t to control to cause the corresponding imaging device 35w , 35n and 35t sequentially captures two-dimensional frame images of the surrounding area around the vehicle V . Then the control circuit receives 46 the two-dimensional frame images sequentially from the respective cameras 32w , 32 and 32t are recorded as pieces of image data.

[0108] The control circuit 46 receives the image data parts from the respective cameras 32w , 32n and 32t and performs various tasks, including an image recognition task, based on the pieces of image data (1) detects lane markings on a planned road on which the vehicle V is to travel, (2) Objects present in the surrounding area around the vehicle V detects.

[0109] Objects recognizable by the image recognition task may include, for example, obstacles such as pedestrians, bicycles, and other vehicles, and structures such as traffic lights, road signs, and / or buildings.

[0110] The camera module 30 contains at least one connector or plug 48 , which is attached to the control circuit board 44 is mounted. The control circuit 46 is with the ECU 10 via the at least one connector 48 communicatively connected. The control circuit 46 is designed to the ECU 10 the image data parts by the respective cameras 32w , 32n , 32t are obtained, and the results of the image recognition task via the at least one connector 48 in response to commands from the ECU 10 to be sent.

[0111] In particular, according to figure 4 each of the cameras 32w , 32n and 32t a corresponding mapping area at least partially different from those of the others.

[0112] In particular, the lens 33w and the imaging device 35w the camera 32w configured to capture an image of a first imaging area defined around the optical axis Aw, and the lens 33n and the imaging device 35n the camera 32n are designed to capture an image of a second imaging area, which is defined about the optical axis An and which at least partially differs from the first imaging area. In a similar way are the lens 33t and the imaging device 35t the camera 32t configured to capture an image of a third imaging area defined about the optical axis At and at least partially different from the first and second imaging areas.

[0113] The respective first to third imaging areas correspond to respective first to third directional areas such as forward areas in the traveling direction of the vehicle V, for example.

[0114] The Lens 33w the wide-angle camera 32w consists, for example, of a wide-angle lens which is formed, for example, as a concave meniscus lens which consists, for example, of a transparent material such as glass. The Lens 33w is arranged such that its main concave surface faces the imaging device 35w is directed.

[0115] The Lens 33n the narrow-angle camera 32n consists, for example, of a narrow-angle lens having an angular field of view θn narrower than an angular field of view θw of the lens 33w the wide-angle camera 32w is. The Lens 33n is formed, for example, as a concave meniscus lens made of, for example, a transparent material such as glass. The Lens 33n is arranged such that its main concave surface faces the imaging device 35n is directed.

[0116] In addition, there is the lens 33t the telephoto camera 32t from, for example, a telephoto lens having an angular field of view θt narrower than the angular field of view θn of the lens 33n the narrow-angle camera 32n is. The Lens 33t is formed, for example, as a concave lens made of, for example, a transparent material such as glass. The Lens 33t is arranged such that its main concave surface is directed toward the front end of the vehicle V .

[0117] Because the wide-angle camera 32w the wide angle lens 33w is the angular field of view θw of the lens 33w the wide-angle camera 32w set to a relatively wide angle of 120 degrees, for example. The wide angle camera 32w has a depth of field Dw within the angular field of view θw of the wide-angle lens 33w on; the depth of field Dw is set to within the range of a predetermined near point Dwc which is, for example, a point of the nearest focal point of the wide-angle lens 33w and a predetermined far point Dwf which is, for example, a farthest focal point of the wide-angle lens 33w is employed.

[0118] The lens barrel 34n the narrow-angle camera 32n is in the first housing segment 41 arranged such that at least one predetermined rear principal point of the narrow-angle lens 33n vertically and horizontally to a corresponding predetermined rear principal point of the wide-angle lens 33w is aligned. In addition, the optical axis On the narrow-angle camera 32n eccentric in the vertical direction with respect to the optical axis ugh the wide-angle camera 32w adjusted to the horizontal position of the optical axis On the narrow-angle camera 32n in vertical alignment to the horizontal position of the optical axis ugh to keep.

[0119] Because the narrow-angle camera 32nthe narrow angle lens 33n used is the angular field of view θn the lens 33n the narrow-angle camera 32n set to an intermediate angle that is narrower than the angular field of view θw the lens 33w is, for example 60 degrees. These settings allow the angular field of view θn of the lens to change 33n the narrow-angle camera 32n partly with the angular field of view θw of the lens 33w the wide-angle camera 32w in the normal direction perpendicular to the corresponding angular fields of view.

[0120] The narrow angle camera 32n has a depth of field Dn within the angular field of view θn the narrow-angle lens 33n on; the depth of focus Dn is on within the range of a predetermined near point Dnc , which is, for example, a point of the closest focal point of the narrow-angle lens 33n and to a predetermined distant point Dnf , which is, for example, a point of the farthest focal point of the narrow-angle lens 33n is employed.

[0121] In particular, the distant point dwf the wide-angle camera 32w on farther from a driver of the vehicle V than the near point Dnc the narrow-angle camera 32n set, and the near point Dnc the narrow-angle camera 32n becomes farther from a driver of the vehicle V than the near point Dwc of the wide-angle camera 32w set. In addition, the far point Dnf of the narrow-angle camera 32n on farther from a driver of the vehicle V than the far point Dwf of the wide-angle camera 32w set.

[0122] These settings allow that (1) the distant point dwf the wide-angle camera 32w between the near and far points Dnc and Dnf the narrow-angle camera 32n are arranged (2) the establishment of a coverage area Rnw in which the depth of field Dn of the narrow-angle camera 32n and the depth of field Dw of the wide-angle camera 32w in the normal direction perpendicular to the corresponding angular fields of view.

[0123] The lens barrel 34t the telephoto camera 32t is in the first housing segment 41 arranged such that at least a predetermined rear principal point of the telephoto lens 33t vertically and horizontally to a corresponding predetermined rear principal point of the narrow angle lens 33n is aligned. Also, the optical axis is At of the telephoto camera 32t eccentric in the vertical direction with respect to the optical axis ugh the wide-angle camera 32w and the optical axis On the narrow-angle camera 32n adjusted to the horizontal position of the optical axis At the telephoto camera 32t in vertical alignment to the horizontal position of the respective optical axis ugh and On to keep.

[0124] Since the telephoto camera 32t the telephoto lens 33t used is the angular field of view θt the lens 33t the telephoto camera 32t set at a small angle narrower than that of the angular field of view θw the lens 33w and the angular field of view θn the lens 33n is, for example 35 degrees. These settings allow that (1) the angular field of view θt the lens 33t the telephoto camera 32t partially with the angular field of view θn the lens 33n the narrow-angle camera 32n occluded in the normal direction perpendicular to the corresponding angular fields of view, (2) the angular field of view θt the lens 33t the telephoto camera 33t partially with the angular field of view θw the lens 33w the wide-angle camera 32w in the normal direction perpendicular to the corresponding angular fields of view.

[0125] The telephoto camera 32t has a depth of field German within the angular field of view θtthe telephoto lens 33t on; the depth of field Dt is set to within the range of a predetermined near point Dtc , which is, for example, a point of the nearest focal point of the telephoto lens 33t and to a predetermined far point Dtf which is, for example, a farthest focal point of the telephoto lens 33t is employed.

[0126] In particular, the far point Dnf of the narrow-angle camera becomes 32n on farther from a driver of the vehicle V than the near point Dtc of the telephoto camera 32t set, and the near point Dtc of the telephoto camera 32t is set to be farther from a driver of the vehicle V than (1) the near point Dnc of the narrow-angle camera 32n , (2) the near point Dwc and the far point dwf the wide-angle camera 32w .

[0127] Also, the far point Dtf becomes the telephoto camera 32t is set to be farther from a driver of the vehicle V than (1) the far point Dnf of the narrow-angle camera 32n , (2) the far point Dwf of the wide-angle camera 32w .

[0128] These settings allow that (1) the far point Dnf the narrow-angle camera 32n between the near and far points Dtc and dtf of telephoto camera 32t is arranged (2) a coverage area Rtn , which is the depth of field German the telephoto camera 32t and the depth of field Dn of the narrow-angle camera 32n in the normal direction of the respective angular fields of view.

[0129] In particular, the distant point dwf the telephoto camera 32w arranged so that it lies outside a range between the near and far points Dtc and Dtf the telephoto camera 32t is defined such that the depth of field Dt of the telephoto camera 32t and the depth of field Dw of the wide-angle camera 32w differ from each other. This causes the depth of field German the telephoto camera 32t and the depth of field Dw the wide-angle camera 32w do not overlap in the normal direction of the corresponding angular fields of view.

[0130] As described above, the wide-angle cameras are 32w , the narrow-angle camera 32n and the telephoto camera 32t in the camera module 30 arranged so that the horizontal positions of the rear principal points of the respective cameras 32w , 32n and 32t in the vertical direction, i.e., the height direction, of the vehicle V are aligned with respect to each other.

[0131] The wide angle camera 32w , the narrow-angle camera 32n and the telephoto camera 32t who want the wide-angle lens 33w , the narrow-angle lens 33n and the telephoto lens 33t included, allow the angular fields of view to expand θw , θn and θt differ from each other while partially overlapping in the normal direction perpendicular to the respective angular fields of view.

[0132] The depths of field within the respective angular fields of respective adjacent pairs of the cameras 32w , 32n and 32t overlap in the normal direction perpendicular to the corresponding angular fields of view.

[0133] Note that the exemplary embodiment changes the imaging range of the wide-angle camera 32w , which is called the wide-angle field of view, is defined as a combination of the angular field of view θw and the corresponding depth of field Dw, and also the field of view of the narrow-angle camera 32n , referred to as a narrow-angle imaging range, is defined as a combination of the angular field of view θn and the corresponding depth of field Dn. Furthermore, the exemplary embodiment defines the imaging range of the telephoto camera 32t , referred to as a telephoto imaging range, as a combination of the angular field of view θt and the corresponding depth of field Dt.

[0134] These definitions therefore lead to the wide-angle imaging range of the wide-angle camera 32w , the narrow-angle field of view of the narrow-angle camera 32n and the telephoto imaging area of ​​the telephoto camera 32t differ from each other while partially overlapping in the normal direction perpendicular to the respective angular fields of view.

[0135] Specifically, since the wide-angle imaging area, the telephoto imaging area, and the narrow-angle imaging area include a relatively nearer area around the vehicle V, a relatively farther area around the vehicle V, and an intermediate area around the vehicle V between the relatively nearer area and the relatively farther cover area.

[0136] A partial overlap between the wide-angle imaging area, the narrow-angle imaging area, and the telephoto imaging area results in frame images captured by the respective cameras 32w , 32n and 32t based on their mapping areas that include a common area.

[0137] The control circuit 46 is designed to perform a known alignment task based on (1) a first reference image of the wide-angle imaging range, (2) a second reference image of the narrow-angle imaging area, (3) a third reference image of the telephoto imaging area.

[0138] The well-known alignment task enables correction of misalignment between the positions, i.e. position coordinates, of the respective optical axes ugh , On and At based on the first to third reference images.

[0139] That is, the ECU 10 is designed to provide two-dimensional frame images, i.e. closer images, of the wide-angle field of view captured by the camera 32w is captured, two-dimensional images, i.e., medium images, of the narrow-angle field of view captured by the camera 32n is captured, and two-dimensional images, i.e. more distant images, of the telephoto imaging area captured by the camera 32t is recorded, to recognize objects that are present in the vicinity of the vehicle V. That is, the ECU 10 enables easy recognition of objects present over a wide range from the relatively nearer area around the vehicle V to the relatively farther area around the vehicle V .

[0140] In addition, since the wide-angle imaging area, the narrow-angle imaging area, and the telephoto imaging area partially overlap, the closer images of the wide-angle imaging area, the middle images of the narrow-angle imaging area, and the farther images of the telephoto imaging area partially overlap each other. Therefore, this prevents erroneous recognition of an object present in at least one of the overlapping areas from the first to third images even if the object moves the at least one overlapping area, thus recognizing the object with higher accuracy.

[0141] As described above, the camera module enables 30 , that from the three cameras 32w , 32n and 32t consists that the ecu 10 recognizes lane markings on a planned road of the vehicle V and / or objects present in the surrounding area around the vehicle V with higher accuracy.

[0142] However, if at least one of the cameras 32w , 32n and 32t of the camera module 30 has a malfunction, the detection accuracy of lane markers and / or objects in a corresponding area among the relatively nearer, farther and middle areas around the vehicle V may decrease. If the ECU 10 an autonomous driving of the vehicle V in cooperation with the other ECUs 24 , 26 and 28with a reduced recognition accuracy in one of the relatively nearer, farther, and middle areas around the vehicle V, it would be difficult to reliably and safely perform autonomous driving of the vehicle V accordingly.

[0143] In view of this is the camera surveillance unit 8 the ECU 10 designed, the respective cameras 32w , 32n and 32 to monitor to determine whether the respective cameras 32w , 32n and 32t function normally during autonomous driving of the vehicle V. The camera surveillance unit 8 is also designed (1) interrupt the autonomous driving of the vehicle V when it is determined that the camera 32n has a malfunction (2) when it is determined that one of the cameras 32t and 32w has a malfunction, the autonomous driving conditions set by the drive control unit 6 be set to limit according to the imaging range of the non-functioning camera while the autonomous driving of the vehicle V is continued.

[0144] As described above, the CPU leads 2 Instructions of the at least one control program, which is in the memory 4 is saved to the functions of the camera surveillance unit 8 to realize. In other words, the CPU 2 executes instructions of the at least one control program every predetermined control period to act as a camera surveillance unit 8 to serve for executing a camera monitoring routine shown in the flowchart of FIG figure 5 is shown.

[0145] When the camera monitoring routine is started as a main routine, the CPU determines 2 in step S110 whether the ECUs 24 , 26 and 28 perform the autonomous driving of the vehicle V based on the autonomous driving conditions set thereby. The CPU 2 ends the camera monitoring routine if it is determined that the ECUs 24 , 26 and 28 do not perform the autonomous driving of the vehicle V (No in step S110 ).

[0146] If it is determined that the ECUs 24 , 26 and 28 perform the autonomous driving of the vehicle V (Yes in step S110 ), sends the CPU 2 in step S120 to the control circuit 46 of the camera module 30 a communication request for the respective cameras 32w , 32n and 32t and receives from the respective cameras 32w , 32n and 32t a response signal via the control circuit 46 , which shows the operating status of the corresponding camera 32w , 32n and 32t indicates to check whether the corresponding cameras 32w , 32n and 32t operate.

[0147] If it is determined that the ECUs 24 , 26 and 28 perform the autonomous driving of the vehicle V (Yes in step S110 ), gets the CPU 2 also from the respective cameras 32w , 32n and 32t via the control circuit 46 an image data part, i.e. a frame image, obtained from the corresponding camera 32w , 32n and 32t is added to step in S130 to check whether the image data part by the corresponding camera 32w , 32n and 32t is obtained is a normally captured image data piece.

[0148] Note that the CPU 2 the operation in step S120 and operation in step S130 can perform in random order or operation in step S120 and operation in step S130 can perform in parallel.

[0149] In addition, the CPU can 2 the response signal from the respective cameras 32w , 32n and 32t received directly and / or the image data part from the respective cameras 32w , 32n and 32t received directly.

[0150] After the operational verification of the respective cameras 32w , 32n , 32t in step S120 and the image data check of the respective cameras 32w , 32n , 32t in step S130 determines the CPU 2 in step S140 based on the verification results in the steps S120 and S130 whether the narrow-angle camera 32n has a malfunction.

[0151] In particular, the CPU determines 2 that the narrow-angle camera 32n malfunctions when due to, for example, no response signal from the camera 32n is returned, it is determined that the verification result in step S120 represents that camera 32n is not operated. In addition, the CPU determines 2 that the narrow-angle camera 32n malfunctions when it is determined that the check result in step S130 represents that part of the image data sent by the camera 32n is obtained due to, for example, overexposure of the camera 32n , an underexposure of the camera 32w , damage to the lens 33n and / or an order on the lens 33n is an abnormally captured image. Note that the CPU 2 on the basis of, for example, a pixel value, i.e. a luminance value, of a respective pixel of the imaging device 35n , which represents the intensity of the corresponding received light component, can determine that the image data part sent by the camera 32w is obtained is an abnormally captured image.

[0152] If it is determined that the camera 32n malfunctions (Yes in step S140 ), gives the CPU 2 in step S150 to a driver of the vehicle V via the output unit 17 a message indicating that the camera 32w has a malfunction, visibly and / or audibly, with which the driver of the vehicle V is forced to interrupt the autonomous driving of the vehicle V.

[0153] That is, the operation in step S150 prompts a driver of the vehicle V, the ECU 10 using the input unit 18 command to change the operating mode of the vehicle V from the autonomous driving mode to the manual driving mode, thereby interrupting the autonomous driving of the vehicle V.

[0154] Alternatively, the CPU 2 in step S150 perform an emergency braking control task to forcibly stop the vehicle V when, for example, a driver of the vehicle V does not start running the vehicle V in the manual running mode. For example, the emergency brake control task is designed to cause the ECUs 22 , 24 and 26 safely move the vehicle V to a safe place while outputting braking force to decelerate the vehicle V, thereby automatically parking the vehicle V at the safe place.

[0155] That is, the exemplary embodiment prohibits execution of the autonomous driving control task by the drive control unit 6 , whereby the autonomous running of the vehicle V is suspended when it is determined that there is a malfunction or failure in the narrow-angle camera 3nw, so that it is difficult to obtain frame images in the central area around the vehicle V used for autonomous Driving the vehicle V are required. This therefore ensures the safety of the vehicle V.

[0156] On the other hand, if it is determined that the camera 32n has no malfunction (No in step S140 ), determines the CPU 2 in step S160 based on the verification results in the steps S120 and S130 whether the telephoto camera 32t has a malfunction.

[0157] In particular, the CPU determines 2 that the telephoto camera 32t malfunctions when it is determined that the check result in step S120 represents that camera 32t is not operated due to, for example, no response signal from the camera 32tis returned. In addition, the CPU determines 2 that the telephoto camera 32t malfunctions when it is determined that the check result in step S130 represents that part of the image data sent by the camera 32t is obtained due to, for example, overexposure of the camera 32t , an underexposure of the camera 32t , damage to the lens 33t and / or dirt on the lens 33t is an abnormally captured image. Note that the CPU 2 based on, for example, a pixel value of a respective pixel of the imaging device 35t , which represents the intensity of the corresponding received light component, can determine that the image data part sent by the camera 32t is obtained is an abnormally captured image.

[0158] If it is determined that the camera 32t malfunctions (Yes in step S160 ), decreases the CPU 2 in step S170 lowers the speed limit in the autonomous driving conditions by a predetermined speed to correspondingly limit the speed of the vehicle V during the autonomous driving of the vehicle V, i.e., to limit the autonomous driving conditions, i.e., to make them narrower.

[0159] In addition, the CPU 2 in step S170 to a driver of the vehicle V via the output unit 17 visibly and / or audibly issues a message indicating at least information about the upper speed limit of the vehicle V being limited or the changed upper speed limit. In addition, the CPU can 2 in S170 to a driver of the vehicle V via the output unit 17 visually and / or audibly issue a message indicating that the camera 32t has a malfunction.

[0160] The reason why the speed limit is reduced when the telephoto camera 32t has a malfunction is that the malfunction of the telephoto camera 32t It can complicate getting closer and middle images from the wide and narrow angle cameras 32w and 32n be taken to recognize objects that exist in the relatively more distant area that is captured by the telephoto camera 32t is covered.

[0161] That is, even if objects present in the relatively nearer area and the middle area are recognized based on the nearer and middle images, an increase in the speed of the vehicle V may result in delayed recognition of objects present in the relatively farther area are present, resulting in sudden braking and / or sudden steering of the vehicle V.

[0162] In view of this, restricting the speed limit in step prevents S170 an unstable running of the vehicle V in the autonomous running mode.

[0163] On the other hand, if it is determined that the camera 32t has no malfunction (No in step 160 ), determines the CPU 2 in step S180 based on the verification results of the step S120 and S130 whether the wide-angle camera 32w has a malfunction.

[0164] In particular, the CPU determines 2 that the wide-angle camera 32w malfunctions when it is determined that the check result in step S120 represents that camera 32w is not operated due to, for example, no response signal from the camera 32w is returned. In addition, the CPU determines 2 that the wide-angle camera 32w malfunctions when it is determined that the check result in step S130 represents that part of the image data sent by the camera 32w is obtained due to, for example, overexposure of the camera 32w , an underexposure of the camera 32w , damage to the lens 33w and / or deposits on the lens 33w is an abnormally captured image. Note that the CPU 2based on, for example, a pixel value of a respective pixel of the imaging device 35w , which represents the intensity of the corresponding received light component, can determine that the image data part sent by the camera 32w is obtained is an abnormally captured image.

[0165] If it is determined that the camera 32w malfunctions (Yes in step 180 ), locks the CPU 2 in step S190 autonomously steering the vehicle V at each traffic intersection on the planned road to correspondingly limit, i.e. narrow, an autonomous travel execution condition included in the autonomous travel conditions.

[0166] In addition, the CPU 2 in step S190 to a driver of the vehicle V via the output unit 17 visibly and / or audibly issues a message indicating at least information about non-execution of automatic steering in a respective traffic intersection. In addition, the CPU can 2 in step S190 to a driver of the vehicle V via the output unit 17 visually and / or audibly issue a message indicating that the camera 32w has a malfunction.

[0167] That is, the malfunction of the wide-angle camera 32w can complicate it from the more distant and medium-sized images captured by the wide- and narrow-angle cameras 32t and 32n are taken to recognize objects present in the relatively nearer area captured by the wide-angle camera 32w is covered. Therefore, this would lower the safety of the autonomous driving of the vehicle V if an autonomous left turn or an autonomous right turning of the vehicle V based on the autonomous driving of the vehicle V was performed in a traffic intersection while there was a malfunction in the wide-angle camera 32w are.

[0168] In view of this, blocking autonomous steering of the vehicle V in each traffic intersection and informing a driver of the vehicle V of prohibition of autonomous steering in each traffic intersection in step S190 that a driver of the vehicle V drives the vehicle V in the manual driving mode, thus ensuring the safety of the vehicle V.

[0169] Note that the CPU 2 limit, i.e. narrow, the autonomous driving conditions to allow the vehicle V to go straight in each traffic intersection while the speed of the vehicle V is lower than the speed limit set in step S19 is set is set, for example, when a driver of the vehicle V does not start running the vehicle V in the manual running mode.

[0170] On the other hand, if it is determined that the camera 32w has no malfunction (No in step S180 ), serves the CPU 2 in step S200 as the drive control unit 6 to continuously perform the autonomous driving of the vehicle, ending a current cycle of the camera monitoring routine.

[0171] Note that the CPU 2 the operation in step S140 and operation in step S160 can be performed in random order.

[0172] As described above is the camera surveillance unit 8 the ECU 10 designed according to the exemplary embodiment, (1) to check that at least one from the wide-angle camera 32w , the narrow-angle camera 32n and the telephoto camera 32t has a malfunction (2) to limit the speed of the vehicle V as one of the autonomous running conditions required for the autonomous running of the vehicle V when it is determined that the telephoto camera 32t malfunctions to continuously perform the autonomous driving of the vehicle V, (3) prohibit autonomous driving of the vehicle V in each traffic intersection as one of the autonomous driving conditions required for the autonomous driving of the vehicle V when it is determined that the wide-angle camera 32n malfunctions to limit autonomous driving tasks executable in each traffic intersection while the autonomous driving of the vehicle V is continuously performed.

[0173] This therefore enables the drive control unit 6 the ECU 10 to continuously perform the autonomous driving of the vehicle V while limiting the autonomous driving conditions even when monitored by the camera monitoring unit 8 it is determined that the telephoto camera 32t or the wide-angle camera 32w has a malfunction.

[0174] Accordingly, the ECU 10 of the exemplary embodiment designed to eliminate the need to interrupt, i.e. suspend, the autonomous driving of the vehicle V, and to change the operation mode of the vehicle V from the autonomous driving mode to the manual driving mode, even if at least the telephoto camera 32t or the wide-angle camera 32w has a malfunction. That is, this configuration of the ECU 10 allows autonomous driving of the vehicle V to be continuously performed based on the limited autonomous driving conditions as long as at least the narrow-angle camera 32n is operated normally. This therefore creates an autonomous driving system 1 , which is more suitable for the vehicle V to travel.

[0175] The present invention is not limited to the exemplary embodiment described above, and can be modified variously, for example, as follows.

[0176] The camera module 30 of the exemplary embodiment consists of the three cameras, i.e. the wide-angle camera 32w , the narrow-angle camera 32n and the telephoto camera 32t , but the present invention is not limited thereto.

[0177] In particular, the camera module 30 consist of a first camera containing a normal lens or a standard lens and a second camera containing a telephoto lens whose angular field of view is narrower than an angular field of view of the normal lens. Note that the normal or standard lens is defined as a lens having a focal length equal to or closer to the diagonal of the light-receiving area of ​​a corresponding imaging device.

[0178] In addition, the camera module 30 consists of four cameras each capturing an upper part, a lower part, a right part, and a left part of a front view, i.e., a front imaging area, of the vehicle V .

[0179] That is, the camera module 30 can contain multiple cameras, each with different imaging areas in the vicinity of the vehicle V and can be designed such that images recorded by the respective different imaging areas enable detection of one or more objects that are present in at least one of the imaging areas. At this time, if some cameras malfunction, the camera surveillance unit 6 be designed to limit autonomous driving conditions. This enables the drive control unit 6 to carry out the autonomous driving of the vehicle V continuously.

[0180] At least one of the autonomous driving conditions set by the ECU 10 can be limited by the ECU 10 be selected depending on which of the cameras is malfunctioning.

[0181] For example, assume that the camera module 30 consists of four cameras each capturing an upper part, a lower part, a right part, and a left part of a front view, i.e., a front imaging area, of the vehicle V in the traveling direction. In this case, if the camera for recording the right part of the front view is malfunctioning, the ECU 10be configured to limit a current lane change of the vehicle V to a right lane adjacent to the current lane and / or a right turn of the vehicle V.

[0182] The imaging areas of multiple cameras may be configured to at least partially overlap as described in the exemplary embodiment. This configuration prevents the ECU 10 misses an object even if the object has moved across at least one of the boundaries between the overlapping imaging areas.

[0183] The cameras 32w , 32n and 32t that in the camera body 40 are installed can be mounted directly on the vehicle V.

[0184] The CPU 2 the ECU 10 In the exemplary embodiment, the at least one control program executes to perform the functions as a drive control unit 6 and functions as a camera surveillance unit 8 to achieve.

[0185] In contrast, the control circuit 46 of the camera module 30 be designed, the functions of the camera surveillance unit 8 to achieve, and the ECU 10 can be designed, the functions of the drive control unit 6 to achieve. If with this modification the control circuit 46 of the camera module 30 consists of a microcomputer, the microcomputer of the control circuit can execute at least one control program stored therein, thereby to in figure 5 camera surveillance routine shown according to the at least one control program as functions of the camera surveillance unit 8 to execute. That is, the control circuit 46 of the camera module 30 can, for example, as a camera surveillance unit 8 to serve.

[0186] I.e. at least a part of all functions that are carried out by the ECU 10 provided may be achieved by at least one processor; the at least one processor may consist of (1) the combination of at least one programmed processing unit, i.e. at least one programmed logic circuit, and at least one memory containing software that causes the at least one programmed logic circuit to achieve all functions, (2) at least one hardwired logic circuit that achieves all functions, (3) at least one hybrid of hardwired logic and programmed logic that achieves all functions.

[0187] While the exemplary embodiment of the present invention has been described above, the present invention is not limited to the embodiment and modifications thereof described herein, but includes any and all embodiments that include modifications, omissions, combinations (e.g., aspects across different embodiments), adaptations, and / or have alternatives as will be apparent to those skilled in the art based on the present invention within the scope of the present invention.

[0188] For example, each of the technical features described in the embodiment and modifications thereof can be replaced with a known structure that has the same function as the corresponding technical feature. Each of the technical features described in the embodiment and modifications thereof can also be combined with at least one other technical feature. In addition, at least one of the technical features described in the embodiment and modifications thereof may be omitted if the at least one technical feature has not been described as an essential element in the present specification.

[0189] The functions of the drive control unit and the camera monitoring unit can be realized by various embodiments; the various embodiments include autonomous driving control ECUs, camera modules, programs for a computer to serve as the functions, storage media such as non-volatile media storing the programs, and autonomous driving control methods.

Claims

[1] Autonomous driving control device (1) that can be installed in a vehicle, comprising: at least first and second cameras (32t, 32n, 32w) designed to capture at least first and second images of at least first and second directed areas of the vehicle, which are at least partially different from each other; and a vehicle control unit (24, 26, 28) designed to control a driving state of the vehicle, wherein the autonomous driving control device comprises: a drive control unit (6) which is designed to to determine a value for at least one controlled variable for autonomous driving of the vehicle based on the first and second images and an autonomous driving condition; and to output the value of at least one controlled variable to the vehicle control unit in order to cause the vehicle control unit to perform a task for autonomous driving of the vehicle; and a camera surveillance unit (8) designed to to determine whether either of the first and second cameras is malfunctioning; and If it is determined that one of the first and second cameras is malfunctioning, the autonomous driving condition is to be limited based on one of the first and second directed areas that corresponds to the one of the first and second cameras that is malfunctioning. wherein the drive control unit is designed to determine the value of the at least one controlled variable according to the limited autonomous driving condition in order to cause the vehicle control unit to continuously perform the task of autonomous driving of the vehicle according to the determined value of the at least one controlled variable. [2] Autonomous driving control device according to claim 1, wherein the first camera is a telephoto camera whose first directional area is farther away from the vehicle than the second directional area of ​​the second camera; the autonomous driving condition includes a vehicle speed; and The camera monitoring unit is designed to limit the vehicle's speed if it is determined that the first camera is malfunctioning. [3] Autonomous driving control device according to claim 1 or 2, wherein the first camera has a predetermined first angular field of view; the second camera is a wide-angle camera that has a predetermined second angular field of view, as it is wider than the first angular field of view; the autonomous driving condition includes an execution condition for autonomous steering of the vehicle; and The camera monitoring unit is designed to limit the operating condition of autonomous vehicle steering if it is determined that the second camera is malfunctioning. [4] Autonomous driving control device according to claim 1, wherein the at least first and second cameras (32t, 32n, 32w) have at least first, second and third cameras; the first camera is a telephoto camera having a predetermined first angular field of view, wherein the first directed area of ​​the first camera is farther from the vehicle than the second directed area of ​​the second camera; the second camera is a wide-angle camera that has a predetermined second angular field of view that is wider than the first angular field of view; the third camera is a narrow-angle camera which features: a predetermined third directed area that is closer to the vehicle than the first directed area and farther away from the vehicle than the second directed area; and a predetermined third angular field of view, which is wider than the first angular field of view and narrower than the second angular field of view; and The camera surveillance unit is designed to perform at least one of the following: a first task to cause a driver of the vehicle to interrupt the task of autonomous driving of the vehicle if it is determined that the third camera is malfunctioning; and a second task to force the interruption of the autonomous driving task of the vehicle if it is determined that the third camera is malfunctioning. [5] Autonomous driving control device according to claim 4, wherein the first, second and third cameras are aligned relative to each other in a vertical direction of the vehicle. [6] Autonomous driving control device according to claim 4 or 5, wherein the first, second and third directed areas are arranged such that they overlap at least partially in a vertical direction of the vehicle. [7] Program product for a vehicle that contains: at least first and second cameras (32t, 32n, 32w) designed to capture at least each first and second images of at least first and second directed areas of the vehicle that are at least partially different from each other; a vehicle control unit (24, 26, 28) designed to control a driving state of the vehicle; and a drive control unit (6) which is designed to to determine a value for at least one controlled variable for autonomous driving of the vehicle based on the first and second images and an autonomous driving condition; and to output the value of at least one controlled variable to the vehicle control unit in order to cause the vehicle control unit to perform a task of autonomous driving of the vehicle, the computer program product exhibits: a non-volatile, computer-readable storage medium; and a set of computer program instructions stored in the computer-readable storage medium, wherein the instructions cause a computer to perform the following actions: a first step in determining whether either of the first and second cameras is malfunctioning; and a second step to, if it is determined that one of the first and second cameras is malfunctioning, limiting the autonomous driving condition based on one of the first and second directed areas that corresponds to the one of the first and second cameras that is malfunctioning.

Images