Driving support device

Abstract

To provide a driving support device capable of displaying a solid without being distorted or extended with respect to an original shape even when a panoramic image and a bird's-eye view image obtained by viewed from an arbitrary virtual viewpoint are to be displayed.SOLUTION: A driving support device performs: cumulatively storing photographing images obtained by photographing a peripheral environment by side cameras 8A, 8B arranged in a vehicle by association with vehicle information when the photographing images are photographed; generating a route periphery image 44 indicating the peripheral environment along a route on which the vehicle travels by synthesizing the cumulatively stored photographing images based on the vehicle information; and displaying the route periphery image 44 along the route on which the vehicle currently travels. The side cameras 8A, 8B are arranged to perform photographing by including side areas positioned on side parts of the vehicle. The driving support device extracts only a part including the side areas positioned on the side parts of the vehicle within the photographed photographing images, so as to store the images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle. [Background technology]

[0002] Conventionally, various means have been used as information providing means for providing vehicle occupants with various information for vehicle driving support, such as route guidance and obstacle warnings. For example, such means include display on a liquid crystal display installed in the vehicle and audio output from a speaker. However, there are blind spots around the vehicle that are difficult to see from the driver's position. In particular, when performing special operations such as parking or leaving a parking space, a surrounding image captured by a camera installed in the vehicle has been displayed on the liquid crystal display to allow the driver to understand the situation in such blind spots.

[0003] For example, Japanese Patent Application Laid-Open Publication No. 2012-178639 discloses a technology for providing parking assistance for parking a vehicle into a parking space, in which, among images of the surrounding environment captured by multiple cameras mounted on the exterior wall of the vehicle, an image captured by a camera corresponding to the vehicle's direction of travel is displayed on an LCD display. The cameras mounted on the vehicle use wide-angle lenses to eliminate blind spots and capture a wider range of images, and the captured images are subject to distortion due to the characteristics of the lenses. Displaying the captured images as they are would cause discomfort to the user, so distortion correction is performed to reduce the image distortion before displaying them. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-178639 A (paragraphs 0028-0040, Figure 6) Summary of the Invention [Problem to be solved by the invention]

[0005] Here, when displaying the vehicle's surroundings to the user, rather than simply displaying an image captured by a camera as in Patent Document 1, it is common to combine multiple images and then display a viewpoint-converted image to improve visibility. As a result, it is possible to display, for example, a bird's-eye view image or a bird's-eye view image viewed from an arbitrary virtual viewpoint in the sky, rather than from a camera viewpoint. However, this viewpoint conversion is performed by projecting and pasting captured images containing three-dimensional objects onto a flat projection surface (ground surface) without treating them as three-dimensional objects. Therefore, positions higher than the ground surface are projected farther than they actually are, resulting in three-dimensional objects in the captured image being distorted or stretched compared to their original shapes. As a result, there is a problem in that three-dimensional objects are displayed enlarged or significantly distorted compared to their original shapes. While the technology in Patent Document 1 can correct the distortion of captured images caused by lenses, it cannot eliminate the distortion and stretching that occur during the viewpoint conversion described above.

[0006] The present invention has been made to solve the above-mentioned problems in the prior art, and aims to provide a driving assistance device that can display three-dimensional objects without distortion or stretching from their original shape, even when displaying overhead or bird's-eye views from any virtual viewpoint, by extracting only a portion of the vehicle's vehicle area from the images captured by an imaging device, accumulating and storing them, and then displaying an image of the route surroundings generated by combining the accumulated and stored images. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the driving assistance device of the present invention comprises an image storage means for cumulatively storing images of the surrounding environment taken by an imaging device installed in the vehicle, in association with vehicle information at the time the images were taken; an image synthesis means for generating a route surroundings image showing the surrounding environment along the route traveled by the vehicle by synthesizing the cumulatively stored images based on the vehicle information; and an image display means for displaying the route surroundings image along the route currently traveled by the vehicle, wherein the imaging device is installed so as to capture images including lateral areas located to the sides of the vehicle, and the image storage means extracts and stores only a portion of the images taken by the imaging device that includes the lateral areas. The "captured image of the surrounding environment" may be an image captured by an imaging device such as a camera, or may be an image obtained by processing the captured image. For example, it may be an image obtained by combining images captured by multiple cameras or an image obtained by converting the viewpoint. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above-mentioned configuration, only a portion of the vehicle area located to the side of the vehicle is extracted from the captured images taken by the imaging device and accumulated and stored, and an image of the route surroundings generated by combining the accumulated and stored images is displayed.Therefore, even when displaying an overhead image or a bird's-eye view image seen from any virtual viewpoint, it is possible to display three-dimensional objects without distortion or stretching from their original shape. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a vehicle according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention. [Figure 3] 10 is a flowchart of a route surroundings image generation processing program according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing a range of captured images to be cumulatively stored. [Figure 5]FIG. 10 is a diagram showing a projection surface of captured images to be cumulatively stored. [Figure 6] FIG. 10 is a diagram showing a route surroundings image generated by combining a portion of cumulatively stored captured images. [Figure 7] FIG. 10 is a diagram showing an example of a route surroundings image. [Figure 8] 4 is a flowchart of a parking assistance processing program according to the present embodiment. [Figure 9] FIG. 10 is a diagram showing a route surroundings image displayed on a liquid crystal display. [Figure 10] FIG. 10 is a diagram illustrating a modified example. [Figure 11] FIG. 10 is a diagram illustrating a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] A detailed description will be given below of a specific embodiment of a driving assistance device according to the present invention with reference to the drawings. First, a vehicle 2 equipped with a driving assistance device 1 according to this embodiment will be described below. Fig. 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as a drive source (hybrid automobile). Furthermore, the vehicle type is not limited, and it may be a standard car, or a large commercial truck, bus, construction machinery, etc. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the driving operation of the user, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the driving operation of the user.

[0013] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which autonomous driving assistance is performed includes all road sections, including general roads and highways, as well as parking lots, and is performed only when the user selects to perform autonomous driving assistance (for example, turns on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is traveling to a parking space when parking (i.e., parking assistance).

[0014] In the vehicle control in the automated driving assistance of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles are detected as needed, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels along the generated travel path at a speed in accordance with the generated speed plan. In particular, when performing parking assistance, the system uses detection results from sensors and cameras to check the parking space where the vehicle is to park and its surrounding conditions, calculates a parking path to the parking space, and automatically controls the vehicle to enter the parking space along the calculated parking path and complete parking. However, only the steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, only guidance on the parking path to the parking space or guidance on vehicle operation may be provided, and the user may manually park the vehicle into the parking space. Furthermore, the automated driving assistance displays on an in-vehicle display a route surrounding image, which is an image of the actual scenery along the currently traveling route, generated from scenery (actual scenery) around the vehicle previously captured by a camera installed in the vehicle.

[0015] 1, the vehicle 2 has an operation unit 3 that accepts operations from the occupant, a liquid crystal display 4 that displays to the occupant an image of the route surroundings (described later) and other information related to driving assistance, a speaker 5 that outputs audio guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the area around the vehicle, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving assistance ECU (electronic control unit) 10 that performs various calculation processes based on input information. The driving assistance ECU 10 and other components are collectively referred to as a driving assistance device 1.

[0016] Each component of the vehicle 2 will be described below. First, the operation unit 3 is arranged, for example, on the front of the handle (also called the steering wheel), and includes operation buttons and the like that are operated when starting automatic driving assistance. By operating the operation unit 3, the user can switch between manual driving, in which the vehicle travels based on the user's driving operation, and automatic driving assistance, in which the vehicle travels automatically without the user's driving operation. The operation unit 3 may have a touch panel provided on the front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0017] The LCD display 4 is mounted on the instrument panel of the vehicle 2 and displays a route periphery image, which is an image of the actual scenery along the route the vehicle is currently traveling. The route periphery image is generated by synthesizing portions of images previously captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, and is a real-world image (so-called street view) showing the surrounding environment along the route traveled by the vehicle. For example, if the vehicle is traveling along the same route as a route the vehicle has previously traveled, it is possible to display a route periphery image along the route, including not only the route portion already traveled but also the route portion to be traveled in the future. On the other hand, if the vehicle is traveling along a route the vehicle has not traveled before, it is possible to display a route periphery image along the route in the opposite direction to the vehicle's current position (e.g., backward if moving forward, or forward if moving backward), i.e., only the route portion already traveled. Furthermore, if there is a warning object, such as a pedestrian, near the vehicle 2, a warning image indicating the presence of the warning object may be displayed at the position of the warning object in the route periphery image. Furthermore, when providing parking assistance, it is possible to display candidate parking spaces and the parking trajectory to the parking spaces within the route surroundings image displayed on the liquid crystal display 4. The liquid crystal display 4 may also be used as a navigation device.

[0018] The speaker 5 is mounted on the instrument panel of the vehicle 2 and outputs voice guidance and warning sounds related to driving assistance. The speaker 5 may also be used for a navigation device.

[0019] The forward camera 6 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0020] The rear camera 7 is an imaging device having a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with the optical axis facing toward the rear of the vehicle.

[0021] Furthermore, the side cameras 8A and 8B are also imaging devices having cameras using solid-state imaging elements such as CCDs, and are installed, for example, on the left and right side mirrors of the vehicle 2 with their optical axes facing the sides of the vehicle. In particular, the imaging areas of the side cameras 8A and 8B include areas that include lateral areas located to the sides of the vehicle 2.

[0022] The driving assistance ECU 10 accumulates and stores images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, particularly some of the images captured by the side cameras 8A and 8B while the vehicle is traveling, along with vehicle information (such as vehicle speed and vehicle position coordinates) at the time of capture, and generates a route surroundings image showing the surrounding environment along the route traveled by the vehicle by combining the accumulated and stored images. Furthermore, during autonomous driving assistance, the driving assistance ECU 10 performs image recognition processing on the captured images to detect lane lines, parking frame lines, and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and performs autonomous driving assistance based on the detection results. In particular, when performing parking assistance, the system also uses the obstacle detection results from the cameras to identify parking spaces and check the surrounding conditions.

[0023] Meanwhile, the ultrasonic sensors 9A-9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle 2. They transmit ultrasonic waves as search waves around the vehicle 2 and receive reflected waves from objects around the vehicle, thereby detecting the objects that reflect the search waves. Specifically, they are a type of distance measurement sensor that can measure the distance (measured distance) to the object that reflected the search wave by measuring the time from transmission to reception. The ultrasonic sensors 9A-9L are also configured to generate output signals (including the distance to the detected object) corresponding to the reception results of the received waves and output them to the control unit. Examples of objects that can be detected by the ultrasonic sensors 9A-9L include obstacles that the vehicle 2 must avoid when traveling, such as people, bicycles, other vehicles, and walls, as well as obstacles that form parking spaces. Instead of ultrasonic sensors, millimeter-wave sensors or laser sensors may be used as distance measurement sensors.

[0024] The installation position and installation direction of each ultrasonic sensor 9A-9L can be set as appropriate. In this embodiment, to detect objects in all directions (forward, backward, left, and right) of the vehicle 2, for example, ultrasonic sensors 9A-9D are installed on the front of the vehicle 2 facing the vehicle's traveling direction so that the transmission direction of the search wave is forward. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the transmission direction of the search wave is to the left of the vehicle's traveling direction. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the transmission direction of the search wave is to the right of the vehicle's traveling direction. Ultrasonic sensors 9I-9L are installed on the rear of the vehicle 2 facing the opposite direction to the vehicle's traveling direction so that the transmission direction of the search wave is to the rear of the vehicle. The ultrasonic sensors 9A-9L are all approximately the same height from the ground surface.

[0025] In this embodiment, among the ultrasonic sensors 9A-9L, the ultrasonic sensors 9A-9D on the front of the vehicle 2 and the ultrasonic sensors 9I-9L on the rear of the vehicle 2 are particularly installed in positions where they can receive reflected waves from adjacent sensors as indirect waves, so that by receiving direct and indirect waves as received waves, it is possible to determine not only the distance to an object but also the specific position of the object (its relative position to the vehicle) using triangulation. The ultrasonic sensors 9E-9H on the sides are installed at a distance from each other and cannot receive indirect waves, but as the vehicle moves, it is also possible to determine the specific position of the object (its relative position to the vehicle) by triangulation using the measured distances at the previous and current positions and the distance traveled between them.

[0026] Meanwhile, the driving assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it constantly detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, including steering, drive source, and braking, so that the vehicle travels along the generated travel path at a speed according to the generated speed plan. In particular, when providing parking assistance, it uses the detection results of the front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A-9L to confirm the parking space and its surrounding conditions, calculates a parking path to the parking space, and controls the vehicle to enter the parking space along the calculated parking path and complete parking. In addition, it accumulates and stores some of the captured images taken by the side cameras 8A and 8B while the vehicle is traveling, along with vehicle information (vehicle position coordinates, orientation, and direction of travel). The accumulated and stored images are combined to generate a route surrounding image showing the surrounding environment along the route traveled by the vehicle. These generated route surrounding images are also displayed during autonomous driving assistance. The driving assistance ECU 10 is connected to the operation unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L described above via an in-vehicle network such as a CAN. The driving assistance ECU 10 is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as to a navigation device or other in-vehicle device. The detailed configuration of the driving assistance ECU 10 will be described later.

[0027] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 1, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0028] Next, a detailed description will be given of the driving assistance ECU 10 in particular of the driving assistance device 1 provided in the vehicle 2. Fig. 2 is a block diagram showing the configuration of the driving assistance device 1 according to this embodiment.

[0029] As shown in FIG. 2, the driving assistance ECU (electronic control unit) 10 is an electronic control unit that controls the entire driving assistance device 1. It includes a CPU 31 as a calculation device and a control device, a RAM 32 that serves as a working memory for the CPU 31 to perform various calculation processes and stores driving trajectory data and other information used when the driving trajectory is calculated, a ROM 33 that stores control programs as well as a route surroundings image generation processing program (see FIG. 3) and a parking assistance processing program (see FIG. 8), and a flash memory 34 that stores programs read from the ROM 33. The driving assistance ECU 10 also includes various processing algorithms. For example, the image storage means cumulatively stores captured images of the surrounding environment captured by a camera installed in the vehicle, in association with vehicle information at the time the captured images were captured. The image synthesis means generates a route surroundings image showing the surroundings along the route traveled by the vehicle by synthesizing the cumulatively stored captured images based on the vehicle information. The image display means displays a route surroundings image along the route the vehicle is currently traveling.

[0030] The driving assistance ECU 10 is also connected to various sensors 37 for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as various vehicle drive units 38, such as the steering, brakes, accelerator, and transmission, and performs automatic driving assistance for the vehicle 2 by detecting the current vehicle behavior based on the detection results of these sensors 37 and controlling the various drive units 38. Specific details of the automatic driving assistance include, for example, detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and controlling the vehicle, such as the steering, drive source, and brakes, so that the vehicle travels along a generated travel trajectory at a speed in accordance with a speed plan that is also generated. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes based on manual operation.

[0031] The flash memory 34 also includes a vehicle information DB 35 and a captured image DB 36. The vehicle information DB 35 stores various information related to the vehicle 2. For example, the installation positions (height from the ground and left / right positions) of the cameras and ultrasonic sensors 9A to 9L installed on the vehicle 2, the detection axes (optical axes for cameras), overall length, vehicle width, wheelbase, minimum turning radius, etc. This information is input in advance by the occupants or a person from the vehicle manufacturer.

[0032] On the other hand, the captured image DB 36 is a storage means for accumulating and storing some of the captured images taken by the side cameras 8A, 8B while the vehicle is traveling together with the vehicle information at the time of capturing the images. The vehicle information includes the position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle when the captured images stored in the captured image DB 36 were captured, and this vehicle information is estimated from, for example, the vehicle speed, wheel speed, wheel speed pulse integration value, steering angle, etc. acquired by the various sensors 37.

[0033] Next, a route periphery image generation processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to FIG. 3. FIG. 3 is a flowchart of the route periphery image generation processing program according to this embodiment. The route periphery image generation processing program is executed after the ACC (accessory power supply) of the vehicle 2 is turned on. The program accumulates and stores some of the captured images taken by the cameras provided in the vehicle 2, and generates a route periphery image showing the surrounding environment along the route traveled by the vehicle by combining the stored captured images. However, if the target for generating the route periphery image is limited to a parking lot, the route periphery image generation processing program may be executed when the vehicle enters the parking lot. Alternatively, if the period for generating the route periphery image is limited to when the vehicle is traveling with autonomous driving assistance, the route periphery image generation processing program may be executed when the autonomous driving assistance is started. The programs shown in the flowcharts in FIGS. 3 and 8 below are stored in the RAM 32 or ROM 33 provided in the driving assistance device 1, and are executed by the CPU 31.

[0034] The following processing is performed for each image captured by the side cameras 8A, 8B. For example, since the frame rate of the side cameras 8A, 8B in this embodiment is 30 fps (capturing 30 images per second), the following processing is performed for the most recently captured image every 33 ms. However, the program execution interval does not necessarily have to be 33 ms; for example, if processing is to be performed on 10 images at a time, it may be performed every 330 ms.

[0035] First, in step (hereinafter abbreviated as S) 1, the CPU 31 acquires the most recently captured images by the side cameras 8A and 8B. As described above, in this embodiment, the side cameras 8A and 8B are attached to the left and right side mirrors of the vehicle 2 and are installed facing the sides of the vehicle. Therefore, as shown in FIG. 4, captured images of the surrounding environment on the sides of the vehicle 2 are acquired.

[0036] Next, in S2, the CPU 31 extracts (trims) a portion of the captured image captured by the side cameras 8A and 8B acquired in S1. Here, the range from which the captured image is extracted is a portion of the lateral region located to the side of the vehicle. For example, when extracting the captured image from the images captured by the side cameras 8A and 8B, extraction ranges 41A and 41B shown in FIG. 4 are used. Extraction range 41A and extraction range 41B are symmetrical and have the same shape. Taking extraction range 41A as an example, it is a long rectangular region perpendicular to the axle extending in the fore-and-aft direction of the vehicle within the captured image captured by side camera 8A. While FIG. 4 shows the direction perpendicular to the side of the vehicle from the position of side camera 8A, this range is not limited to this. More specifically, the horizontal range of the rectangular shape begins as close as possible to the host vehicle and is set to a range that includes at least the aisle along which the host vehicle will travel and the parked vehicles on either side when the host vehicle travels along the aisle in the parking lot. For example, length a is set to approximately 5 m. It is desirable that extraction range 41A include as much of the vicinity of the host vehicle as possible, but it is also desirable that the range be such that the shadow of the host vehicle does not appear. It is also desirable that if a is made longer and more distant areas are included in extraction range 41A, the area that can be displayed in the route surroundings image described below will be wider, but the further away from the host vehicle, the higher the possibility that the image will be stretched or distorted when projected onto the projection surface described below. Therefore, it is desirable to make a as short as possible within a range that includes the road surface on which the host vehicle is traveling and parked vehicles on both sides.

[0037] On the other hand, the longitudinal width of the rectangular shape of extraction range 41A is set to a width that will not result in gaps between the extracted captured images when they are joined together along the traveling direction (S6) as described below. For example, assuming imaging in a parking lot, if the vehicle's traveling speed is assumed to be 20 km / h, the distance the vehicle travels between frames (33 ms) is 0.183 m, so if b = 0.2 m or so, no gaps will occur in the images. Note that if b is made longer, no gaps will occur in the images even when the vehicle is traveling at high speeds, but the further away from the optical axis direction it is, the greater the possibility that the image will be stretched or distorted when projected onto the projection surface described below. Therefore, it is desirable to make b as short as possible within a range that does not result in gaps in the images.

[0038] However, the above values ​​of a and b are merely examples, and may be changed depending on the vehicle speed and the type of road on which the vehicle is traveling. For example, the values ​​of a and b may be set to small values ​​in a parking lot where the vehicle is expected to travel at a low speed, and to larger values ​​on a public road where the vehicle is expected to travel at a high speed. Furthermore, in this embodiment, extraction range 41A and extraction range 41B are bilaterally symmetrical and have the same shape, but the ranges of extraction range 41A and extraction range 41B may be different. For example, in countries where driving on the left side is common, it is expected that parked vehicles are parked on the shoulder to the left of vehicle 2 traveling on a public road. Therefore, left extraction range 41A may be set wider than right extraction range 41B to include these parked vehicles in the extraction range.

[0039] Furthermore, when extracting the captured images of extraction range 41A and extraction range 41B in S2, extraction range 41A and extraction range 41B are extracted from an image obtained by projecting (converting) the captured images onto projection surfaces 42A and 42B, which are curved surfaces that become increasingly farther from the ground surface as the distance from the vehicle increases, as shown in Fig. 5. By curving projection surfaces 42A and 42B in the height direction as shown in Fig. 5, the image extracted in S2 can be an image that shows not only the road surface but also the environment to the side as seen from the vehicle. The curvature angle and height of projection surfaces 42A and 42B can be changed as appropriate.

[0040] On the other hand, projection surfaces 42A and 42B may be the ground surface (horizontal plane) instead of curved surfaces. Furthermore, a bird's-eye image may be generated by converting the captured image projected onto the ground surface into an image seen from a virtual viewpoint looking down vertically from above vehicle 2, and then bird's-eye images corresponding to extraction ranges 41A and 41B shown in Fig. 4 may be extracted. In particular, trimming the image into a rectangular shape so as to include a direction perpendicular to the axle in the fore-and-aft direction of the vehicle at the installation position of the camera makes it possible to trim an image without deformation or distortion.

[0041] Next, in S3, the CPU 31 acquires various detection values ​​indicating vehicle behavior, such as vehicle speed, wheel speed, wheel speed pulse integration value, and steering angle, from various sensors 37 provided in the vehicle and connected via CAN.

[0042] Next, in S4, the CPU 31 estimates the current position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle based on the detection values ​​acquired in S3. The information estimated in S4 is information that estimates the position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle as vehicle information when the captured image extracted in S2 was captured. The position coordinates and orientation estimated in S4 are relative values ​​with respect to a reference position and orientation. This reference position and orientation are, for example, the position and orientation of the vehicle 2 at the timing when cumulative storage of captured images during this travel began (for example, when the ACC power was turned on, when the vehicle entered a parking lot, when autonomous driving began, etc.).

[0043] Thereafter, in S5, the CPU 31 associates the part of the captured images extracted in S2 with the vehicle information estimated in S4 and cumulatively stores them in the captured image DB 36. Specifically, the part of the captured images captured by the side camera 8A and the part of the captured images captured by the side camera 8B are separated and stored in chronological order (or in an order according to the coordinates of the vehicle at the time of capturing).

[0044] Furthermore, in S6, the CPU 31 generates a route surroundings image along the route traveled by the vehicle by synthesizing the plurality of captured images stored in the captured image DB 36 in S5 based on the associated vehicle information.

[0045] Specifically, as shown in FIG. 6, a portion of the image captured by the side camera 8A (hereinafter referred to as the left captured image 43A) extracted in S2 and a portion of the image captured by the side camera 8B (hereinafter referred to as the right captured image 43B) extracted in S2 are stitched together based on the vehicle's position coordinates, orientation, and traveling direction at the time of capture to generate a route surrounding image 44. If an overlapping area occurs in the left captured image 43A or the right captured image 43B as a result of stitching, the overlapping area is overwritten with the new image. FIG. 7 is an illustration of a route surrounding image 44 generated by stitching together the left captured image 43A and the right captured image 43B. The generated route surrounding image 44 also includes information about the route traveled by the vehicle when the route surrounding image 44 was generated, i.e., information identifying the route traveled by the vehicle when the route surrounding image 44 was generated. The route surrounding image 44 generated in S6 is stored in the flash memory 34 together with the route information.

[0046] It should be noted that a gap the width of the vehicle will be generated between the left captured image 43A and the right captured image 43B, but since the road surface will be present in that gap, an image of the road surface may be generated and interpolated, or an image captured by the front camera 6 or the rear camera 7 may be used for interpolation.

[0047] 6, each time a new left captured image 43A or a new right captured image 43B is acquired, it is spliced ​​into the existing route surrounding image 44 at the time of acquisition to generate the route surrounding image 44. That is, the route surrounding image 44 along the route along which the vehicle is currently traveling is generated in real time by targeting the area opposite to the vehicle's current position (for example, rearward if the vehicle is traveling forward, or forward if the vehicle is traveling backward). However, the route surrounding image 44 may also be generated when the vehicle has finished traveling. For example, the processes from S1 to S5 may be repeated while the vehicle is traveling, and the process of S6 may be executed when parking is completed. In this case, when parking is completed, the left captured image 43A and the right captured image 43B acquired during the traveling up to the completion of this parking are read from the captured image DB 36, and the route surrounding image 44 along the route along which the vehicle traveled during the traveling up to the completion of this parking is generated all at once.

[0048] However, with specifications that generate route periphery images 44 all at once when parking is complete, there is a problem in that when the vehicle is traveling on a new route other than the route on which the vehicle previously traveled and for which the route periphery images 44 were generated, the route periphery images 44 along the route the vehicle is currently traveling cannot be displayed. On the other hand, with specifications that generate the route periphery images 44 along the route the vehicle is currently traveling in real time, targeting the direction opposite to the vehicle's current position (for example, behind if the vehicle is traveling forward, or forward if the vehicle is traveling backward), it is possible to display the route periphery images 44 along the route the vehicle is currently traveling, even when traveling on a new route, although this is limited to the direction opposite to the vehicle's current position. In addition, it is also possible to display the route periphery images 44 that reflect the surrounding conditions (for example, the positions of other vehicles) in real time.

[0049] In addition, when a vehicle travels again along the same route for which a route surrounding image 44 has already been generated, there is basically no need to generate the above-mentioned route surrounding image 44. However, when it is desired to display a route surrounding image 44 that reflects the above-mentioned surrounding conditions in real time, for example, when performing parking assistance, the route surrounding image 44 may be generated again.

[0050] Incidentally, a problem with the prior art is that when an image captured by a camera is displayed as a bird's-eye view or an overhead view by changing the viewpoint, three-dimensional objects in the captured image are displayed distorted or stretched relative to their original shape. However, in this embodiment, only a very narrow range perpendicular to the axle extending in the longitudinal direction of the vehicle is extracted as shown in FIG. 4, and similarly extracted captured images are joined together to generate the route surroundings image 44 shown in FIG. 7, so that even if three-dimensional objects are included in the generated route surroundings image 44, they can be prevented from being distorted or stretched relative to their original shape.

[0051] Next, a parking assistance processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to FIG. 8. FIG. 8 is a flowchart of the parking assistance processing program according to this embodiment. Here, the parking assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned on, and is a program that provides parking assistance, particularly when parking the vehicle, as one type of automatic driving assistance. It is assumed that the parking assistance processing program shown in FIG. 8 and the above-mentioned route periphery image generation processing program (FIG. 3) are executed in parallel. In other words, a route periphery image 44 is also generated for the route traveled by the vehicle while parking assistance is being performed. However, if the vehicle travels again along a route for which a route periphery image 44 has already been generated, the route periphery image generation processing program may be controlled not to be executed.

[0052] First, in S11, the CPU 31 determines whether or not to start parking assistance. In particular, the parking assistance in this embodiment automatically moves the vehicle to a parking space where the vehicle is to be parked, and displays the route surroundings image 44 generated by the route surroundings image generation processing program (FIG. 3).

[0053] For example, parking assistance may be started when the user operates the operation unit 3 and selects to start parking assistance, or parking assistance may be started automatically when it is detected that the vehicle has entered a parking lot, when it is determined that the vehicle has arrived at the set destination, or when the vehicle approaches a parking lot.

[0054] If it is determined that parking assistance should be started (S11: YES), the process proceeds to S12. On the other hand, if it is determined that parking assistance should not be started (S11: NO), the parking assistance processing program is terminated.

[0055] In S12, the CPU 31 acquires various detection values ​​indicating the vehicle behavior, such as vehicle speed, wheel speed, wheel speed pulse integration value, and steering angle, from various sensors 37 equipped on the vehicle connected via CAN, and identifies the current vehicle position coordinates, orientation, and direction of travel (forward / reverse).

[0056] Thereafter, in S13, the CPU 31 renders the route periphery image 44 generated by the route periphery image generation processing program (FIG. 3) on the liquid crystal display 4 in a manner that matches the vehicle's current position, orientation, and traveling direction. That is, the CPU 31 reads the route periphery image 44 corresponding to the coordinates of the vehicle's current position, orientation, and traveling direction from the flash memory 34 and renders it on the liquid crystal display 4. Note that the virtual viewpoint setting (from which viewpoint and in which direction the route periphery image 44 is to be rendered) when rendering the route periphery image 44 can be appropriately set. For example, when the host vehicle travels again along a route for which the host vehicle previously traveled and for which the route periphery image 44 was generated, the route periphery image 44 ahead of the vehicle can also be displayed. Therefore, for example, a virtual viewpoint can be set in the sky above the vehicle's current position in the opposite direction to the traveling direction (for example, behind if the vehicle is moving forward, or ahead if the vehicle is moving backward) to generate a bird's-eye view image looking down diagonally from above in the traveling direction. Alternatively, a bird's-eye view image looking down vertically from above the vehicle's current position can be used. On the other hand, when the vehicle travels on a route other than the one on which the vehicle has traveled in the past and for which the route surrounding image 44 was generated, only the route surrounding image 44 of the portion of the route that has already been traveled in the opposite direction to the vehicle's direction of travel (for example, backward if the vehicle is moving forward, or forward if the vehicle is moving backward) can be displayed.Therefore, for example, a virtual viewpoint can be set in the sky ahead of the vehicle's current position in the direction of travel, and a bird's-eye view image can be created looking diagonally downward from the sky in the opposite direction to the direction of travel.

[0057] When the host vehicle travels again along a route on which the host vehicle previously traveled and for which the route surrounding image 44 was generated, a route image showing the route traveled by the host vehicle when traveled in the past may also be drawn on the route surrounding image 44. The specific trajectory position and shape of the route traveled by the host vehicle when traveled in the past in the route surrounding image 44 can be identified from the vehicle information stored in the captured image DB 36. The vehicle information is information that identifies the position coordinates, orientation, and traveling direction (forward / backward) of the vehicle when the captured image stored in the captured image DB 36 was captured.

[0058] Here, a problem with the prior art is that when an image captured by a camera is displayed as a bird's-eye view or an overhead view by changing the viewpoint, three-dimensional objects in the captured image are displayed distorted or stretched relative to their original shape. However, in this embodiment, as described above, only a very narrow range perpendicular to the axle extending in the longitudinal direction of the vehicle is extracted, so that when the virtual viewpoint is converted to an arbitrary position and displayed as described above, even if three-dimensional objects such as parked vehicles are included in the route surroundings image 44, it is possible to prevent the image from being displayed distorted or stretched relative to its original shape.

[0059] Furthermore, in S14, the CPU 31 superimposes a host vehicle icon image 51 indicating the position and orientation of the host vehicle on the route surroundings image 44 drawn in S13, corresponding to the current position and orientation of the host vehicle, and draws the superimposed image on the liquid crystal display 4. Fig. 9 is a diagram showing the route surroundings image 44 drawn with the host vehicle icon image 51 superimposed. The host vehicle icon image 51 is an image that simulates the appearance of the host vehicle, and drawing the host vehicle icon image 51 enables the user to grasp the environment around the vehicle in association with the current position and orientation of the vehicle. Note that Fig. 9 is an example of the route surroundings image 44 that is displayed when the vehicle is moving forward; for example, when the vehicle is moving backward, the route surroundings image 44 of the range ahead of the vehicle's current position is displayed.

[0060] The route surroundings image 44 drawn on the liquid crystal display 4 in S13 changes its drawing range in accordance with a change in the current position of the vehicle. Also, the display position and orientation of the vehicle icon image 51 drawn on the liquid crystal display 4 in S14 also change in accordance with a change in the current position or direction of the vehicle.

[0061] Thereafter, in S15, CPU 31 identifies a parking space for parking the vehicle. The parking space may be identified by, for example, the user arbitrarily specifying a parking space where the user wishes to park in the route surroundings image 44 displayed on LCD display 4, or the user may register a parking space (e.g., a home garage) in advance and identify the registered parking space as the parking space for parking the vehicle. Alternatively, ultrasonic sensors 9A-9L or a camera may be used to detect vacant parking spaces around the vehicle in real time, and the parking space detected as vacant may be identified as the parking space for parking the vehicle.

[0062] When a vacant parking space is detected using the ultrasonic sensors 9A to 9L or a camera, a parking frame 52 may be displayed in the route surroundings image 44 displayed on the liquid crystal display 4, as shown in Fig. 9. This makes it possible to check the status of vacant parking spaces around the current position of the vehicle. In particular, even if there is a vacant parking space in a position that the vehicle has already passed (a position that is in a blind spot for the occupants), it can be easily checked in the route surroundings image 44.

[0063] Next, in S16, the CPU 31 calculates a parking trajectory, which is a driving trajectory for parking the vehicle in the parking space identified in S15. Taking the calculation of a parking trajectory for parallel parking as an example, the parking trajectory is composed of a combination of a preparation trajectory that moves forward from the current vehicle position to the reverse start position, a first half trajectory that turns from the reverse start position to the steering position, a second half trajectory that turns from the steering position to the target parking position, and a turning trajectory that turns as needed after reaching the target parking position (the turning trajectory is not essential). Then, by driving the vehicle 2 along the parking trajectory, the vehicle 2 can enter the target parking position within the parking space and position the vehicle 2 at the target parking position with the appropriate vehicle orientation.

[0064] When the parking trajectory is calculated, the route surroundings image 44 displayed on the liquid crystal display 4 may display a parking trajectory 53 as shown in FIG.

[0065] Thereafter, in S17, the CPU 31 starts parking assistance according to the parking trajectory calculated in S16. Specifically, the current position of the vehicle is detected at any time, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels at a specified speed along the generated parking trajectory. In addition, the shift position is also automatically switched.

[0066] However, when moving the vehicle along the parking path, only the steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, the vehicle may be moved manually rather than automatically. In this case, parking assistance may be provided by displaying the parking space and parking path on the LCD display 4, and providing voice guidance on when to turn the steering wheel.

[0067] Next, in S18, the CPU 31 determines whether parking of the vehicle is complete. Specifically, when the vehicle is positioned in the target parking position set in the parking space identified in S15 and the shift position of the vehicle is changed to "P," parking of the vehicle is determined to be complete.

[0068] If it is determined that the vehicle has been parked (S18: YES), the parking assistance processing program is terminated. On the other hand, if it is determined that the vehicle has not been parked (S18: NO), the parking assistance continues.

[0069] As explained in detail above, according to the driving assistance device 1 and the computer program executed by the driving assistance device 1 of this embodiment, captured images of the surrounding environment taken by the side cameras 8A, 8B installed in the vehicle are cumulatively stored in association with the vehicle information at the time the captured images were taken (S5), and the cumulatively stored captured images are synthesized based on the vehicle information to generate a route surrounding image 44 showing the surrounding environment along the route traveled by the vehicle (S6), and display the route surrounding image 44 along the route the vehicle is currently traveling (S13).On the other hand, the side cameras 8A, 8B are installed so as to capture images including the lateral areas located to the side of the vehicle, and only a portion of the captured captured images including the lateral areas is extracted and stored, so that even when displaying an overhead image or bird's-eye view image seen from any virtual viewpoint, it is possible to display three-dimensional objects without distortion or stretching from their original shape. Furthermore, from the images captured by the side cameras 8A and 8B, images of long rectangular areas perpendicular to the axles extending in the longitudinal direction of the vehicle are extracted and stored, so even if a three-dimensional object is included in the extracted image, it can be displayed without distortion or stretching from the original shape. In addition, since the captured images are stored with projection surfaces 42A and 42B being curved surfaces that become increasingly farther from the ground surface as they move away from the vehicle, it is possible to display images that show not only the road surface but also the environment to the side as viewed from the vehicle. Furthermore, a route surroundings image along the route the vehicle is currently traveling is generated, targeting the direction opposite to the vehicle's current position, and a route surroundings image along the route the vehicle is currently traveling is displayed, targeting the direction opposite to the vehicle's current position. Therefore, even when traveling on a new route that the vehicle has not traveled before, it is possible to display a route surroundings image along the route the vehicle is currently traveling, although it is limited to the direction opposite to the vehicle's current position. Furthermore, it is possible to display a route surroundings image that also reflects the surrounding conditions (for example, the positions of other vehicles) in real time.

[0070] The present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications are possible within the scope of the present invention. For example, in this embodiment, images of a long rectangular area perpendicular to the axle extending in the fore-and-aft direction of the vehicle are extracted from the images captured by the side cameras 8A and 8B, accumulated and stored, and then synthesized to generate the route surroundings image 44. However, the extracted image area does not necessarily have to be perpendicular to the axle, as long as it is a partial area including a lateral area located to the side of the vehicle. Furthermore, the extracted image area does not necessarily have to be an image captured by the side cameras 8A and 8B, and may be a portion of an image captured by the front camera 6 or the rear camera 7, as long as it is a partial area including a lateral area located to the side of the vehicle. For example, when extraction is performed on the image captured by the front camera 6, extraction ranges 101A and 101B shown in FIG. 10 are used. On the other hand, when extraction is performed on the image captured by the rear camera 7, extraction ranges 102A and 102B are used, as shown in FIG. 11. Note that the front camera 6 and the rear camera 7 use wide-angle lenses with wide angles of view, so that the lateral areas can also be included in the image capture range.

[0071] In this embodiment, it is assumed that the route surroundings image 44 shown in Fig. 9 is displayed on the liquid crystal display 4 while parking assistance by automatic driving assistance is being performed, but it may also be displayed while automatic driving assistance other than parking assistance is being performed. Furthermore, the route surroundings image 44 shown in Fig. 9 may also be displayed on the liquid crystal display 4 while driving manually.

[0072] 9 is displayed on an in-vehicle display, it may also be displayed on a display of a terminal outside the vehicle. This allows a user outside the vehicle to easily grasp the current progress towards parking completion, for example, when performing parking assistance by remote control.

[0073] In this embodiment, the route surroundings image generation processing program (FIG. 3) and the parking assistance processing program (FIG. 8) are configured to be executed by the driving assistance ECU 10 of the driving assistance device 1, but the executing entity can be changed as appropriate. For example, they may be executed by the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle device. [Explanation of symbols]

[0074] 1... driving assistance device, 2... vehicle, 3... operation unit, 4... liquid crystal display, 6... front camera (imaging device), 7... rear camera (imaging device), 8A, 8B... side cameras (imaging devices), 10... driving assistance ECU, 31... CPU, 36... captured image DB, 41A, 41B... extraction range, 42A, 42B... projection surface, 43A... left captured image, 43B... right captured image, 44... route surrounding image

Claims

1. an image storage means for cumulatively storing captured images of the surrounding environment captured by an imaging device installed in the vehicle in association with vehicle information at the time the captured images were captured; an image synthesis means for synthesizing the cumulatively stored captured images based on the vehicle information to generate a route surroundings image showing the surrounding environment along the route traveled by the vehicle; an image display means for displaying a route surrounding image along the route on which the vehicle is currently traveling; the imaging device is installed so as to capture an image including a lateral area located laterally with respect to the vehicle, The image storage means extracts and stores only a portion of the image captured by the imaging device, the portion including the side region.

2. 2. The driving assistance device according to claim 1, wherein the image storage means extracts and stores an image of a long rectangular area perpendicular to an axle extending in the longitudinal direction of the vehicle from the captured image captured by the imaging device.

3. The driving assistance device according to claim 1 , wherein the image storage means stores the captured images with a projection plane that is a curved surface that becomes increasingly farther from the ground surface as the distance from the vehicle increases.

4. the image synthesis means generates the route surroundings image along the route on which the vehicle is currently traveling, targeting a direction opposite to the traveling direction of the current position of the vehicle; 4. The driving assistance device according to claim 1, wherein the image display means displays the route surroundings image along the route on which the vehicle is currently traveling, targeting a direction opposite to the traveling direction of the current position of the vehicle.

Images