Driving support device

Abstract

To provide a driving support device capable of indicating a state in the periphery of a travel route for succeeding traveling without limiting the neighborhood of a vehicle even when performing long-range parking with a long movement to a parking space.SOLUTION: A driving support device performs: cumulatively storing photographing images obtained by photographing a peripheral environment by a front camera 6, a rear camera 7, and side cameras 8A, 8B arranged in a vehicle by association with vehicle information when the photographing images are photographed; generating a route periphery image 48 indicating the peripheral environment along a route on which the vehicle travels by synthesizing the cumulatively stored photographing images based on the vehicle information; acquiring a travel plan route of the vehicle; and displaying the route periphery image 48 along the travel plan route together with an own vehicle icon image 51 indicating a present position of the vehicle.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle. [Background technology]

[0002] Conventionally, various means have been used as information providing means for providing vehicle occupants with various information for vehicle driving support, such as route guidance and obstacle warnings. For example, such means include display on a liquid crystal display installed in the vehicle and audio output from a speaker. However, there are blind spots around the vehicle that are difficult to see from the driver's position. In particular, when performing special operations such as parking or leaving a parking space, a surrounding image captured by a camera installed in the vehicle has been displayed on the liquid crystal display to allow the driver to understand the situation in such blind spots.

[0003] For example, Japanese Patent Application Laid-Open No. 2023-176548 discloses a technology for assisting a vehicle in parking into a parking space, in which multiple cameras installed on the exterior walls of the vehicle each capture images of the surrounding environment, and the captured images are combined to generate a forward direction image showing the direction the vehicle is traveling and an overhead image looking down on the area around the vehicle from above, which are then displayed on an LCD display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-176548 A (paragraphs 0071-0088) Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 has a problem in that the forward direction image and bird's-eye view image displayed can only show the situation in the area near the vehicle. Similarly, the technology disclosed in Patent Document 1 also has a problem in that only the area near the vehicle can be displayed for the vehicle's future path (parking trajectory). While the technology disclosed in Patent Document 1 can provide parking assistance for parking spaces near the vehicle, for example, in long-range parking, where the vehicle moves to a predetermined distant parking space within a large site and parks there, the technology is unable to provide sufficient assistance to the user because it is unable to provide information about the situation around the driving route to the parking space.

[0006] The present invention has been made to solve the above-mentioned problems in the prior art, and aims to provide a driving assistance device that displays an image of the route surroundings generated by combining images taken when the vehicle was traveling in the past, making it possible to show the situation around the route that will be traveled, not just in the vicinity of the vehicle, even when performing long-range parking, which requires a long journey to a parking space, for example. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the driving assistance device of the present invention comprises an image storage means for cumulatively storing captured images of the surrounding environment taken by an imaging device installed in a vehicle, in association with vehicle information at the time the captured images were taken; an image synthesis means for generating a route surroundings image showing the surrounding environment along the route traveled by the vehicle by synthesizing the cumulatively stored captured images based on the vehicle information; and an image display means for displaying the route surroundings image. The "captured image of the surrounding environment" may be an image captured by an imaging device such as a camera, or may be an image obtained by processing the captured image. For example, it may be an image obtained by combining images captured by multiple cameras or an image obtained by converting the viewpoint. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above configuration, by displaying a route surroundings image generated by combining images captured when the vehicle was traveling in the past, it is possible to show the situation around the route along which the vehicle will travel, not just the vicinity of the vehicle, even when performing long-range parking, which requires a long journey to a parking space. Furthermore, by displaying a vehicle image within the route surroundings image, it is possible to easily grasp the current position of the vehicle along the traveling route. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a vehicle according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention. [Figure 3] 10 is a flowchart of a route surroundings image generation processing program according to the present embodiment. [Figure 4] FIG. 10 is a diagram illustrating a method for generating an overhead image. [Figure 5] FIG. 10 is a diagram showing a generated overhead image. [Figure 6] FIG. 10 is a diagram showing the range of overhead images to be cumulatively stored. [Figure 7] FIG. 10 is a diagram showing a route surroundings image generated by combining parts of cumulatively stored bird's-eye images. [Figure 8] FIG. 10 is a diagram showing an example of a route surroundings image. [Figure 9] 4 is a flowchart of a parking assistance processing program according to the present embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a route surroundings image displayed on a liquid crystal display. [Figure 11] FIG. 10 is a diagram showing an example of a route surroundings image displayed on a liquid crystal display. DETAILED DESCRIPTION OF THE INVENTION

[0010] A detailed description will be given below of a specific embodiment of a driving assistance device according to the present invention with reference to the drawings. First, a vehicle 2 equipped with a driving assistance device 1 according to this embodiment will be described below. Fig. 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as a drive source (hybrid automobile). Furthermore, the vehicle type is not limited, and it may be a standard car, or a large commercial truck, bus, construction machinery, etc. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the driving operation of the user, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the driving operation of the user.

[0013] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which autonomous driving assistance is performed includes all road sections, including general roads and highways, as well as parking lots, and is performed only when the user selects to perform autonomous driving assistance (for example, turns on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is traveling to a parking space when parking (i.e., parking assistance).

[0014] In the vehicle control in the automated driving assistance of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles are detected as needed, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels along the generated travel trajectory at a speed according to the generated speed plan. In particular, when performing parking assistance, the system uses detection results from sensors and cameras to check the parking space where the vehicle is to park and its surrounding conditions, calculates a parking trajectory to the parking space, and automatically controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. In addition to the above-mentioned normal parking assistance, the system also supports long-range parking, in which parking is targeted at a distant parking space, such as a home garage or a monthly parking space in a parking lot. In parking assistance for long-range parking, the system automatically controls the vehicle to move to a distant, predetermined parking space and complete parking. However, in the parking assistance, only steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, the system may provide only guidance on the parking trajectory to the parking space or guidance on vehicle operation, and the user may manually perform the parking operation into the parking space. On the other hand, the above-mentioned autonomous driving assistance involves displaying on an in-vehicle display a route surrounding image, which is an image of the actual scenery along the currently traveling route, generated from the scenery (actual scenery) around the vehicle previously captured by a camera installed in the vehicle.

[0015] 1, the vehicle 2 has an operation unit 3 that accepts operations from the occupant, a liquid crystal display 4 that displays to the occupant an image of the route surroundings (described later) and other information related to driving assistance, a speaker 5 that outputs audio guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the area around the vehicle, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving assistance ECU (electronic control unit) 10 that performs various calculation processes based on input information. The driving assistance ECU 10 and other components are collectively referred to as a driving assistance device 1.

[0016] Each component of the vehicle 2 will be described below. First, the operation unit 3 is arranged, for example, on the front of the handle (also called the steering wheel), and includes operation buttons and the like that are operated when starting automatic driving assistance. By operating the operation unit 3, the user can switch between manual driving, in which the vehicle travels based on the user's driving operation, and automatic driving assistance, in which the vehicle travels automatically without the user's driving operation. The operation unit 3 may have a touch panel provided on the front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0017] The LCD display 4 is mounted on the instrument panel of the vehicle 2 and displays a route periphery image, which is an image of the actual scenery along the route the vehicle is currently traveling. The route periphery image is generated by synthesizing portions of images previously captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, and is an actual scene image (so-called street view) showing the surrounding environment along the route traveled by the vehicle. Therefore, if the vehicle is traveling along the same route as a route the vehicle has previously traveled, it is possible to display a route periphery image along the route, including not only the route already traveled but also the route to be traveled. Furthermore, if there is a warning object, such as a pedestrian, near the vehicle 2, a warning image indicating the presence of the warning object may be displayed at the position of the warning object in the route periphery image. Furthermore, when performing parking assistance, it is possible to display candidate parking spaces and the parking trajectory to the parking space in the route periphery image displayed on the LCD display 4. The LCD display 4 may also be used for a navigation device.

[0018] The speaker 5 is mounted on the instrument panel of the vehicle 2 and outputs voice guidance and warning sounds related to driving assistance. The speaker 5 may also be used for a navigation device.

[0019] The forward camera 6 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0020] The rear camera 7 is an imaging device having a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with the optical axis facing toward the rear of the vehicle.

[0021] Furthermore, the side cameras 8A and 8B are imaging devices each having a camera using a solid-state imaging element such as a CCD, and are attached to the left and right side mirrors of the vehicle 2, for example, with their optical axes directed to the sides of the vehicle.

[0022] The driving assistance ECU 10 accumulates and stores images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, particularly some of the images captured by each camera while the vehicle is traveling, along with vehicle information (such as vehicle speed and vehicle position coordinates) at the time of capture, and generates a route surroundings image showing the surrounding environment along the route traveled by the vehicle by combining the accumulated and stored images. Furthermore, during autonomous driving assistance, the driving assistance ECU 10 performs image recognition processing on the captured images to detect lane lines, parking frame lines, and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and performs autonomous driving assistance based on the detection results. In particular, when performing parking assistance, the system also uses the obstacle detection results from the cameras to identify parking spaces, check the surrounding conditions, and so on.

[0023] Meanwhile, the ultrasonic sensors 9A-9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle 2. They transmit ultrasonic waves as search waves around the vehicle 2 and receive reflected waves from objects around the vehicle, thereby detecting the objects that reflect the search waves. Specifically, they are a type of distance measurement sensor that can measure the distance (measured distance) to the object that reflected the search wave by measuring the time from transmission to reception. The ultrasonic sensors 9A-9L are also configured to generate output signals (including the distance to the detected object) corresponding to the reception results of the received waves and output them to the control unit. Examples of objects that can be detected by the ultrasonic sensors 9A-9L include obstacles that the vehicle 2 must avoid when traveling, such as people, bicycles, other vehicles, and walls, as well as obstacles that form parking spaces. Instead of ultrasonic sensors, millimeter-wave sensors or laser sensors may be used as distance measurement sensors.

[0024] The installation position and installation direction of each ultrasonic sensor 9A-9L can be set as appropriate. In this embodiment, to detect objects in all directions (forward, backward, left, and right) of the vehicle 2, for example, ultrasonic sensors 9A-9D are installed on the front of the vehicle 2 facing the vehicle's traveling direction so that the transmission direction of the search wave is forward. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the transmission direction of the search wave is to the left of the vehicle's traveling direction. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the transmission direction of the search wave is to the right of the vehicle's traveling direction. Ultrasonic sensors 9I-9L are installed on the rear of the vehicle 2 facing the opposite direction to the vehicle's traveling direction so that the transmission direction of the search wave is to the rear of the vehicle. The ultrasonic sensors 9A-9L are all approximately the same height from the ground surface.

[0025] In this embodiment, among the ultrasonic sensors 9A-9L, the ultrasonic sensors 9A-9D on the front of the vehicle 2 and the ultrasonic sensors 9I-9L on the rear of the vehicle 2 are particularly installed in positions where they can receive reflected waves from adjacent sensors as indirect waves, so that by receiving direct and indirect waves as received waves, it is possible to determine not only the distance to an object but also the specific position of the object (its relative position to the vehicle) using triangulation. The ultrasonic sensors 9E-9H on the sides are installed at a distance from each other and cannot receive indirect waves, but as the vehicle moves, it is also possible to determine the specific position of the object (its relative position to the vehicle) by triangulation using the measured distances at the previous and current positions and the distance traveled between them.

[0026] Meanwhile, the driving assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it constantly detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drive source, and brakes, so that the vehicle travels along the generated travel trajectory at a speed according to the generated speed plan. In particular, when providing parking assistance, it uses the detection results of the front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L to check the parking space and its surrounding conditions, calculates a parking trajectory to the parking space, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. Meanwhile, when providing parking assistance for long-range parking, it calculates a parking trajectory including movement to a distant parking space, such as a pre-registered home garage or a monthly parking space in a parking lot, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. In addition, some of the images captured by each camera while the vehicle is traveling are cumulatively stored along with vehicle information (vehicle position coordinates, orientation, and direction of travel) at the time of capture, and the cumulatively stored images are combined to generate a route surroundings image showing the surrounding environment along the route traveled by the vehicle. These generated route surroundings images are also displayed during autonomous driving assistance. The driving assistance ECU 10 is connected to the operation unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L via an in-vehicle network such as a CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as an in-vehicle navigation device. The detailed configuration of the driving assistance ECU 10 will be described later.

[0027] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 1, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0028] Next, a detailed description will be given of the driving assistance ECU 10 in particular of the driving assistance device 1 provided in the vehicle 2. Fig. 2 is a block diagram showing the configuration of the driving assistance device 1 according to this embodiment.

[0029] As shown in FIG. 2, the driving assistance ECU (electronic control unit) 10 is an electronic control unit that controls the entire driving assistance device 1. It includes a CPU 31, which functions as a calculation device and a control device; a RAM 32, which serves as a working memory for the CPU 31 to perform various calculation processes and stores driving trajectory data and other information used when the driving trajectory is calculated; a ROM 33, which stores control programs as well as a route surroundings image generation processing program (see FIG. 3) and a parking assistance processing program (see FIG. 9), which will be described later; and a flash memory 34, which stores programs read from the ROM 33. The driving assistance ECU 10 also includes various processing algorithms. For example, the image storage means cumulatively stores captured images of the surrounding environment captured by the front camera 6, rear camera 7, and side cameras 8A and 8B installed on the vehicle, in association with vehicle information at the time the captured images were captured. The image synthesis means synthesizes the cumulatively stored captured images based on the vehicle information to generate a route surroundings image showing the surroundings along the route traveled by the vehicle. The image display means displays the route surroundings image.

[0030] The driving assistance ECU 10 is also connected to various sensors 37 for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as various vehicle drive units 38, such as the steering, brakes, accelerator, and transmission, and performs automatic driving assistance for the vehicle 2 by detecting the current vehicle behavior based on the detection results of these sensors 37 and controlling the various drive units 38. Specific details of the automatic driving assistance include, for example, detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and controlling the vehicle, such as the steering, drive source, and brakes, so that the vehicle travels along a generated travel trajectory at a speed in accordance with a speed plan that is also generated. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes based on manual operation.

[0031] The flash memory 34 also includes a vehicle information DB 35 and a captured image DB 36. The vehicle information DB 35 stores various information related to the vehicle 2. For example, the installation positions (height from the ground and left / right positions) of the cameras and ultrasonic sensors 9A to 9L installed on the vehicle 2, the detection axes (optical axes for cameras), overall length, vehicle width, wheelbase, minimum turning radius, etc. This information is input in advance by the occupants or a person from the vehicle manufacturer.

[0032] On the other hand, the captured image DB 36 is a storage means for accumulating and storing some of the captured images taken by the camera while the vehicle is traveling together with the vehicle information at the time of capturing the images. The vehicle information includes the position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle when the captured images stored in the captured image DB 36 were captured, and such vehicle information is estimated from, for example, the vehicle speed, wheel speed, wheel speed pulse integration value, steering angle, etc. acquired by the various sensors 37.

[0033] Next, a route periphery image generation processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to FIG. 3. FIG. 3 is a flowchart of the route periphery image generation processing program according to this embodiment. The route periphery image generation processing program is executed after the ACC (accessory power supply) of the vehicle 2 is turned on. The program accumulates and stores some of the captured images taken by the cameras provided in the vehicle 2, and generates a route periphery image showing the surrounding environment along the route traveled by the vehicle by combining the stored captured images. However, if the target for generating the route periphery image is limited to a parking lot, the route periphery image generation processing program may be executed when the vehicle enters the parking lot. Alternatively, if the period for generating the route periphery image is limited to when the vehicle is traveling with autonomous driving assistance, the route periphery image generation processing program may be executed when the autonomous driving assistance is started. The programs shown in the flowcharts in FIGS. 3 and 9 below are stored in the RAM 32 or ROM 33 provided in the driving assistance device 1, and are executed by the CPU 31.

[0034] The following steps S1 to S6 are performed for each image captured by the front camera 6, rear camera 7, and side cameras 8A and 8B until the vehicle has stopped traveling. For example, in this embodiment, the front camera 6, rear camera 7, and side cameras 8A and 8B have a frame rate of 30 fps (capturing 30 images per second), so the following processing is performed for the most recently captured image every 33 ms. However, the program execution interval does not necessarily have to be 33 ms; for example, if processing is performed on 10 images at a time, it may be performed every 330 ms.

[0035] First, in step (hereinafter abbreviated as S) 1, the CPU 31 acquires the most recent captured images by the front camera 6, rear camera 7, and side cameras 8A and 8B. As described above, in this embodiment, the front camera 6 is installed above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle. The rear camera 7 is installed near the upper center of the license plate attached to the rear of the vehicle 2, with its optical axis facing rearward in the vehicle. Furthermore, the side cameras 8A and 8B are installed on the left and right side mirrors of the vehicle 2, with their optical axes facing sideways in the vehicle. Images capturing the surrounding environment in all directions, in the front, rear, left, and right directions of the vehicle 2, are acquired.

[0036] Next, in S2, the CPU 31 generates a bird's-eye image of the vehicle periphery viewed vertically downward from the sky based on the real-time images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B acquired in S1. The bird's-eye image generation method will be described below as an example. As shown in FIG. 4, the real-time images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The images projected on the virtual projection plane are then converted into images viewed from a virtual viewpoint looking down vertically from above the vehicle 2, thereby generating bird's-eye images of each camera. The conversion to the image viewed from the virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate in the captured image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, into each coordinate in a ground coordinate system, which is set along the ground surface, and then converting it into each coordinate in the bird's-eye image coordinate system. The conversion formulas used for each coordinate conversion are already known, so a description thereof will be omitted. 5, overhead image 41 obtained by viewpoint-converting the image captured by front camera 6, overhead image 42 obtained by viewpoint-converting the image captured by rear camera 7, overhead image 43 obtained by viewpoint-converting the image captured by side camera 8A, and overhead image 44 obtained by viewpoint-converting the image captured by side camera 8B are synthesized (spliced ​​together), and an illustration image 45 that schematically shows the vehicle is inserted between each of the overhead images 41 to 44, thereby generating an overhead image. However, because the overhead image to be displayed is not generated in S2, there is no need to insert illustration image 45.

[0037] Next, in S3, CPU 31 extracts (trims) a portion of the overhead image generated in S2. Here, the range from which the overhead image is extracted is extraction range 46 shown in FIG. 6. Extraction range 46 is a U-shaped area (the shape of the letter U in katakana) surrounding the vehicle in the direction of travel of the host vehicle (forward if moving forward, backward if moving backward) and on three sides: left and right. FIG. 6 particularly shows extraction range 46 extracted when the host vehicle is moving forward, with the rear end positioned at the rear wheel axle and surrounding as close to the host vehicle as possible without including the host vehicle's shadow. When the vehicle is moving backward, the area is U-shaped in the opposite direction to when moving forward, with the front end positioned at the rear wheel axle and surrounding as close to the host vehicle as possible without including the host vehicle's shadow. By positioning the rear end of extraction range 46 near the rear wheel axle, it becomes possible to fill in the area around the route traveled by the vehicle without any gaps, even when overhead images of the vehicle turning are connected along the direction of travel as described below (S8). Note that while expanding extraction range 46 to the left or right increases the area that can be displayed in the route periphery image described below, the further away from the vehicle, the higher the possibility that the image will be stretched or distorted in the overhead image, so it is desirable to avoid expanding it to the left or right as much as possible.

[0038] On the other hand, the width a of the extraction range 46 is set to a width that will not result in gaps between the images when the extracted overhead images are connected along the traveling direction (S8) as described below. For example, assuming imaging in a parking lot, if the vehicle's traveling speed is assumed to be 20 km / h, the distance the vehicle travels between frames (33 ms) is 0.183 m, so if a = 0.2 m or so, no gaps will occur in the images. Note that if a is made longer, no gaps will occur in the images even when the vehicle is traveling at high speeds, but if an area far from the host vehicle is included, there is a high possibility that the image will be stretched or distorted in the overhead image. Therefore, it is desirable to make a as short as possible within a range that will not result in gaps in the images.

[0039] However, the value of a above is merely an example, and may be changed depending on the vehicle speed and the type of road on which the vehicle is traveling. For example, the value of a may be set to a small value in a parking lot where the vehicle is expected to travel at a low speed, and to a larger value on a public road where the vehicle is expected to travel at a high speed. Furthermore, although the extraction range 46 has a symmetrical shape in this embodiment, it may also have an asymmetrical shape. For example, in countries where driving is on the left side of the road, it is expected that parked vehicles are parked on the shoulder to the left of the vehicle 2 traveling on a public road, so the left extraction range 46 may be set wider than the right extraction range 46 to include these parked vehicles in the extraction range.

[0040] In this embodiment, a bird's-eye view image is generated by converting the captured image projected onto the ground surface into an image viewed from a virtual viewpoint looking down vertically from above the vehicle 2, and then the bird's-eye view image corresponding to the extraction range 46 shown in Fig. 6 is extracted, but it is also possible to generate a bird's-eye view image viewed diagonally downward instead of a bird's-eye view image, and then similarly extract the extraction range 46 shown in Fig. 6. Furthermore, the projection surface does not necessarily have to be the ground surface.

[0041] Next, in S4, the CPU 31 acquires various detection values ​​indicating vehicle behavior, such as vehicle speed, wheel speed, wheel speed pulse integration value, and steering angle, from various sensors 37 provided in the vehicle and connected via CAN.

[0042] Next, in S5, the CPU 31 estimates the current position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle based on the detection values ​​acquired in S4. The information estimated in S5 is information that estimates the position coordinates, orientation, and traveling direction (forward / reverse) of the vehicle as vehicle information when the overhead image extracted in S3 was captured. The position coordinates and orientation estimated in S5 are relative values ​​with respect to a reference position and orientation. This reference position and orientation are, for example, the position and orientation of the vehicle 2 at the timing when cumulative storage of overhead images for the current trip started (for example, when the ACC power was turned on, when the vehicle entered a parking lot, when autonomous driving started, etc.).

[0043] Thereafter, in S6, the CPU 31 associates some of the overhead images extracted in S3 with the vehicle information estimated in S5 and cumulatively stores them in the captured image DB 36. Specifically, the CPU 31 stores some of the overhead images in chronological order (or in an order according to the coordinates of the vehicle at the time of capturing the images).

[0044] Furthermore, in S7, the CPU 31 determines whether the vehicle has completed its current driving. Specifically, the CPU 31 determines that the vehicle has completed its driving when the shift position of the vehicle is changed to "P." This basically corresponds to the timing when parking in the parking space is completed.

[0045] If it is determined that the vehicle has completed its travel (S7: YES), the process proceeds to S8. On the other hand, if it is determined that the vehicle has not completed its travel (S7: NO), the process returns to S1, and overhead images continue to be generated and stored in the DB.

[0046] In S8, the CPU 31 generates a route surroundings image along the route traveled by the vehicle by synthesizing the plurality of overhead images stored in the captured image DB 36 in S6 based on the associated vehicle information.

[0047] Specifically, as shown in FIG. 7, a portion of the overhead image extracted in S3 (hereinafter referred to as extracted overhead image 47) is stitched together in accordance with the position coordinates, orientation, and traveling direction of the vehicle at the time of image capture to generate a route surrounding image 48. If an overlapping area occurs in the extracted overhead image 47 as a result of stitching, the overlapping area is overwritten with a newer image. FIG. 8 is an illustration of a route surrounding image 48 generated by stitching together extracted overhead images 47. The generated route surrounding image 48 also includes information about the route traveled by the vehicle when generating the route surrounding image 48, i.e., information specifying the route traveled by the vehicle when the route surrounding image 48 was generated. The route surrounding image 48 generated in S8 is stored in flash memory 34 together with the route information.

[0048] The route surroundings image 48 stored in the flash memory 34 is displayed when performing parking assistance for long-range parking, as described below, but can also be used as a thumbnail image. For example, when there are multiple parking spaces to park in and the user needs to select which parking space to park in, it is possible to display all or part of the route surroundings image 48 along the driving route to each parking space as a thumbnail image. The user can then refer to the displayed thumbnail images to select a parking space in which to park the vehicle.

[0049] Furthermore, in this embodiment, when the vehicle has finished traveling, the extracted overhead images 47 acquired during the current trip are read from the captured image DB 36, and the route surrounding images 48 along the route traveled by the vehicle during the current trip are generated all at once, but the route surrounding images 48 may be generated while the vehicle is traveling. For example, each time a new extracted overhead image 47 is acquired, the route surrounding image 48 may be joined to the existing route surrounding image 48 at the time of acquisition to generate the route surrounding image 48. In other words, the route surrounding image 48 along the route the vehicle is currently traveling may be generated in real time, targeting the area behind the current position of the vehicle.

[0050] In a specification that generates a route surrounding image 48 along the route the vehicle is currently traveling, targeting the area behind the current position of the vehicle in real time, the area is limited to the area behind the current position of the vehicle, but it is also possible to display a route surrounding image 48 along the route the vehicle is currently traveling even while traveling on a new route. It is also possible to display a route surrounding image 48 that reflects the surrounding conditions (for example, the positions of other vehicles) in real time. However, as will be described later, in long-range parking assistance it is important to display a route surrounding image 48 up to the parking space ahead in the traveling direction, so in this embodiment, the route surrounding image 48 is generated all at once when traveling ends.

[0051] In addition, when a vehicle travels again along the same route for which a route surrounding image 48 has already been generated, there is basically no need to generate the above-mentioned route surrounding image 48. However, if it is desired to display a route surrounding image 48 that reflects the surrounding conditions more closely to the present when performing parking assistance, for example, the route surrounding image 48 may be generated again and updated.

[0052] Here, a problem with the conventional technology is that when an image captured by a camera is displayed as an overhead or bird's-eye view image by changing the viewpoint, three-dimensional objects in the captured image are displayed distorted or stretched relative to their original shape. In this embodiment, however, only a very narrow range near the vehicle is extracted from the generated overhead view images as shown in FIG. 6, and similarly extracted overhead view images are joined together to generate the route surroundings image 48 shown in FIG. 8. Therefore, even if three-dimensional objects are included in the generated route surroundings image 48, they can be prevented from being distorted or stretched relative to their original shape.

[0053] Next, a parking assistance processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to Fig. 9. Fig. 9 is a flowchart of the parking assistance processing program according to this embodiment. Here, the parking assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned on, and is a program that provides parking assistance, particularly when the vehicle is parking, as one type of automatic driving assistance. The following description will focus on parking assistance when the vehicle is performing long-range parking.

[0054] Here, long-range parking refers to parking that targets a predetermined parking space located far away, such as a home garage or a monthly parking space in a parking lot, and is parking that requires traveling a relatively long distance to the target parking space. The following explanation also assumes that the vehicle has previously traveled a route to a parking space for long-range parking using the route surrounding image generation processing program (FIG. 3), and that a route surrounding image 48 along the route to the parking space has already been generated.

[0055] First, in S11, the CPU 31 determines whether to start parking assistance for long-range parking. In particular, the parking assistance in this embodiment automatically moves the vehicle to a parking space where the vehicle will be parked, and displays the route surroundings image 48 generated by the route surroundings image generation processing program (FIG. 3).

[0056] For example, parking assistance may be initiated when the user operates the operation unit 3 to select parking assistance, when the vehicle approaches home, or automatically when the vehicle detects that it has entered a pre-registered parking lot or when the vehicle approaches a pre-registered parking space. Furthermore, if there are multiple parking spaces that are candidates for parking and route surrounding images 48 along the driving route to each parking space are stored in the flash memory 34, all or part of the route surrounding images 48 along the driving route to each parking space may be displayed as thumbnail images in a list. In this case, the user can select a parking space in which to park the vehicle by referring to the displayed thumbnail images. When the route surrounding images 48 are displayed as thumbnail images, a route image 52 and a target parking position image 53 (described later) are also displayed, but the vehicle icon image 51 is excluded from the display (see FIGS. 10 and 11). Also, when displaying as a thumbnail image, it is desirable to display a route surrounding image 48 of the entire route on one screen, as shown in Figure 10, so that the user can understand at a glance what kind of route it is to the target parking space.

[0057] If it is determined that parking assistance for long-range parking should be started (S11: YES), the process proceeds to S12. On the other hand, if it is determined that parking assistance for long-range parking should not be started (S11: NO), the parking assistance processing program is terminated.

[0058] In S12, the CPU 31 acquires a parking space (parking position) to be used for the current long-range parking. In this embodiment, the parking space to be used for parking is a parking space registered in advance by the user. The driving route (planned driving route) from the vehicle's current position to the parking space is also acquired. As described above, this embodiment is based on the premise that a route surrounding image 48 along the driving route to the parking space has already been generated. The route surrounding image 48 also includes information about the route traveled by the vehicle when the route surrounding image 48 was generated, i.e., information specifying the route traveled by the vehicle when the route surrounding image 48 was generated. Therefore, the driving route from the vehicle's current position to the parking space is acquired by reading out the route information stored together with the route surrounding image 48. In other words, the planned driving route for this parking is acquired on the assumption that the vehicle will travel the same route as when the vehicle was previously parked. However, it is also possible to calculate the driving route to the parking space each time parking is performed.

[0059] In S13, the CPU 31 reads out, from among the route surrounding images 48 generated by the route surrounding image generation processing program (FIG. 3), the route surrounding image 48 corresponding to the driving route to the parking space acquired in S12, i.e., the route surrounding image 48 along the planned driving route along which the vehicle will travel.

[0060] In S14, the CPU 31 acquires various detection values ​​indicating the vehicle behavior, such as vehicle speed, wheel speed, wheel speed pulse integration value, and steering angle, from various sensors 37 equipped on the vehicle connected via CAN, and identifies the current vehicle position coordinates, orientation, and direction of travel (forward / reverse).

[0061] Thereafter, in S15, the CPU 31 draws the route surroundings image 48 read out in S13 on the liquid crystal display 4 in a manner that matches the current position, orientation, and traveling direction of the vehicle. That is, the route surroundings image 48 viewed from a virtual viewpoint set in correspondence with the coordinates of the current position, orientation, and traveling direction of the vehicle is drawn on the liquid crystal display 4. Note that the setting of the virtual viewpoint when drawing the route surroundings image 48 (from which viewpoint and in which direction the route surroundings image 48 is drawn) can be set as appropriate.

[0062] For example, FIG. 10 shows an example of a bird's-eye view image of the entire route viewed vertically downward from above the vehicle's current position. Specifically, a bird's-eye view image of a route surroundings image 48 covering the entire planned driving route from the vehicle's current position to the parking space is displayed on one screen. In the example shown in FIG. 10, it is possible to objectively display the entire route to the parking space and the surrounding environment. However, it is also possible to display a bird's-eye view image of only a portion of the route centered on the vehicle's current position, rather than the entire route. On the other hand, FIG. 11 shows an example of a bird's-eye view image obtained by setting a virtual viewpoint in the sky behind the vehicle's current position and looking diagonally downward from above in the direction of the parking space. In the example shown in FIG. 11, it is possible to subjectively display the route to the parking space and the surrounding environment from the viewpoint of the vehicle. Note that the bird's-eye view image shown in FIG. 10 and the bird's-eye view image shown in FIG. 11 may be switched and displayed based on a user operation.

[0063] Furthermore, in S16, the CPU 31 superimposes a host vehicle icon image 51 indicating the position and orientation of the host vehicle on the route surroundings image 48 drawn in S15, corresponding to the current position and orientation of the host vehicle, and draws the superimposed image on the LCD display 4. The host vehicle icon image 51 is an image that simulates the appearance of the host vehicle. By drawing the host vehicle icon image 51 as shown in FIGS. 10 and 11, the user can understand the environment around the vehicle in association with the current position and orientation of the host vehicle. Furthermore, in the overhead view image showing the entire route as shown in FIG. 10, displaying the host vehicle icon image 51 also makes it possible to indicate the current position of the host vehicle along the entire route to the parking space. Note that, in addition to the route surroundings image 48, the LCD display 4 may also display information indicating the progress of the movement to the parking space. For example, a meter indicating the ratio of the route length from the parking assistance start position to the current position of the host vehicle, assuming that the route length from the parking assistance start position to the parking space is 100, may be displayed on the LCD display 4. By checking the meter, the user can easily grasp how much time remains until parking is complete.

[0064] Furthermore, in S16, the CPU 31 also draws a route image 52 showing the travel route of the host vehicle when it traveled in the past (which also corresponds to the planned travel route for this time) on the route surrounding image 48. Note that the specific position and shape of the trajectory of the travel route of the host vehicle when it traveled in the past in the route surrounding image 48 can be identified from the vehicle information stored in the captured image DB 36. The vehicle information is information that identifies the position coordinates, orientation, and traveling direction (forward / backward) of the vehicle when the captured image stored in the captured image DB 36 was captured.

[0065] Furthermore, in S16, the CPU 31 also draws a target parking position image 53 indicating the position of the target parking space on the route surroundings image 48. Note that although the target parking position image 53 is an image that imitates the external shape of the vehicle in, for example, Figures 10 and 11, it may also be an image of a flag or an image of a frame.

[0066] By visually checking the route image 52 and the target parking position image 53 superimposed on the route surroundings image 48, the user can easily understand the route they will take to get to the parking position.

[0067] In the route surroundings image 48 shown in Fig. 11, the range of the route surroundings image 48 to be displayed changes as the current position of the vehicle changes, so the drawing range is sequentially switched as the vehicle moves. On the other hand, in the overhead image shown in Fig. 10, the route surroundings image 48 is fixed. Furthermore, with regard to the host vehicle icon image 51 drawn on the liquid crystal display 4 in S16, the display position and orientation in the route surroundings image 48 change accordingly as the current position and orientation of the vehicle change.

[0068] Here, a problem with the prior art is that when an image captured by a camera is displayed as a bird's-eye view or an overhead view by changing the viewpoint, three-dimensional objects in the captured image are displayed distorted or stretched relative to their original shape. However, in this embodiment, as described above, only a very narrow range in the vicinity of the vehicle is extracted, so that when the virtual viewpoint is converted to an arbitrary position and displayed as described above, even if three-dimensional objects such as parked vehicles are included in the route surroundings image 48, it is possible to prevent them from being displayed distorted or stretched relative to their original shape.

[0069] Next, in S17, the CPU 31 calculates a parking trajectory, which is a driving trajectory for parking the vehicle into the parking space acquired in S12. Here, the parking trajectory when performing long-range parking includes a trajectory for traveling through the parking lot or the passageways within the lot to move to the vicinity of the parking space, and a trajectory for actually entering the parking space from the vicinity of the parking space, including turning and reversing. As described above, this embodiment is based on the assumption that a route peripheral image 48 along the driving route to the parking space has already been generated. As described above, the route peripheral image 48 also includes information about the route traveled by the vehicle when generating the route peripheral image 48, i.e., information identifying the route traveled by the vehicle when the route peripheral image 48 was generated. Therefore, the parking trajectory may be acquired based on the route information stored together with the route peripheral image 48. In other words, the same parking trajectory as used in previous parking may be acquired as the current parking trajectory.

[0070] Thereafter, in S18, the CPU 31 starts parking assistance according to the parking trajectory calculated in S17. Specifically, the current position of the vehicle is detected at any time, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels at a specified speed along the generated parking trajectory. In addition, the shift position is also automatically switched.

[0071] However, when moving the vehicle along the parking path, only the steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, the vehicle may be moved manually rather than automatically. In this case, parking assistance may be provided by displaying the parking space and parking path on the LCD display 4, and providing voice guidance on when to turn the steering wheel.

[0072] Next, in S19, the CPU 31 determines whether parking of the vehicle is complete. Specifically, when the vehicle is positioned at the target parking position set within the parking space acquired in S12 and the shift position of the vehicle is changed to "P," parking of the vehicle is determined to be complete.

[0073] If it is determined that the vehicle has been parked (S19: YES), the parking assistance process program is terminated. On the other hand, if it is determined that the vehicle has not been parked (S19: NO), the parking assistance continues.

[0074] As described above in detail, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to this embodiment cumulatively store captured images of the surrounding environment taken by the front camera 6, rear camera 7, and side cameras 8A, 8B installed on the vehicle in association with vehicle information at the time the captured images were taken (S6). The cumulatively stored captured images are then combined based on the vehicle information to generate a route surroundings image 48 showing the surrounding environment along the route traveled by the vehicle (S8). The route surroundings image 48 is then displayed (S15, S16). Therefore, even in the case of long-range parking, which requires a long drive to a parking space, it is possible to display the situation around the route along which the vehicle will be driven, not just the vicinity of the vehicle. Furthermore, by displaying a vehicle image within the route surroundings image, the current position of the vehicle along the driving route can be easily grasped. Furthermore, the images captured by the front camera 6, rear camera 7, and side cameras 8A, 8B are converted into overhead images looking down vertically from a virtual viewpoint located in the sky (S2), and from within the overhead image, an image of a U-shaped area surrounding the vehicle from three sides (the direction of travel of the vehicle and the left and right sides) is extracted and stored (S6). Therefore, even when an overhead or bird's-eye image viewed from any virtual viewpoint is displayed, it is possible to display three-dimensional objects without distortion or stretching from their original shape. In addition, as the vehicle moves, the route surroundings images 48 to be displayed are sequentially switched and displayed together with the vehicle's icon image 51 indicating the vehicle's current position, making it possible to subjectively display the driving route and surrounding environment from the perspective of the vehicle. Furthermore, the planned driving route is the route to the parking position of the vehicle, and the route surroundings image 48 is displayed on one screen, so that the entire route and the surrounding environment can be objectively displayed.

[0075] The present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications are possible within the scope of the present invention. For example, in this embodiment, images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B are converted into bird's-eye views, and then portions of the bird's-eye views are accumulated and stored, and these are combined to generate the route surroundings image 48. However, portions of bird's-eye views may be stored instead of the bird's-eye views to generate the route surroundings image 48. Furthermore, when portions of the bird's-eye views are stored, they are formed into a U-shape as shown in Fig. 6, but they may be formed into a shape other than a U-shape as long as they are not far from the vehicle.

[0076] In this embodiment, it is assumed that the route surroundings image 48 shown in Fig. 10 or 11 is displayed on the liquid crystal display 4 while parking assistance by automatic driving assistance is being performed, but it may also be displayed while automatic driving assistance other than parking assistance is being performed. Furthermore, the route surroundings image 48 shown in Fig. 10 or 11 may also be displayed on the liquid crystal display 4 while driving manually.

[0077] 10 and 11 is displayed on an in-vehicle display, it may also be displayed on a display of a terminal outside the vehicle. This allows a user outside the vehicle to easily grasp the current progress towards parking completion, for example, when performing parking assistance by remote control.

[0078] In this embodiment, the route surroundings image generation processing program (FIG. 3) and the parking assistance processing program (FIG. 9) are configured to be executed by the driving assistance ECU 10 of the driving assistance device 1, but the executing entity can be changed as appropriate. For example, they may be executed by the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle device. [Explanation of symbols]

[0079] 1... driving assistance device, 2... vehicle, 3... operation unit, 4... liquid crystal display, 6... front camera (imaging device), 7... rear camera (imaging device), 8A, 8B... side cameras (imaging devices), 10... driving assistance ECU, 31... CPU, 36... captured image DB, 46... extraction range, 47... extracted overhead image, 48... route surrounding image, 51... host vehicle icon image (vehicle image), 52... route image

Claims

1. an image storage means for cumulatively storing captured images of the surrounding environment captured by an imaging device installed in the vehicle in association with vehicle information at the time the captured images were captured; an image synthesis means for synthesizing the cumulatively stored captured images based on the vehicle information to generate a route surroundings image showing the surrounding environment along a route that the vehicle has traveled in the past or is currently traveling; and an image display means for displaying the route surroundings image.

2. The image storage means converting the image captured by the imaging device into a bird's-eye view image viewed vertically downward from a virtual viewpoint located in the sky; 2. The driving assistance device according to claim 1, wherein an image of a U-shaped region surrounding the vehicle from three sides, ie, the direction of travel of the vehicle and the left and right sides, is extracted from the overhead image and stored.

3. The image display means 3. The driving support device according to claim 1, wherein the route surroundings images to be displayed are sequentially switched as the vehicle moves, and are displayed together with a vehicle image showing the current position of the vehicle.

4. 3. The driving support device according to claim 1, wherein the image display means displays the route surroundings image on one screen.

5. The image storage means converting the image captured by the imaging device into a bird's-eye view image viewed vertically downward from a virtual viewpoint located in the sky; The driving support device according to claim 1 , wherein an image of a region including a part of the traveling direction of the vehicle is extracted from the overhead image and stored.

Images