Driving assistance device

Abstract

To provide a driving assistance device capable of displaying a three-dimensional object without causing distortion or stretching relative to its original shape, even when displaying an image captured by an imaging device.SOLUTION: A driving assistance device is configured to: detect tires of other vehicles included in a captured image where a periphery of an own vehicle is captured; acquire a model image 55 showing appearance of a vehicle excluding tire portions; and based on positions of the tires of the other vehicles included in the captured image, perform editing, on the model image 55, including at least one or more of enlarging, reducing, and rotating; and generate a support image 61 by combining the edited model image 55 while aligning the combined support image with the position of the tires of the other vehicles included in the captured image; and display the generated support image on the liquid crystal display 4.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle. [Background technology]

[0002] Conventionally, various means have been used as information providing means for providing vehicle occupants with various information for vehicle driving support, such as route guidance and obstacle warnings. For example, such means include display on a liquid crystal display installed in the vehicle and audio output from a speaker. However, there are blind spots around the vehicle that are difficult to see from the driver's position. In particular, when performing special operations such as parking or leaving a parking space, it has conventionally been the case that a surrounding image captured by a camera installed in the vehicle is displayed on the liquid crystal display to allow the driver to understand the situation in such blind spots.

[0003] Furthermore, when displaying the vehicle's surroundings to a user, instead of displaying an image captured by a camera as is, multiple images are synthesized and then subjected to viewpoint conversion to improve visibility. For example, a technique for displaying a bird's-eye view image or a bird's-eye view image viewed from an arbitrary virtual viewpoint in the sky has been disclosed. However, such viewpoint conversion involves projecting and pasting captured images containing three-dimensional objects onto a flat projection surface (ground surface) without treating the captured images as three-dimensional objects. Therefore, positions higher than the ground surface are projected farther than they actually are, resulting in three-dimensional objects in the captured image being distorted or stretched relative to their original shapes. As a result, there is a problem in that three-dimensional objects are displayed enlarged or significantly distorted compared to their original shapes. Therefore, for example, Japanese Patent Publication No. 5292874 proposes a technology for reducing the stretching of three-dimensional objects by using the ground surface as the projection surface for areas near the vehicle, while using a bowl-shaped projection surface for areas farther from the vehicle instead of a flat surface. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5292874 (paragraph 0035, Figure 4) Summary of the Invention [Problem to be solved by the invention]

[0005] However, although Patent Document 1 reduces the elongation of three-dimensional objects compared to when all projection surfaces are flat (ground surface), the projection still occurs near the vehicle because the projection is on a flat surface. Furthermore, even when projecting onto a bowl-shaped projection surface, it is difficult to completely eliminate the elongation of three-dimensional objects, and some elongation of three-dimensional objects remains.

[0006] The present invention has been made to solve the above-mentioned problems in the conventional art, and aims to provide a driving assistance device that makes it possible to display three-dimensional objects without distortion or stretching of their original shapes, even when displaying images captured by an imaging device. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the driving assistance device of the present invention comprises an image acquisition means for acquiring an image of the surroundings of the vehicle, a tire detection means for detecting the tires of another vehicle included in the image, a model image acquisition means for acquiring a model image showing the vehicle exterior excluding the tire portion, an image editing means for editing the model image including at least one of enlarging, reducing, and rotating based on the position of the tire of the other vehicle included in the image, an assistance image generation means for generating an assistance image by synthesizing the edited model image to match the position of the tire of the other vehicle included in the image, and an image display means for displaying the assistance image on a display device. The "captured image of the surrounding environment" may be an image captured by an imaging device such as a camera, or may be an image obtained by processing the captured image. For example, it may be an image obtained by combining images captured by multiple cameras or an image obtained by converting the viewpoint. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above configuration, by synthesizing a model image showing the vehicle exterior at the location where the other vehicle is located in the captured image captured by the imaging device, it is possible to display the other vehicle included in the captured image without distortion or stretching from its original shape when displaying the captured image. In particular, even when displaying an overhead image or a bird's-eye view image seen from an arbitrary virtual viewpoint, it is possible to display the other vehicle, which is a three-dimensional object, without distortion or stretching from its original shape. On the other hand, even when the other vehicle is displayed by replacing it with a model image, the tire portion is displayed as a real image, so that when the other vehicle starts to move, it can be quickly recognized on the captured image. Furthermore, because the tire portion is low above the ground surface, it is less likely to stretch from its original shape, and the impact of displaying a real image can be minimized. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a vehicle according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention. [Figure 3] 4 is a flowchart of a driving assistance processing program according to the present embodiment. [Figure 4] 10A and 10B are diagrams illustrating a method for converting a captured image into a bird's-eye view image. [Figure 5] FIG. 10 is a diagram illustrating a method for generating a bird's-eye view image. [Figure 6] FIG. 2 is a diagram illustrating other vehicles and tires included in a bird's-eye view image. [Figure 7] FIG. 10 is a diagram showing an example of a model image. [Figure 8] FIG. 10 is a diagram showing a support image obtained by combining a bird's-eye view image with a model image. [Figure 9] FIG. 10 is a diagram illustrating a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] A detailed description will be given below of a specific embodiment of a driving assistance device according to the present invention with reference to the drawings. First, a vehicle 2 equipped with a driving assistance device 1 according to this embodiment will be described below. Fig. 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as a drive source (hybrid automobile). Furthermore, the vehicle type is not limited, and it may be a standard car, or a large commercial truck, bus, construction machinery, etc. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the driving operation of the user, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the driving operation of the user.

[0013] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which autonomous driving assistance is performed includes all road sections, including general roads and highways, as well as parking lots, and is performed only when the user selects to perform autonomous driving assistance (for example, turns on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is traveling to a parking space when parking (i.e., parking assistance).

[0014] In the vehicle control in the automated driving assistance of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles are detected as needed, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. In particular, when performing parking assistance, the system uses detection results from sensors and cameras to check the parking space where the vehicle is to park and its surrounding conditions, calculates a parking trajectory to the parking space, and automatically controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. However, in the parking assistance, only the steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, the system may only provide guidance on the parking trajectory to the parking space or guidance on vehicle operation, and the user may manually park the vehicle into the parking space. Meanwhile, when performing the automated driving assistance, the system also displays the scenery (actual view) around the vehicle on an in-vehicle display, as described below.

[0015] 1, the vehicle 2 has an operation unit 3 that accepts operations from the occupant, a liquid crystal display 4 that displays bird's-eye and overhead images of the vehicle's surroundings and other driving assistance-related information to the occupant, a speaker 5 that outputs audio guidance related to the driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the vehicle's surroundings, ultrasonic sensors 9A to 9F that detect obstacles around the vehicle, and a driving assistance ECU (electronic control unit) 10 that performs various calculations based on input information. The driving assistance ECU 10 and other components are collectively referred to as a driving assistance device 1.

[0016] Each component of the vehicle 2 will be described below. First, the operation unit 3 is arranged, for example, on the front of the handle (also called the steering wheel), and includes operation buttons and the like that are operated when starting automatic driving assistance. By operating the operation unit 3, the user can switch between manual driving, in which the vehicle travels based on the user's driving operation, and automatic driving assistance, in which the vehicle travels automatically without the user's driving operation. The operation unit 3 may have a touch panel provided on the front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0017] The liquid crystal display 4 is mounted on the instrument panel of the vehicle 2, and displays bird's-eye and overhead images of the vehicle surroundings generated by performing viewpoint conversion and synthesis processing on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B while autonomous driving assistance is being performed. Furthermore, if there is a warning object such as a pedestrian around the vehicle 2, a warning image indicating the presence of the warning object at the position of the warning object in the bird's-eye and overhead images may also be displayed. The liquid crystal display 4 may also be used for a navigation device.

[0018] The speaker 5 is mounted on the instrument panel of the vehicle 2 and outputs voice guidance and warning sounds related to driving assistance. The speaker 5 may also be used for a navigation device.

[0019] The forward camera 6 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0020] The rear camera 7 is an imaging device having a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with the optical axis facing toward the rear of the vehicle.

[0021] Furthermore, the side cameras 8A and 8B are imaging devices each having a camera using a solid-state imaging element such as a CCD, and are attached to the left and right side mirrors of the vehicle 2, for example, with their optical axes directed to the sides of the vehicle.

[0022] The driving assistance ECU 10 generates bird's-eye and overhead images of the surroundings of the vehicle by performing viewpoint conversion and synthesis processing on the captured images taken by the front camera 6, rear camera 7, and side cameras 8A and 8B. During execution of autonomous driving assistance, the driving assistance ECU 10 also performs image recognition processing on the captured images to detect lane lines and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and executes autonomous driving assistance based on the detection results.

[0023] Meanwhile, the ultrasonic sensors 9A-9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle 2. They transmit ultrasonic waves as search waves around the vehicle 2 and receive reflected waves from objects around the vehicle, thereby detecting the objects that reflect the search waves. Specifically, they are a type of distance measurement sensor that can measure the distance (measured distance) to the object that reflected the search wave by measuring the time from transmission to reception. The ultrasonic sensors 9A-9L are also configured to generate output signals (including the distance to the detected object) corresponding to the reception results of the received waves and output them to the control unit. Examples of objects that can be detected by the ultrasonic sensors 9A-9L include obstacles that the vehicle 2 must avoid when traveling, such as people, bicycles, other vehicles, and walls, as well as obstacles that form parking spaces. Instead of ultrasonic sensors, millimeter-wave sensors or laser sensors may be used as distance measurement sensors.

[0024] The installation position and installation direction of each ultrasonic sensor 9A-9L can be set as appropriate. In this embodiment, in order to detect objects in all directions (forward, backward, left, and right) of the vehicle 2 in the direction of travel, for example, ultrasonic sensors 9A-9D are installed on the front of the vehicle 2 so that the transmission direction of their search waves is in front of the vehicle. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing leftward so that the transmission direction of their search waves is on the left side of the vehicle. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing rightward so that the transmission direction of their search waves is on the right side of the vehicle. Ultrasonic sensors 9I-9L are installed on the rear of the vehicle 2 so that the transmission direction of their search waves is behind the vehicle. The heights from the ground surface of each ultrasonic sensor 9A-9L are approximately the same.

[0025] In this embodiment, among the ultrasonic sensors 9A-9L, the ultrasonic sensors 9A-9D on the front of the vehicle 2 and the ultrasonic sensors 9I-9L on the rear of the vehicle 2 are particularly installed in positions where they can receive reflected waves from adjacent sensors as indirect waves, so that by receiving direct and indirect waves as received waves, it is possible to determine not only the distance to an object but also the specific position of the object (its relative position to the vehicle) using triangulation. The ultrasonic sensors 9E-9H on the sides are installed at a distance from each other and cannot receive indirect waves, but as the vehicle moves, it is also possible to determine the specific position of the object (its relative position to the vehicle) by triangulation using the measured distances at the previous and current positions and the distance traveled between them.

[0026] Meanwhile, the driving assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it constantly detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, including steering, drive source, and braking, so that the vehicle travels along the generated travel path and at a speed according to the generated speed plan. In particular, when performing parking assistance, it uses the detection results of the front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A-9L to check the parking space and its surrounding conditions, calculates a parking path to the parking space, and controls the vehicle to enter the parking space along the calculated parking path and complete parking. The LCD display 4 also displays the scenery (actual view) around the vehicle. The driving assistance ECU 10 is connected to the operation unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A-9F via an in-vehicle network such as CAN. The driving assistance ECU 10 is also connected to various sensors such as a vehicle speed sensor, an acceleration sensor, a steering sensor, etc., mounted on the vehicle 2, as well as a navigation device, etc. The detailed configuration of the driving assistance ECU 10 will be described later.

[0027] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 1, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0028] Next, a detailed description will be given of the driving assistance ECU 10 in particular of the driving assistance device 1 provided in the vehicle 2. Fig. 2 is a block diagram showing the configuration of the driving assistance device 1 according to this embodiment.

[0029] As shown in FIG. 2, the driving assistance ECU (electronic control unit) 10 is an electronic control unit that controls the entire driving assistance device 1. It includes a CPU 31, which functions as a calculation device and a control device; a RAM 32, which is used as a working memory when the CPU 31 performs various calculation processes and stores driving trajectory data and other information used when the driving trajectory is calculated; a ROM 33, which stores control programs and a driving assistance processing program (see FIG. 3 ), which will be described later; and a flash memory 34, which stores programs read from the ROM 33. The driving assistance ECU 10 includes various processing algorithms. For example, the captured image acquisition means acquires captured images of the surroundings of the vehicle. The tire detection means detects the tires of other vehicles included in the captured images. The model image acquisition means acquires a model image showing the vehicle exterior excluding the tire portion. The image editing means performs editing on the model image, including at least one of enlarging, reducing, and rotating, based on the positions of the tires of other vehicles included in the captured images. The support image generation means generates a support image by combining the edited model image with the positions of the tires of other vehicles included in the captured images. The image display means displays the support image on the liquid crystal display 4.

[0030] The driving assistance ECU 10 is also connected to various sensors 37 for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as various vehicle drive units 38, such as the steering, brakes, accelerator, and transmission, and performs automatic driving assistance for the vehicle 2 by detecting the current vehicle behavior based on the detection results of these sensors 37 and controlling the various drive units 38. Specific details of the automatic driving assistance include, for example, detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and controlling the vehicle, such as the steering, drive source, and brakes, so that the vehicle travels along a generated travel trajectory at a speed in accordance with a speed plan that is also generated. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes based on manual operation.

[0031] The flash memory 34 also includes a vehicle information DB 35 and a model image DB 36. The vehicle information DB 35 stores various information related to the vehicle 2. For example, the installation positions (height from the ground and left / right positions) of the cameras and ultrasonic sensors 9A to 9L installed on the vehicle 2, the detection axes (optical axes for cameras), overall length, vehicle width, wheelbase, minimum turning radius, etc. This information is input in advance by the occupants or a person from the vehicle manufacturer.

[0032] Meanwhile, the model image DB 36 stores various information used when drawing 3D model images showing vehicle exteriors, which will be described later. Specifically, the DB 36 stores information necessary for executing various processes for creating 3DCG, such as modeling, scene layout setting, and rendering, as well as texture images to be applied to the created 3DCG models. The model image DB 36 may also store model images showing the exteriors of completed vehicles. In this case, only one model image may be stored, or multiple model images may be prepared for each manufacturer and vehicle type. Alternatively, model images may be prepared for each vehicle type, such as a light vehicle, a standard car, or a minivan.

[0033] Next, a driving assistance processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to Fig. 3. Fig. 3 is a flowchart of the driving assistance processing program according to this embodiment. Here, the driving assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned on, and is a program that assists the user using bird's-eye images and overhead images of the vehicle surroundings while automatic driving assistance is being performed for the vehicle. The program shown in the flowchart in Fig. 3 below is stored in the RAM 32 and ROM 33 provided in the driving assistance device 1, and is executed by the CPU 31.

[0034] First, in step (hereinafter abbreviated as S) 1, the CPU 31 determines whether or not assisted driving is being performed by automatic driving assistance. In this embodiment, when parking assistance is being performed as automatic driving assistance, a bird's-eye view image or an overhead view image of the surroundings of the vehicle is displayed on the liquid crystal display 4, as described below.

[0035] If it is determined that assisted driving by the autonomous driving assistance is being performed (S1: YES), the process proceeds to S2. On the other hand, if it is determined that assisted driving by the autonomous driving assistance is not being performed (S1: NO), the driving assistance processing program is terminated.

[0036] The following processing is performed for each image captured by the front camera 6, rear camera 7, and side cameras 8A, 8B. For example, since the front camera 6, rear camera 7, and side cameras 8A, 8B in this embodiment have a frame rate of 30 fps (capturing 30 images per second), the following processing is performed for the most recently captured image every 33 ms.

[0037] In S2, the CPU 31 generates a bird's-eye image of the vehicle periphery viewed diagonally downward from the sky based on real-time captured images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. As an example, a method for generating a bird's-eye image looking down on the vehicle periphery will be described below. As shown in FIG. 4, the real-time captured images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The captured images projected on the virtual projection plane are then converted into images viewed from a virtual viewpoint that looks diagonally downward from above the vehicle 2 in the forward direction of travel, thereby generating a bird's-eye image of each camera. The conversion to an image viewed from the virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate in a captured image coordinate system defined along a plane perpendicular to the optical axis of the camera into each coordinate in a ground coordinate system defined along the ground surface, and then further converting it into each coordinate in a bird's-eye image coordinate system. The conversion formulas used for each coordinate conversion are already known, so a description thereof will be omitted. 5, a bird's-eye view image 41 obtained by viewpoint-converting an image captured by front camera 6, a bird's-eye view image 42 obtained by viewpoint-converting an image captured by rear camera 7, a bird's-eye view image 43 obtained by viewpoint-converting an image captured by side camera 8A, and a bird's-eye view image 44 obtained by viewpoint-converting an image captured by side camera 8B are synthesized (spliced ​​together), and a host vehicle image 45 that schematically shows the host vehicle is inserted between each of the bird's-eye view images 41 to 44, thereby generating a bird's-eye view image. Note that a bird's-eye view image may also be generated in addition to the bird's-eye view image, and the process for generating the bird's-eye view image is basically the same as that for generating the bird's-eye view image, except that the angle of the line of sight when converting the viewpoint is different, so a description thereof will be omitted.

[0038] 5 also includes bird's-eye view image 42 obtained by converting the viewpoint of the image captured by rear camera 7, but if only the area ahead of the vehicle is displayed, bird's-eye view image 42 obtained by converting the viewpoint of the image captured by rear camera 7 may be excluded from the objects to be synthesized. When the vehicle is reversing, a bird's-eye view image looking diagonally down behind the vehicle is generated.

[0039] Next, in S3, the CPU 31 performs image recognition processing on the bird's-eye image generated in S2 to detect the tires of other vehicles included in the bird's-eye image. The process of detecting tires from the bird's-eye image can involve, for example, performing brightness correction based on the brightness difference between the road surface and the tires, followed by binarization processing to separate the tires from the image, geometric processing to correct distortion, and smoothing processing to remove noise from the image, thereby detecting the boundary between the road surface and the tires. The tires may also be detected using well-known template matching processing or feature point detection processing. The image recognition processing of the captured image is not limited to the above example, and may also be performed using, for example, machine learning. Instead of detecting only the tires, the entire other vehicle may be detected, and then only the tire portion may be isolated and detected.

[0040] For example, FIG. 6 shows an example of a bird's-eye view image 51 generated in S2. In the example shown in FIG. 6, the left front and left rear wheels of a parked vehicle 52 parked to the right of the vehicle and the right front and right rear wheels of a parked vehicle 53 parked to the left of the vehicle are detected. A problem with the prior art is that when a bird's-eye view image or a bird's-eye view image is generated by changing the viewpoint of an image captured by a camera, three-dimensional objects in the captured image are distorted or stretched relative to their original shapes. For example, in the bird's-eye view image 51 shown in FIG. 6, the parked vehicle 53 parked to the left of the vehicle is displayed greatly enlarged and distorted, making it difficult to grasp its shape. However, since the tire portion is located close to the ground surface, which is the projection surface, even though it is a three-dimensional object, the distortion and stretching are small and it is relatively easy to detect.

[0041] In addition to detecting tires, the image recognition process in S3 also preferably detects, as much as possible, the longitudinal orientation, vehicle type (e.g., light vehicle, standard vehicle, minivan), and exterior color of other vehicles equipped with the detected tires. Such information is detected using, for example, machine learning. The detected information is then used to generate and synthesize a model image 55, which will be described later.

[0042] Thereafter, in S4, the CPU 31 determines whether or not the tires of another vehicle were included in the bird's-eye image generated in S2 as a result of the image recognition in S3. However, as will be described later, in order to align the model image with the tire when combining the vehicle model image with the bird's-eye image (S8), it is desirable that tires be detected in at least two or more locations on the same vehicle. Therefore, in S4, it may be determined whether or not tires are detected in two or more locations on at least one of the same vehicles.

[0043] If it is determined that the tires of another vehicle are included in the bird's-eye image generated in S2 (S4: YES), the process proceeds to S6. On the other hand, if it is determined that the tires of another vehicle are not included in the bird's-eye image generated in S2 (S4: NO), the process proceeds to S5.

[0044] In S5, the CPU 31 displays the real-time bird's-eye view image, which shows the current environment around the vehicle and was generated in S2, on the liquid crystal display 4 as an assistance image for assisting in parking the vehicle. If an overhead view image has also been generated in S2, the overhead view image may also be displayed on the liquid crystal display 4 together with the bird's-eye view image. Alternatively, the bird's-eye view image and the overhead view image may be displayed so as to be switchable by a user operation. Then, the process proceeds to S10.

[0045] Meanwhile, in S6, the CPU 31 acquires a model image 55 showing the exterior of the vehicle excluding the tire portion. Here, the model image 55 is a three-dimensional polygon image, and is a 3D image of the body portion of the vehicle excluding only the tire portion, as shown in FIG. 7. The model image 55 may be, for example, a "wireframe model" that displays only the edges, or a "surface model" that displays the faces. The model image 55 has a shape that makes it possible to identify the vehicle and at least to identify the front and rear directions of the vehicle.

[0046] Furthermore, various material settings and mapping processes are performed on the model image 55. Examples of mapping processes include texture mapping, which applies a texture image to the surface of an object, and bump mapping, which changes the direction of light reflection to create fine irregularities. For example, a texture image is an image that depicts effects such as shadow, transparency, and reflection, and the effects of the texture image are drawn so as not to look unnatural when the model image 55 is placed on the bird's-eye view image 51 in S8, which will be described later, taking into consideration the orientation of the placement, the brightness of the surroundings, the position of a light source (e.g., sunlight, streetlights), and the like. Furthermore, it is desirable that the display color of the texture image be the same color family as the colors of other vehicles included in the bird's-eye view image 51.

[0047] 7 by performing processes such as modeling, scene layout setting, and rendering using information stored in the model image DB 36, or alternatively, a completed model image 55 may be stored in the model image DB 36 in advance, and the model image 55 may be acquired by reading it from the model image DB 36 in S6. When creating the model image 55, it is desirable to identify the make or type (e.g., light car, standard car, minivan) of other vehicles included in the bird's-eye view image using the image recognition process described above, and create a model image 55 of the corresponding make or type. When reading out the model image 55, it is desirable to store in advance a large number of model images 55 sorted by manufacturer, model, and type in the model image DB 36, and similarly identify the make or type (e.g., light car, standard car, minivan) of other vehicles included in the bird's-eye view image, and read out a model image 55 of the corresponding make or type.

[0048] Next, in S7, CPU 31 performs editing on model image 55 acquired in S6, including at least one of enlarging, reducing, and rotating. Specifically, first, the orientation of the other vehicle included in bird's-eye image 51 (the orientation of the other vehicle relative to the orientation of the host vehicle) is detected from the position and shape of the tire of the other vehicle detected by image recognition of bird's-eye image 51 in S3, and model image 55 is rotated to match the orientation of the other vehicle. Thereafter, the size of the other vehicle included in bird's-eye image 51 is detected from the position and shape of the detected tire of the other vehicle, and model image 55 is enlarged or reduced to match the size of the detected other vehicle. Specifically, enlargement or reduction is performed so that the blank tire portion of model image 55 matches the tire detected in bird's-eye image 51, in other words, so that the wheelbase of model image 55 matches the spacing between the tires detected in bird's-eye image 51.

[0049] Thereafter, in S8, CPU 31 composites model image 55 edited in S7 with the position of the tire of the other vehicle included in bird's-eye image 51. That is, the other vehicle and model image 55 are composited so as to be superimposed. Note that bird's-eye image 51 after composited with model image 55 will be referred to as support image 61 hereinafter. As described above, model image 55 is in the same position, orientation, and size as the other vehicle included in bird's-eye image 51, and its appearance also reflects the vehicle color, make, and type (kei car, regular car, minivan, etc.). As a result, model image 55 that imitates the other vehicle is displayed in support image 61 in place of the other vehicle included in bird's-eye image 51.

[0050] FIG. 8 shows an example in which a model image 55 is superimposed on a bird's-eye view image 51 to generate a support image 61. In the example shown in FIG. 8, a first model image 55a is superimposed on a parked vehicle 52 parked to the right of the host vehicle, and a second model image 55b is superimposed on a parked vehicle 53 parked to the left of the host vehicle. The texture images attached to the surfaces of the first model image 55a and the second model image 55b are drawn to avoid unnaturalness, taking into consideration the orientation of the first model image 55a and the second model image 55b, the ambient brightness, the position of the light source, and the like. Furthermore, it is desirable that the colors be similar to the colors of the other vehicles to be superimposed, and that the vehicle models and types be similar as well. For example, the first model image 55a is drawn with a texture image of the same color as the parked vehicle 52, and the appearance is made as similar as possible. The second model image 55b is drawn with a texture image of the same color as the parked vehicle 53, and the appearance is made as similar as possible.

[0051] As a result of combining the first model image 55a and the second model image 55b, for example, in the bird's-eye view image 51 shown in FIG. 8, the parked vehicle 53, particularly the one parked to the left of the vehicle, is displayed greatly enlarged and distorted, making it difficult to grasp its shape. However, by displaying the second model image 55b that resembles the parked vehicle 53 in place of the parked vehicle 53 in the support image 61, the distorted other vehicle included in the support image 61 can be replaced with a model image that resembles the other vehicle. As a result, the other vehicle included in the support image 61 can be displayed without distortion or stretching from its original shape. Note that although the tire portion is an actual image, the tire portion is low in height from the ground surface, so stretching from its original shape is unlikely, and the impact of displaying an actual image can be minimized. Furthermore, because the tire portion is an actual image, when the other vehicle starts to move, the tire portion can be confirmed to move, allowing the user to quickly grasp the other vehicle in the support image 61. Furthermore, when synthesizing the model image 55, the model image 55 may be simply superimposed on the actual scene image of the other vehicle, or the actual scene image of the other vehicle may be erased by painting over (masking) it, and then the model image 55 may be superimposed in the same position to display it.

[0052] 8, model image 55 to be synthesized with bird's-eye view image 51 is an opaque image with a transmittance of 0% so that other vehicles in the superimposed real image cannot be seen, but model image 55 may be a semi-transparent image. In that case, it is possible to intentionally allow the user to see other vehicles in the real image.

[0053] Then, in S9, the CPU 31 displays on the liquid crystal display 4 an image obtained by combining the model image 55 with the real-time bird's-eye view image showing the current environment around the vehicle, which was generated in S8, as a support image 61 for supporting parking of the vehicle. Note that if an overhead image has also been generated in S2, the overhead image may also be displayed on the liquid crystal display 4 together with the bird's-eye view image. Alternatively, the bird's-eye view image and the overhead view image may be displayed so as to be switchable by a user operation. Thereafter, the process proceeds to S10.

[0054] The support image 61 displayed on the LCD display 4 allows the user to clearly understand the environment around the vehicle relative to the vehicle's position, including areas that are difficult to see directly. The CPU 31 may also calculate the vehicle's future movement trajectory based on detection values ​​from a vehicle speed sensor, a steering sensor, etc., and display the movement trajectory superimposed on the support image 61. Thereafter, the support image 61 continues to be displayed until assisted driving by autonomous driving assistance is terminated (S10: YES).

[0055] Furthermore, in the support image 61, the surface of the model image 55 that faces the host vehicle may be displayed in a different display mode from the other surfaces. For example, in the example shown in FIG. 9, the left side surface of the first model image 55a that faces the host vehicle may be displayed in a different color from the others, or may be made to blink. Similarly, the right side surface of the second model image 55b that faces the host vehicle may be displayed in a different color from the others, or may be made to blink. Other vehicles that are close to the host vehicle are targets that require particular attention when providing driving assistance for the host vehicle. By displaying the support image 61 in the mode shown in FIG. 9, it is possible to warn the driver that the surface facing the host vehicle is a location that requires particular attention.

[0056] Thereafter, in S10, the CPU 31 determines whether or not to end assisted driving by the automatic driving assistance. Here, the assisted driving by the automatic driving assistance may be ended, for example, on the condition that the user performs a predetermined end operation on the operation unit 3, or on the condition that the shift position is shifted to "P" or the engine is turned off. Alternatively, the end condition may be on the condition that the automatic driving assistance cannot be continued.

[0057] If it is determined that the assisted driving by the autonomous driving assistance has ended (S10: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the assisted driving by the autonomous driving assistance has not ended (S10: NO), the process returns to S2, and the display of the assistance image on the LCD display 4 continues.

[0058] As described above in detail, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to this embodiment detect the tires of another vehicle included in a captured image of the vicinity of the vehicle (S3), acquire a model image 55 showing the vehicle exterior excluding the tire portion (S6), perform editing on the model image 55, including at least one of enlarging, reducing, and rotating, based on the position of the tire of the other vehicle included in the captured image (S7), generate a support image 61 by combining the edited model image 55 with the position of the tire of the other vehicle included in the captured image (S8), and display the support image on the LCD display 4 (S9). Therefore, when displaying the captured image, the other vehicle included in the captured image can be displayed without distortion or stretching relative to its original shape. In particular, even when displaying an overhead image or a bird's-eye view image viewed from an arbitrary virtual viewpoint, the other vehicle, which is a three-dimensional object, can be displayed without distortion or stretching relative to its original shape. On the other hand, even when the other vehicle is replaced with the model image 55 and displayed, the tire portion is displayed as a real image, so that the other vehicle can be quickly identified in the captured image when it starts to move. Furthermore, since the tire portion is low above the ground surface, it is difficult for it to stretch from its original shape, and the effect can be reduced even when the actual image is displayed. Furthermore, the model image 55 is an image in which the front and rear directions of the vehicle can be identified, and while the orientation of other vehicles included in the captured image is detected, the model image 55 is synthesized according to the orientation of the other vehicles detected. Therefore, it is possible to replace other vehicles included in the support image with model image 55 that imitates the other vehicles at the same position, orientation, and size as the other vehicles and display them. In addition, the exterior color of other vehicles included in the captured image is detected, and the display color of the model image is set to match the detected exterior color of the other vehicles, so that other vehicles included in the support image 61 can be displayed by replacing them with model image 55 having the same exterior color as the other vehicles. Furthermore, for the model image 55 included in the support image 61, the surface facing the vehicle is displayed in a different display mode from the other surfaces, so that the user can be warned of areas that require particular attention.

[0059] The present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications are possible within the scope of the present invention. For example, in this embodiment, the model image 55 is an image in which the front-to-rear direction of the vehicle can be identified, but it may be a general-purpose image in which the front-to-rear direction cannot be identified (front-to-rear symmetrical). In this case, it is not necessary to detect the front-to-rear direction of other vehicles in the image recognition process of S3, which reduces the processing load.

[0060] In this embodiment, a bird's-eye view image generated from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B is displayed on the liquid crystal display 4 as a landscape image around the vehicle, but a bird's-eye view image may also be displayed. Furthermore, the image captured by the camera as is, rather than a processed image, may be displayed on the liquid crystal display 4. In that case, the image to be synthesized with the model image 55 in steps S6 to S8 described above may be a bird's-eye view image or an unprocessed captured image, rather than a bird's-eye view image.

[0061] In addition, in this embodiment, it is assumed that the support image 61 shown in Figure 8 is displayed while driving is being performed using automatic driving assistance, but the support image 61 shown in Figure 8 may also be displayed while driving is being performed using manual driving.

[0062] In this embodiment, the driving assistance processing program (FIG. 3) is executed by the driving assistance ECU 10 of the driving assistance device 1, but the execution entity can be changed as appropriate. For example, the program may be executed by the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle device. [Explanation of symbols]

[0063] 1...driving assistance device, 2...vehicle, 3...operation unit, 4...liquid crystal display, 6...front camera, 7...rear camera, 8A, 8B...side cameras, 10...driving assistance ECU, 31...CPU, 51...bird's-eye view image, 52, 53...parked vehicle (other vehicle), 55...model image, 61...assistance image

Claims

1. an image acquisition means for acquiring an image of the surroundings of the vehicle; a tire detection means for detecting tires of other vehicles included in the captured image; a model image acquisition means for acquiring a model image showing the exterior of a vehicle excluding a tire portion; an image editing means for editing the model image, including at least one of enlarging, reducing, and rotating, based on the position of the tire of the other vehicle included in the captured image; a support image generating means for generating a support image by synthesizing the edited model image in accordance with the position of the tire of the other vehicle included in the captured image; and an image display means for displaying the assistance image on a display device.

2. The model image is an image in which the front and rear directions of the vehicle can be identified, an other vehicle direction detection means for detecting the direction of another vehicle included in the captured image; The driving assistance device according to claim 1 , wherein the assistance image generating means synthesizes the model image in accordance with the orientation of the other vehicle that has been detected.

3. an other vehicle color detection means for detecting the exterior color of another vehicle included in the captured image; The driving support device according to claim 1 , wherein the image editing means sets a display color of the model image to match an exterior color of the detected other vehicle.

4. 4. The driving support device according to claim 1, wherein the image display means displays a surface of the model image included in the support image that faces the vehicle in a different display mode from other surfaces of the model image.

Images