Driving assistance device

Abstract

To provide a driving support device capable of allowing a user to surely recognize even an attention object in a stopped state when guiding the attention object by using a vehicle periphery image.SOLUTION: An attention object which is present in the periphery of a vehicle and can take at least one of a moving state and a stopping state and is an attention object to a vehicle is detected, a model image 65 showing the appearance of the attention object is acquired, a support image 66 obtained by synthesizing the model image 65 with a vehicle periphery image showing the periphery of the vehicle in accordance with the position of the attention object is generated, and the generated support image 66 is displayed on a liquid crystal display 4, while the model image 65 corresponding to the attention object is emphatically displayed when the attention object is in the stopping state.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle. [Background technology]

[0002] Conventionally, various means have been used as information providing means for providing vehicle occupants with various information for vehicle driving support, such as route guidance and obstacle warnings. For example, such means include display on a liquid crystal display installed in the vehicle and audio output from a speaker. However, there are blind spots around the vehicle that are difficult to see from the driver's position. In particular, when performing special operations such as parking or leaving a parking space, a vehicle surroundings image showing the surroundings of the vehicle has conventionally been displayed on a liquid crystal display to allow the driver to understand the situation in such blind spots.

[0003] Furthermore, a technique has been proposed for displaying a vehicle periphery image and, if there is an object of attention around the vehicle that should be taken into account by the vehicle, making the user aware of the presence of the object of attention by using the vehicle periphery image. For example, Japanese Patent Application Laid-Open No. 2005-5978 discloses a technique for displaying the situation seen from the driver's seat of the vehicle in a simple graphic representation, and for rendering other vehicles or people if they are present within the display range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-5978 A (paragraphs 0061-0062, Figure 8) Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned Patent Document 1, a model image showing the appearance of a target object can be drawn within a vehicle surroundings image as a result of rendering. However, since the movement of the model image is linked to the movement of the actual target object, for example, if the target object is stationary, the model image will also be stationary. Humans have a tendency to have difficulty recognizing stationary objects compared to moving objects. Therefore, when a model image is stationary, it blends into the background, and the user may not notice the presence of the model image, i.e., the target object around the vehicle. Furthermore, compared to a moving target object, it is difficult to predict in advance the direction a stationary target object will move if it subsequently starts moving. Therefore, it is important to make the user aware of the presence of a stationary target object.

[0006] The present invention has been made to solve the above-mentioned problems in the conventional art, and aims to provide a driving assistance device that enables the user to more reliably recognize objects of interest even when the object of interest is in a stopped state when providing guidance on objects of interest using images of the vehicle's surroundings. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the driving assistance device of the present invention comprises an object detection means for detecting an object of attention that is located in the vicinity of the vehicle and is capable of being in at least one of a moving state and a stopped state and that is a target of attention for the vehicle, a model image acquisition means for acquiring a model image showing the appearance of the object of attention, an assistance image generation means for generating an assistance image by combining the model image with a vehicle surroundings image showing the vicinity of the vehicle in accordance with the position of the object of attention, an image display means for displaying the assistance image on a display device, and an emphasis display means for highlighting the model image corresponding to the object of attention when the object of attention is in a stopped state. The "vehicle surroundings image showing the surroundings of the vehicle" may be an actual image of the surroundings of the vehicle captured by an imaging device such as a camera, or may be a virtual image of the surroundings of the vehicle reproduced using CG (computer graphics). Furthermore, if it is an actual image, it may be the captured image itself, or an image obtained by processing the captured image. For example, it may be an image obtained by combining images captured by multiple cameras or an image obtained by viewpoint conversion. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above configuration, when guiding the user to an object of attention using an image of the vehicle's surroundings, a model image showing the appearance of the object of attention is displayed in accordance with the position of the object of attention, and in particular, when the object of attention is in a stopped state, the model image corresponding to the object of attention is highlighted, making it possible for the user to more reliably recognize objects of attention that are in a stopped state. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a vehicle according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention. [Figure 3] 4 is a flowchart of a driving assistance processing program according to the present embodiment. [Figure 4] 10A and 10B are diagrams illustrating a method of converting a captured image into a bird's-eye view image and an overhead view image. [Figure 5] 1A and 1B are diagrams illustrating a method for generating a bird's-eye view image and an overhead view image. [Figure 6] 10A and 10B are diagrams illustrating attention objects included in a bird's-eye view image and a bird's-eye view image. [Figure 7] FIG. 10 is a diagram showing an example of a model image. [Figure 8] 10A and 10B are diagrams illustrating a support image obtained by combining a bird's-eye view image and an overhead view image with a model image. [Figure 9]10A and 10B are diagrams illustrating a display mode of a model image when an attention object is in a stationary state. [Figure 10] 10A and 10B are diagrams illustrating an example of highlighting a model image. DETAILED DESCRIPTION OF THE INVENTION

[0010] A detailed description will be given below of a specific embodiment of a driving assistance device according to the present invention with reference to the drawings. First, a vehicle 2 equipped with a driving assistance device 1 according to this embodiment will be described below. Fig. 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as a drive source (hybrid automobile). Furthermore, the vehicle type is not limited, and it may be a standard car, or a large commercial truck, bus, construction machinery, etc. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the driving operation of the user, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the driving operation of the user.

[0013] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which autonomous driving assistance is performed includes all road sections, including general roads and highways, as well as parking lots, and is performed only when the user selects to perform autonomous driving assistance (for example, turns on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is traveling to a parking space when parking (i.e., parking assistance).

[0014] In the vehicle control in the automated driving assistance of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles are detected as needed, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels along the generated travel trajectory at a speed according to the generated speed plan. In particular, when performing parking assistance, the system uses detection results from sensors and cameras to check the parking space where the vehicle is to park and its surrounding conditions, calculates a parking trajectory to the parking space, and automatically controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. In addition to the above-mentioned normal parking assistance, the system also supports long-range parking, in which parking is targeted at a distant parking space, such as a home garage or a monthly parking space in a parking lot. In parking assistance for long-range parking, the system automatically controls the vehicle to move to a distant, predetermined parking space and complete parking. However, in the parking assistance, only steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Alternatively, only guidance on the parking path to the parking space or guidance on vehicle operation may be provided, and the user may manually perform the parking operation into the parking space.

[0015] Furthermore, while vehicle control such as steering, drive source, and braking is being performed automatically, the vehicle occupants can cancel the automated driving assistance at any time and can also apply the brakes to stop the vehicle. When the automated driving assistance is performed, as described below, the scenery around the vehicle (which may be a real scene or a virtual scene created using CG) is displayed on the in-vehicle display, and a warning is given of objects that require the vehicle's attention. The vehicle occupants can interrupt the automated driving assistance and stop the vehicle by applying the brakes if necessary while watching the display.

[0016] 1, the vehicle 2 has an operation unit 3 that accepts operations from the occupant, a liquid crystal display 4 that displays bird's-eye and overhead images of the vehicle's surroundings and other driving assistance-related information to the occupant, a speaker 5 that outputs audio guidance related to the driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the vehicle's surroundings, ultrasonic sensors 9A to 9F that detect obstacles around the vehicle, and a driving assistance ECU (electronic control unit) 10 that performs various calculations based on input information. The driving assistance ECU 10 and other components are collectively referred to as a driving assistance device 1.

[0017] Each component of the vehicle 2 will be described below. First, the operation unit 3 is arranged, for example, on the front of the handle (also called the steering wheel), and includes operation buttons and the like that are operated when starting automatic driving assistance. By operating the operation unit 3, the user can switch between manual driving, in which the vehicle travels based on the user's driving operation, and automatic driving assistance, in which the vehicle travels automatically without the user's driving operation. The operation unit 3 may have a touch panel provided on the front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0018] The liquid crystal display 4 is mounted on the instrument panel of the vehicle 2, and displays bird's-eye and overhead images of the surroundings of the vehicle generated by performing viewpoint conversion and synthesis processing on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B during autonomous driving assistance. In addition, if there is an object of attention that the vehicle should be wary of, such as a pedestrian or bicycle, around the vehicle 2, the liquid crystal display 4 also displays a model image showing the appearance of the object of attention superimposed on the position of the object of attention in the bird's-eye and overhead images. The liquid crystal display 4 may also be used for a navigation device.

[0019] The speaker 5 is mounted on the instrument panel of the vehicle 2 and outputs voice guidance and warning sounds related to driving assistance. The speaker 5 may also be used for a navigation device.

[0020] The forward camera 6 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0021] The rear camera 7 is an imaging device having a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with the optical axis facing toward the rear of the vehicle.

[0022] Furthermore, the side cameras 8A and 8B are imaging devices each having a camera using a solid-state imaging element such as a CCD, and are attached to the left and right side mirrors of the vehicle 2, for example, with their optical axes directed to the sides of the vehicle.

[0023] The driving assistance ECU 10 then performs viewpoint conversion and synthesis processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B to generate bird's-eye and overhead images of the vehicle's surroundings. During autonomous driving assistance, the driving assistance ECU 10 also performs image recognition processing on the captured images to detect lane markings and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and performs autonomous driving assistance based on the detection results. In particular, when performing parking assistance, the system also uses the obstacle detection results from the cameras to check the parking space and its surroundings.

[0024] Meanwhile, the ultrasonic sensors 9A-9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle 2. They transmit ultrasonic waves as search waves around the vehicle 2 and receive reflected waves from objects around the vehicle, thereby detecting the objects that reflect the search waves. Specifically, they are a type of distance measurement sensor that can measure the distance (measured distance) to the object that reflected the search wave by measuring the time from transmission to reception. The ultrasonic sensors 9A-9L are also configured to generate output signals (including the distance to the detected object) corresponding to the reception results of the received waves and output them to the control unit. Examples of objects that can be detected by the ultrasonic sensors 9A-9L include obstacles that the vehicle 2 must avoid when traveling, such as people, bicycles, other vehicles, and walls, as well as obstacles that form parking spaces. Instead of ultrasonic sensors, millimeter-wave sensors or laser sensors may be used as distance measurement sensors.

[0025] The installation position and installation direction of each ultrasonic sensor 9A-9L can be set as appropriate. In this embodiment, to detect objects in all directions (forward, backward, left, and right) of the vehicle 2, for example, ultrasonic sensors 9A-9D are installed on the front of the vehicle 2 facing the vehicle's traveling direction so that the transmission direction of the search wave is forward. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the transmission direction of the search wave is to the left of the vehicle's traveling direction. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the transmission direction of the search wave is to the right of the vehicle's traveling direction. Ultrasonic sensors 9I-9L are installed on the rear of the vehicle 2 facing the opposite direction to the vehicle's traveling direction so that the transmission direction of the search wave is to the rear of the vehicle. The ultrasonic sensors 9A-9L are all approximately the same height from the ground surface.

[0026] In this embodiment, among the ultrasonic sensors 9A-9L, the ultrasonic sensors 9A-9D on the front of the vehicle 2 and the ultrasonic sensors 9I-9L on the rear of the vehicle 2 are particularly installed in positions where they can receive reflected waves from adjacent sensors as indirect waves, so that by receiving direct and indirect waves as received waves, it is possible to determine not only the distance to an object but also the specific position of the object (its relative position to the vehicle) using triangulation. The ultrasonic sensors 9E-9H on the sides are installed at a distance from each other and cannot receive indirect waves, but as the vehicle moves, it is also possible to determine the specific position of the object (its relative position to the vehicle) by triangulation using the measured distances at the previous and current positions and the distance traveled between them.

[0027] Meanwhile, the driving assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it constantly detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drive source, and brakes, so that the vehicle travels along the generated travel trajectory at a speed according to the generated speed plan. In particular, when providing parking assistance, it uses the detection results of the front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L to check the parking space and its surrounding conditions, calculates a parking trajectory to the parking space, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. Meanwhile, when providing parking assistance for long-range parking, it calculates a parking trajectory including movement to a distant parking space, such as a pre-registered home garage or a monthly parking space in a parking lot, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking. The LCD display 4 also displays bird's-eye and overhead images of the vehicle's surroundings generated from the images captured by the cameras. When there is a pedestrian, bicycle, or other object of interest near the vehicle 2, the LCD display 4 may superimpose a model image showing the appearance of the object of interest at the position of the object of interest in the bird's-eye and overhead images, or may output a warning sound. The driving assistance ECU 10 is connected to the operation unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L via an in-vehicle network such as a CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as a navigation system or other in-vehicle device. The detailed configuration of the driving assistance ECU 10 will be described later.

[0028] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 1, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0029] Next, a detailed description will be given of the driving assistance ECU 10 in particular of the driving assistance device 1 provided in the vehicle 2. Fig. 2 is a block diagram showing the configuration of the driving assistance device 1 according to this embodiment.

[0030] As shown in FIG. 2, the driving assistance ECU (electronic control unit) 10 is an electronic control unit that controls the entire driving assistance device 1. It includes a CPU 31, which functions as a calculation device and a control device; a RAM 32, which is used as a working memory when the CPU 31 performs various calculation processes and stores driving trajectory data and other information when the driving trajectory is calculated; a ROM 33, which stores control programs as well as a driving assistance processing program (see FIG. 3 ), which will be described later; and a flash memory 34, which stores programs read from the ROM 33. The driving assistance ECU 10 also includes various processing algorithms. For example, the object detection means detects a cautionary object that is located around the vehicle and can be in at least one of a moving and a stationary state and that should be taken into account by the vehicle. The model image acquisition means acquires a model image showing the appearance of the cautionary object. The support image generation means generates a support image by combining the model image with a vehicle surroundings image showing the vehicle's surroundings in accordance with the position of the cautionary object. The image display means displays the support image on the LCD display 4. The highlighting means highlights a model image corresponding to the attention object when the attention object is in a stationary state. The captured image acquiring means acquires captured images of the periphery of the vehicle. That is, the driving assistance ECU is an example of an object detection means, a model image acquiring means, an assistance image generation means, an image display means, a highlighting means, and a captured image acquiring means.

[0031] The driving assistance ECU 10 is also connected to various sensors 37 for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as various vehicle drive units 38, such as the steering, brakes, accelerator, and transmission, and performs automatic driving assistance for the vehicle 2 by detecting the current vehicle behavior based on the detection results of these sensors 37 and controlling the various drive units 38. Specific details of the automatic driving assistance include, for example, detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and controlling the vehicle, such as the steering, drive source, and brakes, so that the vehicle travels along a generated travel trajectory at a speed in accordance with a speed plan that is also generated. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes based on manual operation.

[0032] The flash memory 34 also includes a vehicle information DB 35 and a model image DB 36. The vehicle information DB 35 stores various information related to the vehicle 2. For example, the installation positions (height from the ground and left / right positions) of the cameras and ultrasonic sensors 9A to 9L installed on the vehicle 2, the detection axes (optical axes for cameras), overall length, vehicle width, wheelbase, minimum turning radius, etc. This information is input in advance by the occupants or a person from the vehicle manufacturer.

[0033] Meanwhile, the model image DB 36 stores various information used when rendering 3D model images showing the appearance of people and bicycles. In this embodiment, an attention object that is a target of attention for a vehicle is defined as an object that can be at least either moving or stationary, i.e., an object that moves or stops, rather than a fixed object such as a wall or a utility pole. Any object that moves or stops may be used as an attention object, and preferably, a specific object among these may be used as an attention object. For example, the specific object may be a person or a bicycle. However, attention objects are not limited to people and bicycles, and may also include, for example, vehicles and motorcycles. In this case, information used when rendering 3D model images showing the appearance of vehicles and motorcycles is also stored in the model image DB 36.

[0034] The model image DB 36 stores various types of information used when rendering 3D model images showing the appearance of people and bicycles, including information required for executing each process for creating 3DCG, such as modeling, scene layout setting, and rendering, as well as texture images to be applied to the created 3DCG models. The model image DB 36 may also store model images showing the appearance of completed people and bicycles. In this case, only one model image may be prepared for each type of attention object, or multiple model images may be prepared for each type. For example, different model images may be prepared for children and adults in the case of people, or for city bikes and mountain bikes in the case of bicycles. However, in the following explanation, only one model image may be prepared for each type of attention object.

[0035] Next, a driving assistance processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to Fig. 3. Fig. 3 is a flowchart of the driving assistance processing program according to this embodiment. Here, the driving assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned on, and is a program that provides assistance to the user using bird's-eye view images and overhead view images of the area around the vehicle while automatic driving assistance is being performed for the vehicle.

[0036] However, in the following embodiment, bird's-eye and overhead images of the surroundings of the vehicle are displayed while automatic driving assistance is being performed for the vehicle, but it is not necessary to display bird's-eye and overhead images of the surroundings of the vehicle only while automatic driving assistance is being performed, and bird's-eye and overhead images of the surroundings of the vehicle may also be displayed while the vehicle is being driven manually. Alternatively, bird's-eye and overhead images of the surroundings of the vehicle may be displayed only while specific automatic driving assistance, such as parking assistance for long-range parking, is being performed. The program shown in the flowchart in FIG. 3 below is stored in RAM 32 and ROM 33 provided in driving assistance device 1, and is executed by CPU 31.

[0037] First, in step (hereinafter abbreviated as S) 1, the CPU 31 determines whether assisted driving by automatic driving assistance is being performed. As described above, the determination condition may be whether specific automatic driving assistance such as parking assistance is being performed.

[0038] In this embodiment, when the user selects to perform automatic driving assistance by operating the operation unit 3 and it is determined that automatic driving assistance is possible, assisted driving is performed. The automatic driving assistance includes detecting the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and automatically controlling the vehicle, including steering, drive source, and brakes, so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. In particular, in the case of parking assistance, the vehicle is automatically controlled until the vehicle is parked in a parking space. However, it is also possible to automatically perform steering operation alone, and manually control the drive source and brakes.

[0039] If it is determined that assisted driving by the autonomous driving assistance is being performed (S1: YES), the process proceeds to S2. On the other hand, if it is determined that assisted driving by the autonomous driving assistance is not being performed (S1: NO), the driving assistance processing program is terminated.

[0040] The following processing is performed for each image captured by the front camera 6, rear camera 7, and side cameras 8A, 8B. For example, since the front camera 6, rear camera 7, and side cameras 8A, 8B in this embodiment have a frame rate of 30 fps (capturing 30 images per second), the most recently captured image is acquired every 33 ms, and the following processing is performed on the acquired image.

[0041] In S2, the CPU 31 generates a bird's-eye image of the vehicle periphery viewed diagonally downward from the sky and a bird's-eye image of the vehicle periphery viewed vertically downward from the sky based on real-time captured images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. As an example, a method for generating a bird's-eye image of the vehicle periphery viewed diagonally downward from the sky is described below. As shown in FIG. 4, the real-time captured images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The captured images projected on the virtual projection plane are converted into images viewed from a virtual viewpoint that looks diagonally downward from above the vehicle 2 in the forward direction of travel, thereby generating a bird's-eye image of each camera. The conversion to the image viewed from the virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate in the captured image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, into each coordinate in a ground coordinate system, which is set along the ground surface, and then converting it into each coordinate in the bird's-eye image coordinate system. The conversion formulas used for each coordinate conversion are already known, so a description thereof will be omitted. Then, as shown in FIG. 5, bird's-eye view image 41 obtained by viewpoint-converting an image captured by front camera 6, bird's-eye view image 42 obtained by viewpoint-converting an image captured by rear camera 7, bird's-eye view image 43 obtained by viewpoint-converting an image captured by side camera 8A, and bird's-eye view image 44 obtained by viewpoint-converting an image captured by side camera 8B are synthesized (spliced ​​together), and a host vehicle image 45, which schematically shows the host vehicle, is inserted between each of bird's-eye view images 41 to 44 to generate a bird's-eye view image. The virtual viewpoint of the bird's-eye view image may be the interior of the vehicle (i.e., the viewpoint of a passenger). Furthermore, while the bird's-eye view image shown in FIG. 5 also includes bird's-eye view image 42 obtained by viewpoint-converting an image captured by rear camera 7, bird's-eye view image 42 obtained by viewpoint-converting an image captured by rear camera 7 may be excluded from the synthesis if only the area ahead of the vehicle is displayed. Furthermore, when the vehicle is reversing, a bird's-eye view image looking diagonally down behind the vehicle is generated.

[0042] The overhead view image is generated by inserting a host vehicle image 55, which schematically shows the host vehicle, between each of the overhead view images 51 to 54.

[0043] Next, in S3, the CPU 31 performs image recognition processing on the bird's-eye image and the overhead image generated in S2 to detect each of the attention objects included in the bird's-eye image and the overhead image. As described above, the attention objects can be in at least one of a moving state and a stationary state and are attention objects for the vehicle, such as a person or a bicycle. In this embodiment, the image recognition processing is performed on the bird's-eye image and the overhead image generated in S2. However, the image recognition processing may also be performed on the image captured by the camera from which the bird's-eye image and the overhead image are synthesized. As described above, in detecting the attention objects in S3, the attention objects are detected using at least the image captured by the imaging device (one or more of the front camera 6, the rear camera 7, and the side cameras 8A and 8B). Then, if the category of an object determined to be included in the image displayed on the LCD display 4 using at least the image captured by the imaging device matches the category of the attention object, the object is detected as the attention object.

[0044] The process of detecting an object of attention from the bird's-eye view image and the overhead view image in S3 may involve, for example, performing brightness correction based on the brightness difference between the road surface and the object of attention, followed by binarization processing to separate the object of attention from the image, geometric processing to correct distortion, smoothing processing to remove noise from the image, and so on, to detect the boundary between the road surface and the object of attention. Furthermore, known template matching processing, feature point detection processing, etc. may also be used for detection. Furthermore, the image recognition processing for the captured image is not limited to the above examples, and may also be performed using, for example, machine learning.

[0045] Furthermore, ultrasonic sensors 9A-9L may also be used in addition to image recognition processing to detect objects to be warned about. Detection using ultrasonic sensors 9A-9L can identify the position of objects around the vehicle, but cannot fundamentally identify the type of detected object, i.e., whether the detected object corresponds to an object to be warned about. However, for example, if the object is moving, it is possible to estimate that the object is an object to be warned about. Furthermore, cameras may also be used in combination to identify the type of object detected by ultrasonic sensors 9A-9L.

[0046] For example, FIG. 6 shows examples of bird's-eye view images 61 and overhead view images 62 generated in S2. In the example shown in FIG. 6, a person 63 located to the left front of the vehicle is detected as a cautionary object. A problem with the prior art is that when a bird's-eye view image or a bird's-eye view image is generated by changing the viewpoint of a captured image captured by a camera, three-dimensional objects in the captured image are distorted or stretched relative to their original shape. For example, in the example shown in FIG. 6, the person 63 is displayed significantly stretched vertically, making it difficult for a user to determine whether or not it is a person even when viewing the bird's-eye view image 61 or the overhead view image 62. Furthermore, when the person 63 is located near the synthesis boundary of the captured images, the person 63 may disappear from the generated bird's-eye view image 61 or the overhead view image 62. In such cases, it is difficult to detect the person using image recognition processing, but it is possible to detect the person using ultrasonic sensors 9A to 9L.

[0047] Thereafter, in S4, CPU 31 determines whether or not an attention object is included in at least one of bird's-eye image 61 and overhead image 62 generated in S2 as a result of the image recognition in S3. Note that, since the display ranges of bird's-eye image 61 and overhead image 62 do not completely match, the attention object may be included in only one of the images depending on its position. Also, as described above, if the attention object is located near the synthesis boundary, a person 63 that actually exists may disappear in bird's-eye image 61 or overhead image 62. However, if it can be detected that the attention object is located within the display range of bird's-eye image 61 or overhead image 62, it is determined that the attention object is included in bird's-eye image 61 or overhead image 62 even if it is not displayed.

[0048] If it is determined that the attention object is included in at least one of the bird's-eye image 61 and the overhead image 62 generated in S2 (S4: YES), the process proceeds to S6. On the other hand, if it is determined that the attention object is not included in either the bird's-eye image 61 or the overhead image 62 generated in S2 (S4: NO), the process proceeds to S5.

[0049] In S5, the CPU 31 displays on the liquid crystal display 4 a real-time bird's-eye view image 61 and an overhead view image 62, which are generated in S2 and show the current environment around the vehicle, as support images for supporting automatic driving of the vehicle. However, it is not necessary to display both the bird's-eye view image 61 and the overhead view image 62 at the same time, and the bird's-eye view image 61 and the overhead view image 62 may be switched and displayed by a user operation. Then, the process proceeds to S11.

[0050] Meanwhile, in S6, the CPU 31 acquires a model image 65 showing the appearance of the attention object included in the bird's-eye view image 61 and the overhead view image 62. Here, the model image 65 is a three-dimensional polygon image. FIG. 7 shows an example of a model image 65 of a person, which is a 3D image showing the appearance of a person. Note that the model image 65 may be, for example, a "wireframe model" that displays only the edges, or a "surface model" that displays the faces.

[0051] As shown in FIG. 7, the human model image 65 is recognizable as a person, but is intentionally designed with a simplified shape that makes it difficult to distinguish between front and back (which is the front and which is the back). Similarly, the human model image 65 is intentionally designed with a simplified shape that makes it difficult to distinguish between left and right (which is the right and which is the left). On the other hand, the human model image 65 is designed with a simplified shape that makes it difficult to distinguish between front and back and left and right. By using such a simplified shape, when the model image 65 is synthesized with the bird's-eye view image 61 or the overhead view image 62, as described below, no sense of incongruity is created even if the orientation of the actual person and the orientation of the model image 65 differ. Furthermore, when the person moves, the model image 65 also moves, but this prevents the orientation of the model image 65 from matching the direction of movement, resulting in an unnatural display.

[0052] Furthermore, various material settings and mapping processes may be performed on the model image 65. Examples of mapping processes include texture mapping, which applies a texture image to the surface of an object, and bump mapping, which changes the direction of light reflection to create fine irregularities. For example, a texture image is an image depicting effects such as shadow, transparency, and reflection. When the model image 65 is placed on the bird's-eye view image 61 or the overhead view image 62 in S7 (described later), the effects of the texture image are drawn so as not to appear unnatural, taking into consideration the orientation of the placement, the brightness of the surroundings, and the position of a light source (e.g., sunlight or streetlights). Furthermore, it is desirable that the display color of the texture image be a color (e.g., white, yellow, red, etc.) whose brightness difference or contrast ratio with the background color (the color of the road surface or soil) is greater than or equal to a threshold value.

[0053] 7 by performing processes such as modeling, scene layout setting, and rendering using the information stored in the model image DB 36, or alternatively, a completed model image 65 may be stored in the model image DB 36 in advance, and the model image 65 may be acquired by reading it from the model image DB 36 in S6. When creating the model image 65, it is preferable to determine the types of attention objects included in the bird's-eye image 61 and the overhead image 62 by the image recognition process described above, and create the corresponding type of model image 65. On the other hand, when reading out the model image 65, it is preferable to store a plurality of model images 65 in advance in the model image DB 36, sorted by type of attention object, and similarly determine the types of attention objects included in the bird's-eye image 61 and the overhead image 62 by the image recognition process described above, and read out the corresponding type of model image 65.

[0054] Thereafter, in S7, the CPU 31 composites the model image 65 acquired in S6 in accordance with the position of the attention object included in the bird's-eye image 61 and the overhead image 62. That is, the composite is performed so that the actual attention object and the model image 65 are superimposed. However, as described above, since the attention object included in the bird's-eye image 61 and the overhead image 62 may be stretched or distorted, it is preferable to composite the images so that the contact point between the road surface and the model image 65 is aligned with the contact point between the road surface and the attention object, rather than aligning the entire image. Alternatively, it is possible to measure the position coordinates (relative position with respect to the vehicle) of the attention object and composite the model image 65 at the measured position coordinates. In particular, if a person 63 is located near the composite boundary of the captured images, the person 63 may disappear from the generated bird's-eye image 61 or the overhead image 62. However, by compositing the model image 65 in accordance with the position coordinates, it is possible to superimpose the model image 65 in accordance with the position of the attention object that is actually present but not displayed. The bird's-eye view image 61 and the overhead view image 62 after the model image 65 has been synthesized will be referred to as a support image 66 hereinafter.

[0055] Furthermore, in S7, the CPU 31 performs editing including at least one of enlarging, reducing, and rotating the model image 65 when synthesizing the model image 65. Specifically, when synthesizing the model image 65 with the bird's-eye image 61, the model image 65 is first enlarged or reduced to match the position of the attention object detected by the image recognition processing in S3. Specifically, the closer the attention object is to the virtual viewpoint of the bird's-eye image 61, the larger the model image 65 is. Conversely, the farther the attention object is from the virtual viewpoint of the bird's-eye image 61, the smaller the model image 65 is. For example, in the case of a human model image 65, the size is adjusted to an estimated average height of a person as seen from the virtual viewpoint. Furthermore, the model image 65 is rotated to match the position and line of sight of the virtual viewpoint of the bird's-eye image 61. In other words, the angle is adjusted to match the angle at which the model image 65 would be seen from the virtual viewpoint if it were located on the road surface at the same position as the attention object. Furthermore, if the orientation of the attention object can be detected, the model image 65 may be rotated to match the orientation of the attention object. On the other hand, when combining the model image 65 with the overhead image 62, only the rotation process is performed without the above-mentioned scaling. Because the overhead image 62 is an image viewed vertically from above, the angle of the model image 65 standing on the road surface is also adjusted to that of a view viewed vertically from above. However, as mentioned above, the model image 65 has a simple shape that makes it difficult to distinguish between front and back or left and right, so the rotation process in the direction parallel to the road surface may be omitted.

[0056] If the bird's-eye view image 61 or the overhead view image 62 includes a plurality of attention objects, the processes in S6 and S7 are performed for each attention object.

[0057] Then, in S8, CPU 31 displays an image obtained by combining real-time bird's-eye view image 61 and overhead view image 62 showing the environment around the vehicle at the current time, generated in S7, with model image 65, on liquid crystal display 4 as support image 66 for supporting automatic driving of the vehicle. However, it is not necessary to display both bird's-eye view image 61 and overhead view image 62 at the same time, and bird's-eye view image 61 and overhead view image 62 may be switched between and displayed by a user operation.

[0058] 8 is a diagram showing an example of a support image 66 displayed on the liquid crystal display 4. In the example shown in Fig. 8, a model image 65 is displayed superimposed on a person 63 located to the left front of the vehicle. Note that the orientation of the model image 65 displayed differs between the bird's-eye view image 61 and the overhead view image 62; in the bird's-eye view image 61, the model image 65 is visible when looking down diagonally from a virtual viewpoint, while in the overhead view image, the model image 65 is visible when looking down vertically from the virtual viewpoint.

[0059] As described above, the processes from S2 onwards are repeatedly executed for the most recently captured image, and the support image 66 displayed on the LCD display 4 is also updated in real time, so if the attention object moves, the position of the model image 65 included in the support image 66 will also move accordingly within the peripheral image (within the scenery), as shown in Fig. 8. On the other hand, if the attention object is stationary, the position of the model image 65 included in the support image 66 will also be fixed within the peripheral image (however, if the vehicle is moving, the position of the virtual viewpoint will change, and so the position on the screen will change), but in that case the model image 65 will be highlighted as described below.

[0060] 8, the model image 65 to be synthesized with the bird's-eye view image 61 and the overhead view image 62 is an opaque image with a transmittance of 0% so that the range of the superimposed real image cannot be seen, but the model image 65 may be a semi-transparent image. In that case, it is possible to allow the user to see the attention object in the real image.

[0061] In the bird's-eye view image 61 and the overhead view image 62 shown in FIG. 8 , a person 63 located to the left front of the vehicle is displayed enlarged and distorted, making it difficult to grasp the presence and location of the person 63. However, by superimposing a model image 65 resembling the person 63 on the position of the person 63 in the support image 66, the presence and location of the person 63 can be clearly identified and grasped by the user. In particular, if the person 63 is located near the synthesis boundary of the captured images, the actual person 63 may disappear from the generated bird's-eye view image 61 or the overhead view image 62. Even in this case, however, displaying the model image 65 allows the user to grasp the presence and location of the person 63. Furthermore, the support image 66 displayed on the LCD display 4 allows the user to clearly grasp the environment around the vehicle, including areas that are difficult to see directly, by clarifying the relative position of the person 63 with the vehicle's position. The CPU 31 may also calculate the vehicle's future movement trajectory based on detection values ​​from a vehicle speed sensor, a steering sensor, etc., and display the movement trajectory superimposed on the support image 66. Thereafter, the support image 66 continues to be displayed until the assisted driving by the automatic driving assistance is ended (S11: YES).

[0062] Thereafter, in S9, the CPU 31 determines whether the object of attention included in the bird's-eye view image 61 or the overhead view image 62 is stationary. For example, by comparing the position of the object of attention detected between previous and next frames, it is possible to determine whether the object of attention is moving or stationary. Alternatively, the ultrasonic sensors 9A to 9L may be used to determine whether the object of attention is moving or stationary.

[0063] If it is determined that the attention object included in the bird's-eye image 61 or the overhead image 62 is stationary (S9: YES), the process proceeds to S10. On the other hand, if it is determined that the attention object included in the bird's-eye image 61 or the overhead image 62 is moving (S9: NO), the process proceeds to S11. If the attention object moves, the position of the model image 65 included in the support image 66 will also move accordingly within the peripheral image, as shown in FIG.

[0064] In S10, the CPU 31 highlights the model image 65 corresponding to the attention object determined to be in a stationary state in the support image 66 displayed on the liquid crystal display 4. Here, examples of methods for highlighting the model image 65 include (A) rotating the model image 65, (B) moving (jumping) the model image 65 up and down (vertical direction), (C) enlarging or reducing the size of the model image 65, (D) changing the display color of the model image 65, and (E) blinking the model image 65.

[0065] In the following, the example of "(A) rotating the model image 65" will be described in particular. As shown in FIG. 9, the model image 65 is rotated around a vertical axis while its position in the peripheral image is fixed (however, if the host vehicle is moving, the position of the virtual viewpoint changes, and therefore the position on the screen changes). The direction of rotation is not limited, and may be clockwise or counterclockwise. Human perception tends to make it more difficult to recognize stationary objects compared to moving objects. When the model image 65 is stationary, it blends into the background, causing the user to fail to notice the presence of the model image 65, i.e., the presence of an attention object around the host vehicle. However, in this embodiment, the highlighting described above allows the user to more reliably recognize even stationary attention objects. The highlighting of the model image 65 in S10 is continued until the attention object ceases to be stationary (starts to move) or disappears off the screen. However, the highlighting may be performed only for a predetermined period of time after the highlighting is started, regardless of whether the attention object ceases to be stationary.

[0066] Furthermore, a stationary attention object may subsequently move in any direction. However, if the model image 65 is displayed fixedly, the user may be given the impression that the object will only move in the direction of the displayed model image 65, as shown in Figure 10. On the other hand, if the model image 65 is rotated, it is possible to suggest to the user that the object may move in any direction within 360 degrees.

[0067] Furthermore, in the overhead image 62, if the method "(B) moving (jumping) the model image 65 up and down" is used, there is a risk that the displayed model image 65 will not move, but if the method "(A) rotating the model image 65" is used, it is possible to cause movement in the model image 65.

[0068] Note that even during automated driving assistance, vehicle occupants can cancel the automated driving assistance at any time of their own volition and can also apply the brakes to stop the vehicle. The vehicle occupants can also interrupt automated driving assistance and stop the vehicle by applying the brakes if necessary while viewing the assistance image 66 displayed on the LCD display 4.

[0069] Thereafter, in S11, the CPU 31 determines whether or not to end assisted driving by the automatic driving assistance. Here, the condition for ending assisted driving by the automatic driving assistance is, for example, in the case of parking assistance, completion of parking in a parking space. Alternatively, the condition may be that the user has performed a predetermined ending operation on the operation unit 3, or that the shift position has been shifted to "P" or the engine has been turned off. Alternatively, the condition for ending may be that a situation has arisen in which it is no longer possible to continue the automatic driving assistance.

[0070] If it is determined that the assisted driving by the autonomous driving assistance has ended (S11: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the assisted driving by the autonomous driving assistance has not ended (S11: NO), the process returns to S2, and the display of the assistance image on the LCD display 4 continues.

[0071] As explained in detail above, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to this embodiment detect an object of attention that is in the vicinity of the vehicle and that can be in at least one of a moving state and a stationary state and that is a focus of attention for the vehicle (S3), obtain a model image 65 showing the appearance of the object of attention (S6), generate a support image 66 by combining the model image 65 with a vehicle surroundings image showing the periphery of the vehicle in accordance with the position of the object of attention (S7), and display the generated support image 66 on the liquid crystal display 4 (S8). Meanwhile, if the object of attention is in a stationary state, the model image 65 corresponding to the object of attention is highlighted (S10), thereby enabling the user to more reliably recognize objects of attention that are in a stationary state. Furthermore, as a highlighting display, the model image 65 corresponding to the attention object is rotated around a vertical axis while being fixed in position, so it is possible to suggest to the user that the attention object that is stationary may move in any direction afterwards. Also, it is possible to make the displayed model image 65 appear to move even when an overhead image is displayed. Furthermore, the image of the vehicle's surroundings displayed on the liquid crystal display 4 includes at least one of a bird's-eye view image 61 looking diagonally down at the vehicle's surroundings from a virtual viewpoint in the sky, and an overhead view image 62 looking vertically down at the vehicle's surroundings from a virtual viewpoint in the sky, so the user can view the vehicle's surroundings from various viewpoints and grasp the environment around the vehicle by clarifying the relative relationship with the vehicle's position, including areas that are difficult to view directly. Furthermore, the vehicle surroundings image displayed on the liquid crystal display 4 is a captured image of the vehicle's surroundings or an image obtained by processing the captured image, and a support image 66 is generated by combining a model image 65 with the captured image in accordance with the position of the object of attention, so that even if the object of attention in the captured image is displayed distorted or stretched compared to its original shape, the user can be made to understand the existence and position of the object of attention. In particular, when capturing images are combined, if the object of attention is located near the combining boundary, the object that actually exists may disappear in the combined captured image. However, even in this case, the user can be made to understand the existence and position of the object of attention by displaying the model image.

[0072] The present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications are possible within the scope of the present invention. For example, in this embodiment, the vehicle surroundings image displayed on the LCD display 4 is a bird's-eye view image 61 and an overhead view image 62 generated by processing images captured by the front, rear, left, and right cameras. However, the images captured by the cameras themselves may also be used. For example, the images may be images captured by the front camera 6. In that case, the processing in S2 is unnecessary. In addition, in S3, an attention object is detected using the image captured by the front camera 6. Specifically, if the category of an object determined to be included in the captured image in the image displayed on the LCD display 4 matches the category of an attention object, the object is detected as an attention object. In addition, in S7, a model image 65 is synthesized with the captured image. In addition, the viewpoint of the vehicle surroundings image displayed on the LCD display 4 at this time is the viewpoint of the camera.

[0073] Furthermore, the vehicle surroundings image displayed on the liquid crystal display 4 may not be an image captured by a camera, but may be a virtual landscape image in which the surroundings of the vehicle are reproduced using CG. For example, a three-dimensional map image of the area around the current position may be acquired or generated and displayed on the liquid crystal display 4, and a model image 65 may be superimposed on the displayed three-dimensional map image at a position where an object of attention exists, and displayed as the support image 66. Even in this case, the effect of allowing the user to more reliably recognize an object of attention that is in a stopped state can be achieved.

[0074] Furthermore, in this embodiment, the model image 65 is synthesized with both the bird's-eye view image 61 and the overhead view image 62, but the model image 65 may be synthesized with only one of the images.

[0075] In this embodiment, it is assumed that the support image 66 shown in Fig. 8 is displayed while the vehicle is being driven with automatic driving assistance, but the support image 66 shown in Fig. 8 or 9 may also be displayed while the vehicle is being driven manually. In this case, the determination condition for S1 is that the user has instructed the display of the support image or has performed an operation on the vehicle, such as switching the shift lever.

[0076] 8 and 9 have been used to explain display examples of model image 65 when a person 63 is present as an object to be warned about, but when a bicycle is present as an object to be warned about, a simple model image of a person may also be used to display the image, just as when a person 63 is present. A model image specific to bicycles may also be a model image of just the bicycle, or a model image of a person riding a bicycle that is more realistic. In the case of automobiles, a model image of the automobile is used, but a simple image in which there is no distinction between front and rear or left and right, as with people, may also be used.

[0077] In this embodiment, the driving assistance processing program (FIG. 3) is executed by the driving assistance ECU 10 of the driving assistance device 1, but the execution entity can be changed as appropriate. For example, the program may be executed by the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle device. [Explanation of symbols]

[0078] 1... driving assistance device, 2... vehicle, 3... operation unit, 4... liquid crystal display (display device), 6... front camera, 7... rear camera, 8A, 8B... side cameras, 10... driving assistance ECU (an example of an object detection means, a model image acquisition means, an assistance image generation means, an image display means, a highlight display means, and a captured image acquisition means), 31... CPU, 61... bird's-eye view image, 62... bird's-eye view image, 63... person (an example of an object to be careful of), 65... model image, 66... ​​assistance image

Claims

1. an object detection means for detecting an attention object that is located around the vehicle and can be in at least one of a moving state and a stopped state and that is an attention object for the vehicle; a model image acquisition means for acquiring a model image showing the appearance of the attention object; a support image generating means for generating a support image by combining the model image with a vehicle surroundings image showing the surroundings of the vehicle in accordance with the position of the attention object; an image display means for displaying the support image on a display device; and highlighting means for highlighting the model image corresponding to the attention object when the attention object is in a stationary state.

2. 2. The driving assistance device according to claim 1, wherein the highlighting means, when the object of attention is in a stationary state, highlights the model image corresponding to the object of attention by rotating the model image around a vertical axis of rotation while keeping the position fixed.

3. 2. The driving assistance device according to claim 1, wherein the vehicle surroundings image includes at least one of a bird's-eye view image obtained by looking down diagonally on the surroundings of the vehicle from a virtual viewpoint in the sky, and an overhead view image obtained by looking down vertically on the surroundings of the vehicle from a virtual viewpoint in the sky.

4. an image acquisition means for acquiring an image of the periphery of the vehicle; The vehicle surroundings image is the captured image or an image obtained by processing the captured image, 4. The driving support device according to claim 1, wherein the support image generating means generates the support image by combining the model image in accordance with the position of the attention object in the captured image.

Images