Driving assistance device

Abstract

The invention provides a driving support device which can reliably recognize an object to be noticed in a stopped state by a user when guiding the object to be noticed by using an image around a vehicle. The device is configured to: detect an object to be noticed that is located in the vicinity of a vehicle, can take at least one of a moving state and a stopped state, and is an object to be noticed to the vehicle; acquiring a model image (65) representing the appearance of the object to be noticed; generating an auxiliary image (66) in which the model image (65) is synthesized in a vehicle periphery image indicating the periphery of the vehicle on the basis of the position of the object to be noticed; when the object to be noticed is in a stopped state, the generated auxiliary image (66) is displayed on the liquid crystal display (4), and when the object to be noticed is in the stopped state, the model image (65) corresponding to the object to be noticed is displayed in an enhanced manner.

Description

Technical Field

[0001] This invention relates to a driving assistance device for assisting the driving of a vehicle. Background Technology

[0002] In the past, various mechanisms have been used to provide information to assist vehicle drivers, such as providing route guidance and obstacle warnings to passengers. These include displays on the vehicle's LCD screen and sound output from speakers. However, blind spots exist around the vehicle, particularly during special operations like parking or exiting a parking space. To help the driver understand these blind spots, conventional methods involved displaying an image of the vehicle's surroundings on the LCD screen.

[0003] In addition, the following technology has been proposed: when displaying images of the vehicle's surroundings, if there are objects of attention around the vehicle that are objects of attention for the vehicle, the images of the vehicle's surroundings are used to help the user recognize the presence of these objects of attention. For example, Japanese Patent Application Publication No. 2005-5978 discloses a method that uses simple graphics to depict the situation viewed from the driver's seat of the vehicle, and renders other vehicles and people within the display range.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2005-5978 (paragraphs 0061-0062) Figure 8 )

[0005] The rendering depicted in Patent Document 1 above allows for the depiction of a model image representing the appearance of an object of attention within an image of the vehicle's surroundings. However, the movement of the model image is linked to the actual movement of the object of attention. Therefore, for example, when the object of attention is stationary, the model image also becomes stationary. In terms of human perception, stationary objects are more difficult to recognize than moving ones. Therefore, when the model image is stationary and blends into the background, the user may not notice its presence, i.e., fail to perceive the existence of an object of attention around the vehicle. Furthermore, compared to moving objects of attention, it is difficult to predict in advance which direction a stationary object of attention will move if it subsequently begins to move. Therefore, it is important for the user to recognize the presence of stationary objects of attention. Summary of the Invention

[0006] The present invention was made to solve the above-mentioned problems and aims to provide a driving assistance device that, when using images of the vehicle's surroundings to guide attention to objects, can reliably identify objects that are in a stationary state.

[0007] To achieve the above objectives, the driving assistance device of the present invention includes: an object detection mechanism for detecting an object of attention located around a vehicle that is either in a moving state or a stationary state and constitutes an object of attention to the vehicle; a model image acquisition mechanism for acquiring a model image representing the appearance of the object of attention; an auxiliary image generation mechanism for generating an auxiliary image obtained by synthesizing the model image in a vehicle perimeter image representing the vehicle perimeter based on the position of the object of attention; an image display mechanism for displaying the auxiliary image on a display device; and an enhanced display mechanism for enhancing the display of the model image corresponding to the object of attention when the object of attention is in a stationary state.

[0008] Furthermore, "images representing the vehicle's surroundings" can be either real-world images captured by a camera or other imaging device, or imaginary images created by CG (computer graphics) to recreate the vehicle's surroundings. In the case of real-world images, these can be the captured image itself or processed images. For example, they can be images synthesized from multiple cameras or images obtained by shifting the viewpoint of multiple cameras.

[0009] According to the driving assistance device of the present invention having the above structure, when guiding the attention object using images of the vehicle's surroundings, a model image representing the appearance of the attention object is displayed according to the position of the attention object, and especially when the attention object is in a stationary state, the model image corresponding to the attention object is enhanced, so that the user can reliably identify the attention object even when it is stationary. Attached Figure Description

[0010] Figure 1 This is a simplified structural diagram of the vehicle according to this embodiment.

[0011] Figure 2 This is a block diagram showing the structure of the driving assistance device in this embodiment.

[0012] Figure 3 This is a flowchart of the driving assistance processing procedure in this embodiment.

[0013] Figure 4 This diagram illustrates the methods for converting captured images into bird's-eye view and overhead view images.

[0014] Figure 5A , Figure 5B These figures illustrate the methods for generating bird's-eye view and aerial view images, respectively.

[0015] Figure 6It is a diagram that illustrates the objects of attention contained in bird's-eye view and overhead view images.

[0016] Figure 7 This is a diagram representing an example of a model image.

[0017] Figure 8 It is a diagram representing an auxiliary image obtained by synthesizing a model image from a bird's-eye view and a bird's-eye view.

[0018] Figure 9 This is a diagram showing how the model image is displayed when the object of attention is in a stationary state.

[0019] Figure 10 This is a diagram illustrating an example of enhanced display of a model image.

[0020] Explanation of reference numerals in the attached figures

[0021] 1…Driver assistance device; 2…Vehicle; 3…Operating unit; 4…Liquid crystal display (display device); 6…Front-facing camera; 7…Rear-facing camera; 8A, 8B…Side cameras; 10…Driver assistance ECU (an example of an object detection mechanism, model image acquisition mechanism, auxiliary image generation mechanism, image display mechanism, enhanced display mechanism, or image acquisition mechanism); 31…CPU; 61…Bird's-eye view; 62…Overhead view; 63…Person (an example of an object); 65…Model image; 66…Auxiliary image. Detailed Implementation

[0022] Hereinafter, one embodiment of the driving assistance device of the present invention will be described in detail with reference to the accompanying drawings. First, the vehicle 2 equipped with the driving assistance device 1 of this embodiment will be described. Figure 1 This is a simplified structural diagram of vehicle 2 in this embodiment.

[0023] Here, vehicle 2 can be, for example, a car powered by an internal combustion engine (engine, etc.) (internal combustion engine car), a car powered by an electric motor (motor, etc.) (electric car, fuel cell car, etc.), or a car powered by both (hybrid car). Furthermore, regardless of the vehicle type, it can be a regular car, or a large commercial truck, bus, construction machinery, etc. Although described below as a four-wheeled vehicle, it can also be a two-wheeled or three-wheeled vehicle.

[0024] Among them, vehicle 2 is a vehicle that can perform assisted driving, which is an automatic driving vehicle that can drive automatically regardless of the user's driving operation, except for manual driving based on the user's driving operation.

[0025] Furthermore, autonomous driving assistance can be implemented only in specific situations such as parking or exiting a parking space, or it can be implemented across all road sections, or it can be configured to only be implemented while the vehicle is traveling on a specific road section (e.g., a highway with gates at the boundary (regardless of whether there are people or no people, or whether it is a toll road or a free road)). In the following description, the autonomous driving area for which autonomous driving assistance is implemented includes all road sections including ordinary roads and highways, as well as parking lots, and it is implemented only when the user selects to implement autonomous driving assistance (e.g., turns on the autonomous driving start button) and it is determined that driving based on autonomous driving assistance is possible. On the other hand, vehicle 2 can also be configured to be a vehicle that can only drive with assistance based on autonomous driving assistance. Alternatively, it can be configured to only target the vehicle's movement from parking to the parking space (i.e., parking assistance) for driving with assistance based on autonomous driving assistance.

[0026] Furthermore, in the vehicle control of the automated driving assistance system in this embodiment, for example, the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles are detected at any time, and the vehicle travels along a generated travel trajectory at a speed planned according to the same generated speed, and the vehicle control of the steering gear, drive system, brakes, etc. is performed automatically. In particular, in the case of parking assistance, the vehicle control is performed automatically as follows: using the detection results of sensors and cameras, the parking space to which the vehicle is to be parked and the surrounding conditions are confirmed, and a parking track to the parking space is calculated, and the vehicle enters the parking space along the calculated parking track and ends the parking. Moreover, in addition to the above-mentioned general parking assistance, for example, the vehicle control is performed automatically as follows: for long-distance parking targeting distant parking spaces such as one's own garage or a monthly parking space in a parking lot, the vehicle moves to the pre-set distant parking space and ends the parking. In the above-mentioned parking assistance, only the steering operation can be performed automatically, and the drive system and brakes can be controlled based on manual operation. Alternatively, it may only guide the vehicle to the parking space via a parking track or guide the vehicle operation, allowing the user to manually park the vehicle in the parking space.

[0027] Furthermore, during the automatic control of the vehicle, including steering, drive system, and brakes, passengers can cancel the automated driving assistance at any time and also apply the brakes to bring the vehicle to a stop. When the aforementioned automated driving assistance is in operation, as described later, the surrounding scenery (which can be real or CG-generated) is displayed on the in-vehicle screen, and warnings are given to guide attention to objects that are of interest to the vehicle. Passengers can identify the display and, if necessary, can also interrupt the automated driving assistance and bring the vehicle to a stop by applying the brakes.

[0028] In addition, such as Figure 1 The vehicle 2 shown includes: an operation unit 3 that receives operations from passengers; a liquid crystal display 4 that displays bird's-eye view, overhead view, and other driving assistance information of the vehicle's surroundings to passengers; a speaker 5 that outputs voice guidance related to driving assistance; a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the vehicle's surroundings; ultrasonic sensors 9A to 9L for detecting obstacles around the vehicle; and a driving assistance ECU (electronic control unit) 10 that performs various calculations based on the input information. Furthermore, a driving assistance device 1 is defined by including various structural components, represented by the aforementioned driving assistance ECU 10.

[0029] The following describes the various structural components of vehicle 2. First, the operation unit 3 is located, for example, in front of the steering wheel (also called the steering wheel) and includes operation buttons that are activated when autonomous driving assistance is initiated. By operating the operation unit 3, the user can switch between manual driving based on the user's driving operations and assisted driving performed automatically by the vehicle without the user's driving operations. Furthermore, the operation unit 3 may also have a touch panel located in front of the LCD display 4. Additionally, it may also include a microphone and a voice recognition device.

[0030] The LCD display 4 is installed on the instrument panel of vehicle 2, displaying bird's-eye view and overhead view images of the vehicle's surroundings, generated by viewpoint conversion and compositing of images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B during autonomous driving assistance. Furthermore, when there are objects of attention such as pedestrians or bicycles around vehicle 2, a model image representing the appearance of these objects is displayed by overlaying their positions in the bird's-eye view and overhead view images. Additionally, the LCD display 4 can also function as a component of a navigation device.

[0031] Additionally, speaker 5 is installed in the instrument panel of vehicle 2, outputting guidance sounds, warning sounds, etc., related to driver assistance. Speaker 5 can also function as a component of the navigation system.

[0032] In addition, the front camera 6 is, for example, a camera device that uses a solid-state imaging element such as a CCD, and is positioned above the front bumper of the vehicle 2 or inside the rearview mirror, with the optical axis pointing in the direction of the vehicle's travel.

[0033] The rear camera 7 is an imaging device that also uses a camera with a solid-state imaging element such as a CCD. For example, it is installed near the center of the license plate number installed at the rear of the vehicle 2, and the optical axis is set to face the rear of the vehicle.

[0034] Furthermore, the side cameras 8A and 8B are imaging devices that also use solid-state imaging elements such as CCDs, for example, mounted on the left and right rearview mirrors of vehicle 2, and set with the optical axis pointing towards the side of the vehicle.

[0035] Furthermore, the driver assistance ECU 10 generates bird's-eye view and overhead view images of the vehicle's surroundings by performing viewpoint conversion and synthesis processing on the images captured by the aforementioned front camera 6, rear camera 7, and side cameras 8A and 8B. Additionally, during automated driving assistance execution, image recognition processing is performed on the captured images to detect lane markings and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and automated driving assistance is executed based on the detection results. In particular, in the case of parking assistance, the obstacle detection results from the aforementioned cameras are used to confirm the parking space and its surrounding conditions.

[0036] On the other hand, ultrasonic sensors 9A to 9L are respectively arranged at predetermined intervals at the front, rear, and sides of the vehicle. They transmit ultrasonic waves as detection waves to the vicinity of the vehicle 2 and receive reflected waves after the transmitted detection waves are reflected by objects around the vehicle, thereby detecting objects that reflected the detection waves. Specifically, a ranging sensor is a ranging sensor that can detect the distance (range value) of an object that reflected the detection waves by measuring the time from transmission to reception. In addition, ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. In addition, objects that are detected by ultrasonic sensors 9A to 9L can include obstacles that the vehicle 2 needs to avoid when driving, such as people, bicycles, other vehicles, walls, etc., or obstacles that form parking spaces. In addition, millimeter-wave sensors or laser sensors can be used instead of ultrasonic sensors as ranging sensors.

[0037] Furthermore, while the placement and orientation of each ultrasonic sensor 9A-9L can be appropriately set, in this embodiment, to cover the entire area in front of, behind, and to the left and right sides of the vehicle 2's direction of travel as the detection range for the object, ultrasonic sensors 9A-9D are positioned in front of the vehicle 2, with the direction of their detection waves aligning with the direction of travel. Ultrasonic sensors 9E and 9F are positioned on the left side of the vehicle 2, with the direction of their detection waves aligning with the left side of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are positioned on the right side of the vehicle 2, with the direction of their detection waves aligning with the right side of the vehicle's direction of travel. Ultrasonic sensors 9I-9L are positioned behind the vehicle 2, with the direction of their detection waves aligning with the rear of the vehicle, in the opposite direction of travel. All ultrasonic sensors 9A-9L are positioned at the same height above the ground surface.

[0038] Furthermore, in this embodiment, among the ultrasonic sensors 9A to 9L, particularly the ultrasonic sensors 9A to 9D at the front of vehicle 2 and the ultrasonic sensors 9I to 9L at the rear of vehicle 2, are positioned so that reflected waves can be received indirectly between adjacent sensors. Therefore, by receiving both direct and indirect waves as receiving waves, triangulation can be used to determine not only the distance to the object but also the specific position of the object (relative to the vehicle). The ultrasonic sensors 9E to 9H are positioned separately on the sides. Although indirect wave reception is not possible with these sensors, the specific position of the object (relative to the vehicle) can still be determined by triangulation using the distance measured at the previous position, the distance measured at the current position, and the distance traveled between them, based on the movement of the vehicle.

[0039] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it controls the vehicle's steering, drive, and braking systems by constantly detecting the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, and driving along a generated driving trajectory at a speed planned based on the same generated speed. In particular, in the case of parking assistance, the vehicle control is performed as follows: using the detection results of the front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L, the parking space to which the vehicle will be parked and the surrounding conditions are confirmed, and a parking trajectory to the parking space is calculated. The vehicle is then driven into the parking space along the calculated parking trajectory, and the parking is completed. On the other hand, in the case of parking assistance for long-distance parking, the vehicle control is performed as follows: calculating a parking trajectory that includes movement to a long-distance parking space, such as a pre-registered garage or a parking space with a monthly parking contract, and the vehicle is driven into the parking space along the calculated parking trajectory, and the parking is completed. Furthermore, the LCD display 4 displays bird's-eye view and overhead view images of the vehicle's surroundings generated from images captured by the aforementioned cameras. Especially when there are pedestrians, bicycles, or other objects of attention around the vehicle that are objects of attention for the vehicle, the display also shows a model image representing the appearance of the object of attention by overlaying its position within the bird's-eye view or overhead view, or outputs a warning sound. The driver assistance ECU 10 is connected to the aforementioned operating unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L via a vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyroscope sensor, steering sensor, and shift position sensor, as well as a navigation device that serves as an in-vehicle unit. The detailed structure of the driver assistance ECU 10 will be described later.

[0040] In addition, vehicle 2 Figure 1 In addition to the structural components shown, there are also basic structural components for vehicle 2, but only the structures related to the control of autonomous driving assistance and the control related to these structures are described.

[0041] Next, the details of the driving assistance device 1, particularly the driving assistance ECU 10, which is equipped in the aforementioned vehicle 2 will be explained. Figure 2 This is a block diagram showing the structure of the driving assistance device 1 in this embodiment.

[0042] like Figure 2As shown, the driving assistance ECU (electronic control unit) 10 is an electronic control unit that performs overall control of the driving assistance device 1. In addition to the CPU 31, which serves as both a computing and control unit, and the RAM 32, which is used as working memory during various calculations by the CPU 31 and stores driving trajectory data such as those used to calculate the driving trajectory, and the control program, it also includes a driving assistance processing program (see reference 10). Figure 3 The ECU 10 includes internal storage devices such as a ROM 33 and a flash memory 34 storing programs read from the ROM 33. Furthermore, the ECU 10 has various mechanisms that function as processing algorithms. For example, an object detection mechanism detects objects of attention located around the vehicle that are either moving or stationary and are objects of attention for the vehicle. A model image acquisition mechanism acquires a model image representing the appearance of the object of attention. An auxiliary image generation mechanism generates an auxiliary image by synthesizing a model image from a vehicle perimeter image representing the vehicle's perimeter based on the position of the object of attention. An image display mechanism displays the auxiliary image on a liquid crystal display 4. An enhanced display mechanism enhances the display of the model image corresponding to the object of attention when the object of attention is stationary. An image acquisition mechanism acquires an image of the vehicle's perimeter. In other words, the ECU 10 is an example of an object detection mechanism, a model image acquisition mechanism, an auxiliary image generation mechanism, an image display mechanism, an enhanced display mechanism, and an image acquisition mechanism.

[0043] Furthermore, the driver assistance ECU 10 is also connected to various sensors 37 used to detect the vehicle's movement, such as vehicle speed sensors, wheel speed sensors, acceleration sensors, gyroscope sensors, steering sensors, and shift position sensors, as well as various drive units 38 of the vehicle, such as the steering gear, brakes, accelerator, and transmission. Based on the detection results of the aforementioned sensors 37, it detects the vehicle's current movement and controls each drive unit 38, thereby implementing autonomous driving assistance for the vehicle 2. Specific aspects of autonomous driving assistance include, for example, continuously detecting the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, driving along a pre-generated driving trajectory at a speed planned based on a similarly generated speed, and controlling the steering gear, drive unit, and brakes. Alternatively, it may automatically perform steering operations and control the drive unit and brakes based on manual operation.

[0044] Additionally, the flash memory 34 contains vehicle information DB35 and model image DB36. Vehicle information DB35 stores various information related to vehicle 2. For example, it stores the installation locations (height from the ground, left and right positions) of the cameras and ultrasonic sensors 9A-9L on vehicle 2, the detection axis (related to the camera, optical axis), overall length, vehicle width, wheelbase, minimum turning radius, etc. This information is pre-input by passengers or personnel from the vehicle manufacturer.

[0045] On the other hand, the model image DB36 stores various information used when depicting a 3D model image representing the appearance of a person or bicycle. Here, in this embodiment, an object of attention that can be in at least one of a moving or stationary state—that is, an object that is either moving or stationary, not a fixed object such as a wall or utility pole—is designated as an object of attention for the vehicle. Furthermore, any object, whether moving or stationary, can be designated as an object of attention. Preferably, specific objects can be designated as objects of attention; for example, a person or bicycle. However, objects of attention are not limited to people or bicycles; for example, they can also include vehicles or motorcycles. In this case, the information used when depicting a 3D model image representing the appearance of a vehicle or motorcycle is also stored in the model image DB36.

[0046] Furthermore, the model image DB36 stores various information used when depicting 3D model images representing the appearance of people and bicycles. This includes information required for various processes in creating 3DCG, such as modeling, scene layout settings, and rendering, as well as texture images used for pasting onto the created 3DCG model. Additionally, the model image DB36 can also store model images representing the appearance of people and bicycles in their completed state. In this case, only one model image can be set for each type of object of attention, or multiple model images can be prepared for each type. For example, for people, different model images can be prepared for children and adults; for bicycles, different model images can be prepared for city bicycles and mountain bicycles. In the following description, only one model image is set for each type of object of attention.

[0047] Next, in the driving assistance device 1 having the above structure, based on Figure 3 The driving assistance processing procedures executed by the driving assistance ECU10 are explained. Figure 3 This is a flowchart of the driving assistance processing procedure in this embodiment. Here, the driving assistance processing procedure is executed after the ACC power supply of vehicle 2 is turned on, and it is a procedure that assists the user by using bird's-eye view and overhead view images of the vehicle's surroundings during the autonomous driving assistance of the vehicle.

[0048] In the following embodiments, bird's-eye view and overhead view images of the vehicle's surroundings are displayed during autonomous driving assistance. However, it is not necessary to display these images only during autonomous driving assistance; they can also be displayed during manual driving. Alternatively, bird's-eye view and overhead view images may only be displayed during specific autonomous driving assistance activities such as long-distance parking assistance. Furthermore, the following... Figure 3 The program shown in the flowchart is stored in RAM32 and ROM33 of the driving assistance device 1 and is executed by CPU31.

[0049] First, in step (hereinafter referred to as S)1, CPU31 determines whether to perform assisted driving based on autonomous driving assistance. As mentioned above, specific autonomous driving assistance such as parking assistance can also be set as the determination condition.

[0050] Furthermore, in this embodiment, when the user selects to engage autonomous driving assistance via operation unit 3 and it is determined that autonomous driving assistance-based driving is feasible, assisted driving based on autonomous driving assistance is performed. Additionally, as part of the autonomous driving assistance, the vehicle automatically controls the steering, drive system, and brakes by continuously detecting the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, traveling along a pre-generated driving trajectory at a speed planned based on a similarly generated speed. In particular, for parking assistance, vehicle control is automatically performed until the vehicle comes to a stop in a parking space. Alternatively, only steering may be performed automatically, while control of the drive system and brakes may be performed manually.

[0051] Then, if it is determined that assisted driving based on autonomous driving assistance is being performed (S1: Yes), proceed to S2. Conversely, if it is determined that assisted driving based on autonomous driving assistance is not being performed (S1: No), the driving assistance process ends.

[0052] Furthermore, the following processing is performed on an image unit captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. For example, the frame rate of the front camera 6, the rear camera 7, and the side cameras 8A and 8B in this embodiment is 30fps (30 images are captured every 1 second), so the most recently captured image is acquired in 33ms units, and the acquired image is used as the object for the following processing.

[0053] In S2, CPU31 generates, based on real-time images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B, two bird's-eye view images of the vehicle's surroundings viewed from above and from below, respectively. The following example illustrates the method for generating the bird's-eye view image of the vehicle's front. Figure 4 As shown, real-time images captured by each camera are projected onto a horizontal plane corresponding to the height of the ground surface, i.e., an imaginary projection plane. The images projected onto the imaginary projection plane are then converted into images viewed from an imaginary viewpoint, thereby generating bird's-eye view images from each camera. The imaginary viewpoint is a viewpoint tilted downwards from above vehicle 2, looking forward in the direction of travel. Furthermore, the conversion to images viewed from the imaginary viewpoint (viewpoint conversion) is performed by first converting the coordinates of the captured image coordinate system, set along a plane perpendicular to the camera's optical axis, to coordinates of a ground coordinate system set along the ground surface, and then to coordinates of the bird's-eye view image coordinate system. The conversion formulas used for each coordinate conversion are already known, so their explanation is omitted. Then, as... Figure 5A , Figure 5B As shown, a bird's-eye view image is generated by combining (stitching together) the bird's-eye view image 41 (viewpoint shifted from the image captured by the front camera 6), the bird's-eye view image 42 (viewpoint shifted from the image captured by the rear camera 7), the bird's-eye view image 43 (viewpoint shifted from the image captured by the side camera 8A), and the bird's-eye view image 44 (viewpoint shifted from the image captured by the side camera 8B), and then inserting a schematic image of the vehicle itself 45 between each bird's-eye view image 41 to 44. Alternatively, the hypothetical viewpoint of the bird's-eye view image can be set to the vehicle interior (i.e., the passenger's viewpoint). Furthermore, in Figure 5A , Figure 5B The bird's-eye view shown also includes a bird's-eye view 42 that has undergone viewpoint transformation of the image taken by the rear camera 7. However, if only images from the front of the vehicle are displayed, the bird's-eye view 42 that has undergone viewpoint transformation of the image taken by the rear camera 7 can be excluded from the composite object. In addition, the bird's-eye view is generated as the vehicle moves backward, showing a bird's-eye view tilted and looking down at the rear of the vehicle.

[0054] Furthermore, for the overhead view, only the angle of the line of sight during viewpoint transitions differs; the process is essentially the same as that for generating the bird's-eye view, therefore, explanation is omitted. The overhead view is generated by inserting an image 55 schematically representing the vehicle between each of the overhead views 51-54.

[0055] Next, in S3, the CPU 31 performs image recognition processing on the bird's-eye view and overhead view images generated in S2, thereby detecting objects of attention contained in the bird's-eye view and overhead view images respectively. Here, as mentioned above, an object of attention is an object that can be in at least one of a moving state or a stationary state and becomes an object of attention for the vehicle, such as a person or a bicycle. In addition, in this embodiment, image recognition processing is performed on the bird's-eye view and overhead view images generated in S2, but image recognition processing can also be performed on images captured by cameras that serve as the source of the bird's-eye view and overhead view images. As described above, in the detection of objects of attention in S3, objects of attention are detected using images captured by at least one of the shooting devices (front camera 6, rear camera 7, and side cameras 8A and 8B). Moreover, if the type of object included in the display image of the liquid crystal display 4 is determined to be the same as the type of object of attention, then the image captured by the shooting device is detected as an object of attention.

[0056] As part of the processing in S3 above for detecting objects of attention based on bird's-eye view and overhead view images, for example, brightness correction is performed based on the brightness difference between the road surface and the object of attention. Then, binarization processing to separate the object of attention from the image, geometric processing to correct distortion, and smoothing processing to remove image noise are performed, which can detect the boundary between the road surface and the object of attention. Furthermore, known template matching processing and feature point detection processing can also be used for detection. In addition, image recognition processing for captured images is not limited to the above examples; for example, machine learning can also be used.

[0057] In addition to image recognition processing, ultrasonic sensors 9A-9L can also be used for detecting objects of attention. However, in detection based on ultrasonic sensors 9A-9L, even if the position of objects around the vehicle can be determined, it is generally impossible to identify the type of the detected object, i.e., whether the detected object is equivalent to an object of attention. For example, simply moving the object can lead to the inference that it is an object of attention. Alternatively, a camera can be used to identify the type of object detected by ultrasonic sensors 9A-9L.

[0058] For example, Figure 6 This is an example of the bird's-eye view image 61 and the overhead view image 62 generated in S2 above. Figure 6 In the example shown, person 63, located to the left front of the vehicle, is detected as the object of attention. Here, a problem with the prior art is that when generating overhead or bird's-eye views from images captured by a camera by changing the viewpoint, three-dimensional objects in the captured images are displayed with distortion or stretching relative to their original shape. For example, in… Figure 6In the example shown, the person 63 is stretched considerably, especially along the vertical direction, making it difficult for the user to determine whether it is a person even if they can identify the bird's-eye view 61 and the overhead view 62. Furthermore, if the person 63 is located near the composite boundary of the captured images, the person 63, which actually exists in the generated bird's-eye view 61 and overhead view 62, may sometimes disappear. In such cases, detection in image recognition processing is difficult, but it can be detected using ultrasonic sensors 9A to 9L.

[0059] Then, in S4, the CPU 31 determines the result of the image recognition in S3, that is, whether at least one of the bird's-eye view image 61 and the overhead view image 62 generated in S2 contains the object of attention. Furthermore, the areas of the object of attention in the bird's-eye view image 61 and the overhead view image 62 are completely different, so depending on the position of the object of attention, it may sometimes be contained in only one image. Also, as mentioned above, when the object of attention is located near the synthesis boundary, the person 63 actually present in the bird's-eye view image 61 and the overhead view image 62 may disappear. However, as long as the object of attention can be detected to be within the display range of the bird's-eye view image 61 and the overhead view image 62, it is assumed that the object of attention is contained in the bird's-eye view image 61 and the overhead view image 62 even if it is not displayed.

[0060] Then, if it is determined that at least one of the bird's-eye view image 61 and the overhead view image 62 generated in S2 contains the object of attention (S4: Yes), proceed to S6. Conversely, if it is determined that neither the bird's-eye view image 61 nor the overhead view image 62 generated in S2 contains the object of attention (S4: No), proceed to S5.

[0061] In S5, CPU31 displays the real-time bird's-eye view image 61 and overhead view image 62, generated in S2, representing the current environment surrounding the vehicle, on the LCD 4 as auxiliary images to assist the vehicle's autonomous driving. It is not necessary for both bird's-eye view image 61 and overhead view image 62 to be displayed simultaneously; the user can switch between displaying them. Then, proceed to S11.

[0062] On the other hand, in S6, CPU31 acquires a model image 65 representing the appearance of the objects of attention contained in the bird's-eye view 61 and the overhead view 62. Here, the model image 65 is a three-dimensional polygonal image. Figure 7 As an example, a human model image 65 is shown, which becomes a 3D image representing the appearance of a person. Alternatively, the model image 65 may also be a "wireframe model" that only shows the edges, or a "surface model" that shows the faces.

[0063] In addition, such as Figure 7As shown, the human model image 65 is recognizable as a human, but due to its simplified shape, it is difficult to distinguish front from back (which is the front and which is the back). Similarly, it is also difficult to distinguish left from right (which is the right and which is the left). On the other hand, it is a shape that can be recognized in both front-back and left-right directions. By setting it to such a simple shape, as will be described later, when the model image 65 is synthesized in the bird's-eye view image 61 and the top-down view image 62, even if the actual orientation of the human is different from the orientation of the model image 65, there will be no sense of disharmony. In addition, when the human moves, the model image 65 also moves, which can prevent the model image 65 from appearing unnatural due to the inconsistency between the orientation and the direction of movement.

[0064] Additionally, various material settings and mapping processes can be applied to the model image 65. As mapping processes, there are texture mapping of images with textures pasted onto the object's surface, and bump mapping to create subtle bumps by changing the direction of light reflection. For example, a texture image depicts effects such as shadows, light transmission, and reflection. When the model image 65 is positioned in the bird's-eye view image 61 and the overhead view image 62 in S7 (described later), the orientation of the placement, the surrounding brightness, and the position of the light source (e.g., sunlight, streetlights) are considered to ensure that the various effects of the texture image are depicted in a way that does not appear unnatural. Furthermore, regarding the display color of the texture image, it is preferable to set it to a color whose brightness difference and contrast with the background color (the color of the road surface, soil) is above a threshold (e.g., white, yellow, red, etc.).

[0065] In addition, in S6 above, CPU31 can also use the information stored in the model image DB36 to perform modeling, scene layout setting, rendering and other processing to create... Figure 7 The model image 65 shown can also be a pre-completed model image 65 stored in the model image DB36, and obtained by reading the model image 65 from the model image DB36 in S6 above. Furthermore, regarding the creation of the model image 65, it is preferable to determine the types of objects of attention contained in the bird's-eye view image 61 and the overhead view image 62 through the image recognition processing described above, and create a model image 65 of the corresponding type. On the other hand, regarding the reading of the model image 65, it is preferable to pre-classify each type of object of attention, store multiple model images 65 in the model image DB36, and similarly determine the types of objects of attention contained in the bird's-eye view image 61 and the overhead view image 62 through the image recognition processing described above, and read out the model image 65 of the corresponding type.

[0066] Then, in S7, CPU31 synthesizes the model image 65 obtained in S6 based on the positions of the objects of attention contained in the bird's-eye view image 61 and the overhead view image 62. That is, the model image 65 is synthesized by overlapping the actual objects of attention with the model image 65. As mentioned above, the objects of attention contained in the bird's-eye view image 61 and the overhead view image 62 are stretched and distorted, so it is preferable to synthesize them so that the grounding points of the road surface and the model image 65 are consistent with the grounding points of the road surface and the objects of attention, rather than making the whole image consistent. Alternatively, the position coordinates of the objects of attention (relative to the vehicle) can be measured, and the model image 65 can be synthesized based on the measured position coordinates. In particular, if the person 63 is located near the synthesis boundary of the captured image, the person 63, who actually exists in the generated bird's-eye view image 61 and the overhead view image 62, may disappear. By synthesizing the model image 65 based on the position coordinates, the model image 65 can be superimposed based on the position of the objects of attention that are not displayed but actually exist. In addition, the bird's-eye view image 61 and the overhead view image 62 after synthesizing the model image 65 are referred to as auxiliary images 66 below.

[0067] Furthermore, in S7 above, the CPU 31 performs at least one editing operation, including enlarging, reducing, and rotating the model image 65, when synthesizing the model image 65. Specifically, when synthesizing the model image 65 from the bird's-eye view image 61, the model image 65 is enlarged or reduced based on the position of the object of attention detected by the image recognition processing in S3 above. Specifically, the closer the object of attention is to the hypothetical viewpoint of the bird's-eye view image 61, the larger the model image 65 becomes; conversely, the farther the object of attention is from the hypothetical viewpoint of the bird's-eye view image 61, the smaller the model image 65 becomes. For example, if it is a model image 65 of a person, the average height of a person seen from the hypothetical viewpoint is adjusted to an assumed size. In addition, rotation processing is performed based on the position and viewing direction of the hypothetical viewpoint of the bird's-eye view image 61. That is, if it is assumed that the model image 65 exists on the road surface at the same position as the object of attention, it is adjusted to the angle seen from the hypothetical viewpoint. Furthermore, if the orientation of the object of attention can be detected, the model image 65 can be rotated according to the orientation of the object of attention. On the other hand, when the model image 65 is synthesized from the overhead image 62, the above-mentioned scaling is not performed, only rotation is performed. The overhead image 62 is an image viewed from vertically, so the model image 65 standing on the road surface is also adjusted to be viewed from vertically. As mentioned above, the model image 65 has a simple shape that makes it difficult to distinguish front from back and left from right, so the rotation process in the direction parallel to the road surface can also be omitted.

[0068] In addition, if the bird's-eye view 61 and the overhead view 62 contain multiple objects of attention, the processing described in S6 and S7 is performed on each object of attention.

[0069] Then, in S8, the CPU 31 synthesizes the model image 65 from the real-time bird's-eye view image 61 and the overhead view image 62 generated in S7, which represent the current environment surrounding the vehicle, and displays it on the LCD 4 as an auxiliary image 66 to assist the vehicle's autonomous driving. It is not necessary for both the bird's-eye view image 61 and the overhead view image 62 to be displayed simultaneously; the user can switch between displaying the bird's-eye view image 61 and the overhead view image 62.

[0070] here, Figure 8 This diagram illustrates an example of an auxiliary image 66 displayed on the liquid crystal display 4. Figure 8 In the example shown, the model image 65 is displayed overlapping with the person 63 located at the left front of the vehicle. In addition, the model image 65 displayed in the bird's-eye view 61 and the overhead view 62 are oriented differently. In the bird's-eye view 61, the model image 65 is identified as being viewed from a hypothetical viewpoint at an angle downwards, while in the overhead view, the model image 65 is identified as being viewed from a hypothetical viewpoint at a vertical angle downwards.

[0071] Furthermore, as mentioned above, the processing after S2 repeatedly executes the most recently captured image as the object, and the auxiliary image 66 displayed on the LCD 4 is also updated in real time. Therefore, if you pay attention to the movement of the object, it will appear as if... Figure 8 As shown, the position of the model image 65 contained in the auxiliary image 66 also moves accordingly within the surrounding image (the landscape). On the other hand, when the object of attention is stationary, the position of the model image 65 contained in the auxiliary image 66 also becomes fixed within the surrounding image (wherein, if the vehicle moves, the position of the imaginary viewpoint changes, and therefore the position on the screen changes), and in this case, the process of enhancing the display of the model image 65 is performed as described later.

[0072] In addition, Figure 8 In the example shown, the model image 65, composited from the bird's-eye view 61 and the overhead view 62, is an opaque image with 0% transmittance, making it impossible to discern the extent of the overlapping actual image. Alternatively, the model image 65 can be set to a semi-transparent image. In this case, the user can still identify the object of attention in the actual image.

[0073] Here, in Figure 8In the bird's-eye view 61 and overhead view 62 shown, the person 63, especially located to the left front of the vehicle, is displayed in a significantly enlarged and distorted manner, making it difficult to grasp the presence and position of the person 63. However, in the auxiliary image 66, a model image 65 simulating the person 63 is superimposed at the position of the person 63, thus allowing the user to clearly grasp the presence and position of the person 63. In particular, if the person 63 is located near the composite boundary of the captured image, the person 63, which actually exists in the generated bird's-eye view 61 and overhead view 62, may sometimes disappear. Even in this case, by displaying the model image 65, the user can grasp the presence and position of the person 63. In addition, the user can grasp the surrounding environment of the vehicle by clearly understanding the relative position of the vehicle to the auxiliary image 66 displayed on the LCD 4, which also includes areas that are difficult to identify directly. Furthermore, the CPU 31 can also calculate the future movement trajectory of the vehicle based on the detection values ​​of the vehicle speed sensor, steering sensor, etc., and display the movement trajectory superimposed on the auxiliary image 66. Then, the auxiliary image 66 will continue to be displayed until the assisted driving based on automatic driving assistance ends (S11: Yes).

[0074] Then, in S9, the CPU 31 determines whether the object of attention contained in the bird's-eye view 61 and the overhead view 62 is in a stationary state. For example, it can determine whether the object of attention is moving or stationary by comparing the positions of the object of attention detected between consecutive frames. Alternatively, ultrasonic sensors 9A to 9L can also be used to determine whether the object of attention is moving or stationary.

[0075] Then, if it is determined that the object of attention contained in the bird's-eye view 61 and the top-view image 62 is in a stationary state (S9: Yes), proceed to S10. Conversely, if it is determined that the object of attention contained in the bird's-eye view 61 and the top-view image 62 is in a moving state (S9: No), proceed to S11. Furthermore, if the object of attention moves, then... Figure 8 As shown, the position of the model image 65 contained in the auxiliary image 66 also moves accordingly within the surrounding image.

[0076] In S10, the CPU 31 enhances the display of the model image 65 corresponding to the object of attention that is determined to be in a stopped state in the auxiliary image 66 displayed on the liquid crystal display 4. Here, as a method to enhance the display of the model image 65, examples include: (A) rotating the model image 65; (B) moving the model image 65 in the vertical direction (jumping); (C) scaling the size of the model image 65; (D) changing the display color of the model image 65; (E) flashing the model image 65, etc.

[0077] The following example, in particular, illustrates the concept of "(A) rotating the model image by 65 degrees," as follows: Figure 9As shown, the model image 65 is rotated around a vertical axis while maintaining a fixed position within the surrounding image (where the position of the imaginary viewpoint changes if the vehicle moves, thus changing its position on the screen). The direction of rotation is not limited; it can be clockwise or counterclockwise. Here, in terms of human perception, stationary objects are more difficult to recognize than moving ones. Therefore, when the model image 65 is assimilated into the background while stationary, the user may not notice its existence, i.e., fail to perceive the presence of an object of attention around the vehicle. However, in this embodiment, by performing the aforementioned enhanced display, the user can reliably identify even stationary objects of attention. Furthermore, the enhanced display of the model image 65 in S10 continues until the stationary state of the object of attention is lifted (movement begins), or until the object of attention disappears from the screen. Alternatively, the enhanced display can be performed for a predetermined duration after the start, regardless of whether the stationary state is lifted.

[0078] Furthermore, an object that is currently stationary may move in any direction after being observed. However, assuming the model image is displayed at a fixed position (65), it is possible to give the user the option to move only along the direction shown. Figure 10 The model image 65, as shown, gives the impression of being moved in the direction it is facing. On the other hand, if the model image 65 is rotated, it can suggest to the user that it can be moved in any direction of 360 degrees.

[0079] In addition, in the overhead view 62, it is assumed that the model image 65 displayed in the method of "(B) moving the model image 65 in the vertical direction (jumping)" may not produce motion, but the model image 65 can be made to move in the method of "(A) rotating the model image 65".

[0080] Furthermore, even in automated driving assistance mode, passengers can cancel the assistance at any time and bring the vehicle to a stop by applying the brakes. Passengers can also identify the assistance image 66 displayed on the LCD 4 and, if necessary, disengage the automated driving assistance and bring the vehicle to a stop by applying the brakes.

[0081] Then, in S11, CPU31 determines whether to terminate the assisted driving based on automatic driving assistance. Here, for the termination of assisted driving based on automatic driving assistance, for example, if it is parking assistance, the completion of parking in the parking space is the termination condition. Alternatively, the termination condition can be set by the user performing a specified termination operation in the operation unit 3, or by shifting the gear position to "P" or turning off the engine. Alternatively, the termination condition can be set by a situation where automatic driving assistance can no longer continue.

[0082] Then, if it is determined that the assisted driving based on automatic driving assistance has ended (S11: Yes), the driving assistance processing procedure ends. Conversely, if it is determined that the assisted driving based on automatic driving assistance has not ended (S11: No), the process returns to S2, and the display of the assistance image continues on the LCD 4.

[0083] As detailed above, according to the driving assistance device 1 of this embodiment and the computer program executed by the driving assistance device 1, an object of attention (S3) that is located around the vehicle and can be in at least one of a moving state or a stationary state and constitutes an object of attention to the vehicle is detected. A model image 65 representing the appearance of the object of attention is acquired (S6). An auxiliary image 66 is generated by synthesizing the model image 65 in a vehicle perimeter image representing the vehicle perimeter based on the position of the object of attention (S7). The generated auxiliary image 66 is displayed on the liquid crystal display 4 (S8). On the other hand, when the object of attention is in a stationary state, the model image 65 corresponding to the object of attention is displayed in an enhanced manner (S10). Therefore, the user can reliably identify objects of attention that are in a stationary state.

[0084] Furthermore, to enhance the display, the model image 65 corresponding to the object of attention is rotated about a vertical axis while in a fixed position, thus indicating to the user that the object of attention, which is currently stationary, can be moved in any direction. Additionally, when displaying a top-down view, the displayed model image 65 can also be made to move.

[0085] In addition, the vehicle surrounding image displayed on the LCD screen 4 includes at least one of a bird's-eye view 61 that looks down at the vehicle surrounding from an imaginary viewpoint in an oblique direction and a bird's-eye view 62 that looks down at the vehicle surrounding from an imaginary viewpoint in a vertical direction. Therefore, the user can identify the vehicle surrounding from various viewpoints and can also make the relative relationship with the vehicle's position clear, including areas that are difficult to identify directly, and understand the environment around the vehicle.

[0086] Furthermore, the vehicle perimeter image displayed on the LCD 4 is either a photographic image obtained from capturing the perimeter of the vehicle or an image that has been processed from a photographic image. An auxiliary image 66 is generated based on a composite model image 65 showing the position of the object of attention in the photographic image. Therefore, even if the object of attention in the photographic image is displayed with distortion or stretching relative to its original shape, the user can still grasp the presence and position of the object of attention. In particular, in the case of composite photographic images, if the object of attention is located near the composite boundary, the object of attention that actually exists in the composite photographic image may sometimes disappear. However, even in this case, by displaying the model image, the user can still grasp the presence and position of the object of attention.

[0087] Furthermore, the present invention is not limited to the above-described embodiments, and various improvements and modifications can be made without departing from the spirit of the present invention.

[0088] For example, in this embodiment, the vehicle perimeter image displayed on the liquid crystal display 4 is a bird's-eye view image 61 and a top-down view image 62 generated by processing images captured by cameras at the front, rear, left, and right. However, it can also be the images captured by the cameras themselves. For example, it can be an image captured by the front camera 6. In this case, the processing described in S2 is not required. In addition, in S3, the image captured by the front camera 6 is used to detect objects of attention. Specifically, if the type of object in the captured image that is determined to be included in the display image of the liquid crystal display 4 is the same as the type of object of attention, it is detected as an object of attention. In addition, in S7, the captured image is used to synthesize a model image 65. Furthermore, the viewpoint of the vehicle perimeter image displayed on the liquid crystal display 4 at this time becomes the viewpoint of the camera.

[0089] Furthermore, the vehicle's surroundings image displayed on the LCD 4 may not be a photographed image, but rather an imaginary landscape image recreated by CG. For example, a three-dimensional map image of the current location's surroundings can be acquired or generated and displayed on the LCD 4. The location of the object of attention within the displayed three-dimensional map image can be synthesized into a model image 65 and displayed as an auxiliary image 66. Even in this case, it can reliably allow the user to identify the object of attention when it is stationary.

[0090] In addition, in this embodiment, a composite model image 65 is created from both the bird's-eye view image 61 and the overhead view image 62, but it is also possible to create a composite model image 65 from only one of the images.

[0091] Additionally, in this embodiment, the display Figure 8 The auxiliary image 66 shown is provided during driving based on automated driving assistance, but it can also be displayed during driving by manual means. Figure 8 , Figure 9 The auxiliary image 66 is shown. In this case, the determination condition of S1 above is based on the user's display instruction on the auxiliary image, the operation of the vehicle such as shifting the gear lever, etc.

[0092] In addition, Figure 8 , Figure 9In this example, an example of displaying the model image 65 when a person 63 is present as the object of attention has been described. However, when a bicycle is present as the object of attention, a simple model image of a person can also be used, just as when a person 63 is present. Furthermore, a model image of a bicycle alone, or a model image of a person riding a bicycle more realistically, can also be used. In the case of a car, a model image of a car is used, but it can also be a simple image with no distinction between front, back, left, and right, just like a person.

[0093] Furthermore, in this embodiment, the driving assistance processing program is executed by the driving assistance ECU 10 of the driving assistance device 1. Figure 3 The processing structure can be modified, but the executing entity can be changed appropriately. For example, it can be configured as the control unit of the LCD display 4, the vehicle control ECU, the control unit of the navigation device, or other vehicle-mounted devices.

Claims

1. A driving assistance device, comprising: An object detection agency detects objects of attention located around a vehicle that are either in a moving or stationary state and that pose an object of attention to the vehicle. A model image acquisition mechanism acquires a model image representing the appearance of the object of attention; An auxiliary image generation mechanism generates an auxiliary image by synthesizing the model image from a vehicle perimeter image representing the vehicle's surroundings based on the position of the object of attention; An image display mechanism displays the auxiliary image on a display device; and The enhanced display mechanism enhances the display of the model image corresponding to the object of attention when the object of attention is in a stopped state.

2. The driving assistance device according to claim 1, wherein, When the object of attention is in a stopped state, the enhanced display mechanism rotates the model image corresponding to the object of attention in a fixed position around a vertical axis of rotation to perform the enhanced display.

3. The driving assistance device according to claim 1, wherein, The vehicle perimeter image includes at least one of a bird's-eye view of the vehicle perimeter from an imaginary viewpoint above in an oblique direction, and a top-down view of the vehicle perimeter from an imaginary viewpoint above in a vertical direction.

4. The driving assistance device according to any one of claims 1 to 3, wherein, It has an image acquisition mechanism that acquires images of the surroundings of the vehicle being photographed. The image surrounding the vehicle is either the captured image or an image that has been processed from the captured image. The auxiliary image generation mechanism generates the auxiliary image by synthesizing the model image based on the position of the object of attention in the captured image.

Images