Driving assistance systems

Abstract

This invention provides a driver assistance system that enables the linking of simulated vehicle surroundings images with actual captured images, and allows for effective assessment of the vehicle's surroundings using the actual images. [Solution] A vehicle surroundings image 41 that simulates the area around the vehicle 2 is displayed on the liquid crystal display 4. Images captured by a camera on the vehicle 2 are superimposed on the vehicle surroundings image 41, onto the imaging area corresponding to the camera's imaging range. If there is an object of attention in the area around the vehicle 2 that is of interest to the vehicle 2, an object image 60 indicating the object of attention is placed and displayed on the vehicle surroundings image 41 according to the position of the object of attention. Furthermore, when the object image 60 is displayed within the imaging area, the transparency of the object image 60 is set to be higher than when the object image 60 is displayed outside the imaging area.

Description

[Technical Field]

[0001] The present invention relates to a driving assistance apparatus that provides driving assistance for vehicles. [Background Art]

[0002] Conventionally, various means have been used as information providing means for providing various types of information to implement driving assistance such as route guidance and obstacle warnings for vehicle occupants. For example, such means include display on a liquid crystal display installed in the vehicle and sound output from a speaker. Here, there exist blind spot areas around the vehicle that are difficult for the driver to visually recognize from the driver's position. Especially when special operations such as parking operation and leaving the garage are performed, in order to let the driver grasp the situation in such blind spots, it has been conventional to display a vehicle surrounding image showing the surroundings of the vehicle on a liquid crystal display.

[0003] Here, as the vehicle surrounding image, for example, a real scene image captured by a camera equipped on the vehicle (hereinafter referred to as a camera image), or an image that pseudo-reproduces the surroundings of the vehicle using CG or the like (hereinafter referred to as a CG image) can be displayed. If a camera image is displayed, it is possible to allow the user to grasp a situation closer to reality, but there is a problem that large distortion and deformation occur in the displayed image. On the other hand, if a CG image is displayed, the aforementioned problems of distortion and deformation do not occur, but there is a problem that the image lacks reality, making it ambiguous for the user to grasp the surrounding situation. Therefore, for example, Japanese Patent Laid-Open No. 2015-139128 proposes a technique in which an icon display image that pseudo-reproduces the surroundings of the vehicle is displayed on a liquid crystal display, and when the vehicle speed of the vehicle becomes equal to or lower than a threshold value, an overhead view image generated from a captured image captured by a camera is displayed instead of the icon display image only in a very close range of 1 to 2 meters around the vehicle. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2015-139128 (paragraphs 0038-0046, 0052, Figures 3, 4) [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, when displaying CG images and camera images on the same screen as described in Patent Document 1 above, there is a problem in coordinating the CG images and camera images. For example, when there are pedestrians that the vehicle needs attention around, and the same pedestrian is displayed in both a CG image and a camera image, the appearance of the simulated pedestrian displayed in the CG image and the actual pedestrian displayed in the camera image will be very different. Therefore, for example, when displaying a pedestrian across from a CG image to a camera image, it is difficult for the user to intuitively understand whether the pedestrian displayed in the CG image and the pedestrian newly displayed in the camera image are the same pedestrian or different pedestrians.

[0006] Furthermore, while Patent Document 1 also displays an icon image of a vehicle superimposed on the position of other vehicles when the overhead view image generated from the captured image includes other vehicles, superimposing the vehicle icon image on the overhead view image makes it impossible to see the actual image of the other vehicles displayed in the overhead view image. This presents a problem in that it becomes difficult to grasp the situation based on the actual scene image, even though the overhead view image is an image of the actual scene.

[0007] The present invention was made to solve the aforementioned problems of the conventional invention, and aims to provide a driving assistance device that enables the linking of a simulated image of the area around the vehicle with an actual image captured from the real scene, and enables effective understanding of the situation around the vehicle using the actual image. [Means for solving the problem]

[0008] To achieve the above objective, the driving assistance device according to the present invention displays a vehicle surrounding image that simulates the area around the vehicle on a display device, displays an image captured by an imaging device provided by the vehicle in an imaging area within the vehicle surrounding image that corresponds to the imaging range of the imaging device, and when there is an object of attention that the vehicle requires attention to in the area around the vehicle, it places and displays an object image representing the object of attention in the vehicle surrounding image according to the position of the object of attention, and when the object image is displayed within the imaging area, it displays the object image with a higher transparency than when the object image is displayed outside the imaging area. Regarding "captured images," these may be the images themselves captured by an imaging device such as a camera, or they may be images that have been processed from captured images. For example, they may be images created by combining images taken by multiple cameras or images with a transformed viewpoint. Furthermore, the "imaging area" may be the same as the imaging range (the range that can be imaged) of the imaging device, or it may be a part of the imaging range. Furthermore, the phrase "display the captured image within the vehicle surrounding image, in the imaging area corresponding to the imaging range of the imaging device" means that the captured image may be displayed in place of the vehicle surrounding image in the imaging area, or the captured image may be displayed superimposed on the vehicle surrounding image. Furthermore, the "object image indicating the object of attention" may be an image that has the same or similar appearance as the object of attention, and it is not necessary for the object of attention and the object image to have the same or similar appearance as long as they can indicate the existence of the object of attention. Furthermore, "displaying with high transmittance" could mean, for example, uniformly increasing the transmittance within the imaging area, or keeping the transmittance unchanged immediately after entering the imaging area, but gradually increasing it according to the passage of time or the distance from the vehicle's position, or any other method. [Effects of the Invention]

[0009] According to the driver assistance device of the present invention having the above configuration, when displaying a vehicle surrounding image that simulates the area around the vehicle in combination with an image captured by an imaging device installed in the vehicle, if there is an object of attention that the vehicle needs to pay attention to in the area around the vehicle, an object image is displayed within the imaging area in accordance with the position of the object of attention, thereby enabling coordination between the simulated vehicle surrounding image and the actual image captured. Furthermore, when displaying an object image superimposed on the captured image, increasing the transparency of the object image makes it possible to effectively grasp the situation around the vehicle using the actual image captured. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This is a block diagram showing the configuration of the driver assistance system according to this embodiment. [Figure 3] This is a flowchart of the driver assistance processing program according to this embodiment. [Figure 4] This diagram shows an image of the area around a vehicle, which is a simulated reproduction of the area surrounding the vehicle. [Figure 5] This diagram illustrates the method for converting captured images into bird's-eye view images. [Figure 6] This diagram illustrates the method for generating a bird's-eye view image. [Figure 7] This diagram shows an example of displaying images of the area around a vehicle. [Figure 8] This figure shows an example of an object image. [Figure 9] This figure shows an example of how an object image is displayed. [Figure 10] This figure shows an example of how an object image is displayed. [Figure 11] This is a diagram illustrating a modified example. [Modes for carrying out the invention]

[0011] Hereinafter, a specific embodiment of the driving assistance device according to the present invention will be described in detail with reference to the drawings. First, a vehicle 2 equipped with the driving assistance device 1 according to the present embodiment will be described below. FIG. 1 is a schematic configuration diagram of the vehicle 2 according to the present embodiment.

[0012] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) using an internal combustion engine (such as an engine) as a driving source, or an automobile (electric automobile, fuel cell automobile, etc.) using an electric motor (such as a motor) as a driving source, or an automobile (hybrid automobile) using both of them as driving sources. There is no particular limitation on the vehicle type: it may be a regular passenger car, a commercial large truck, a bus, construction machinery, or the like. In the following description, the vehicle is assumed to be a four-wheeled automobile, but it may also be a two-wheeled or three-wheeled vehicle.

[0013] However, in addition to manual driving that travels based on the user's driving operation, the vehicle 2 may be a vehicle capable of assisted driving by automatic driving assistance in which the vehicle automatically travels regardless of the user's driving operation. Alternatively, it may be a vehicle that can only perform assisted driving by automatic driving assistance. On the other hand, the vehicle 2 is not necessarily limited to a vehicle capable of assisted driving by the above automatic driving assistance, and may be a vehicle capable of only traveling by manual driving. However, regardless of whether the vehicle is only capable of assisted driving by automatic driving assistance or only capable of traveling by manual driving, as described later, when an object such as a pedestrian or a bicycle approaches the vehicle, the vehicle is capable of driving assistance such as calling attention to such objects.

[0014] When the vehicle is configured to be capable of assisted driving by automatic driving assistance, the assistance may be performed only under specific situations such as parking or leaving a garage, may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, an expressway with gates at boundaries, regardless of whether the gates are manned or unmanned, toll or free).

[0015] Furthermore, in vehicle control for automatic driving support, for example, vehicle control such as for steering, a drive source, a brake, etc., is automatically performed such that the current position of the vehicle, the lane in which the vehicle travels, and the positions of surrounding obstacles are detected at any time, and the vehicle travels along the generated travel route at a speed conforming to the also generated speed plan. Particularly when performing parking support, detection results from sensors and cameras are used to check the parking space that is the target for the vehicle to park in and the surrounding conditions thereof, a parking track to the parking space is calculated, and vehicle control for causing the vehicle to enter the parking space along the calculated parking track and completing parking is automatically performed. However, only steering operation may be performed automatically, and control of the drive source and the brake may be performed based on manual operation. Alternatively, only guidance to the parking space may be provided, and a user may be allowed to manually perform the parking operation into the parking space.

[0016] Furthermore, in the present embodiment, regardless of whether travel is performed via the above automatic driving support or manual driving, when an attention target object that should be noted, such as a pedestrian, a bicycle, or the like, approaches the vehicle, attention calling is performed targeting those attention target objects. Specifically, as will be described later, a landscape around the vehicle is displayed on an in-vehicle display, and an object image indicating the attention target object is displayed at the position of the attention target object in the landscape, thereby calling attention. A vehicle occupant can also interrupt automatic driving support and stop the vehicle by performing a brake operation if necessary while visually checking the display.

[0017] Furthermore, as shown in FIG. 1, the vehicle 2 includes: an operation unit 3 that accepts operations from an occupant; a liquid crystal display 4 that displays a bird's-eye view image, an overhead view image of the vehicle surroundings, and other information related to driving support to the occupant; a speaker 5 that outputs voice guidance related to driving support; a front camera 6, a rear camera 7, and side cameras 8A, 8B for capturing images of the vehicle surroundings; ultrasonic sensors 9A to 9L that detect obstacles around the vehicle; and a driving support ECU (Electronic Control Unit) 10 that performs various types of arithmetic processing based on input information. The driving support device 1 is configured to include the above driving support ECU 10.

[0018] The following describes the various components of vehicle 2. First, the control unit 3 is located, for example, in front of the steering wheel and includes control buttons that are operated when starting the automated driving assistance system. By operating the control unit 3, the user can switch between manual driving, where the vehicle moves based on the user's driving input, and automated driving assistance, where the vehicle moves automatically without user input. The control unit 3 may also have a touch panel located in front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0019] The liquid crystal display 4 is a type of display device mounted on the instrument panel of the vehicle 2. It displays, for example, a simulated image of the area around the vehicle, as well as bird's-eye and overhead images of the area around the vehicle generated by performing viewpoint transformation and synthesis processing on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Furthermore, if there are objects of attention around the vehicle 2, such as pedestrians or bicycles, the liquid crystal display 4 can also superimpose an object image showing the appearance of the object of attention onto the location of the object in the vehicle surrounding image, bird's-eye view, or overhead view image. The liquid crystal display 4 may also be used for navigation devices.

[0020] Speaker 5 is mounted on the instrument panel of vehicle 2 and outputs voice guidance and warning sounds related to driver assistance. Speaker 5 may also be used for the navigation system.

[0021] Furthermore, the front camera 6 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0022] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0023] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0024] The driver assistance ECU 10 generates bird's-eye and overhead views of the vehicle's surroundings by performing viewpoint transformation and synthesis processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. It also detects objects around the vehicle (lane markings, other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) by performing image recognition processing on the captured images. The detected objects are also used for automated driving assistance.

[0025] On the other hand, ultrasonic sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by ultrasonic sensors 9A to 9L include, for example, people, bicycles, other vehicles, walls, and other obstacles that the vehicle 2 needs to avoid when driving, or obstacles that form a parking space. In addition, millimeter-wave sensors or laser sensors may be used as distance measuring sensors instead of ultrasonic sensors.

[0026] Furthermore, while the installation position and direction of each ultrasonic sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of the target object encompass all directions in front of, behind, and to the left and right of the vehicle's direction of travel, for example, ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is in front of the vehicle's direction of travel. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Ultrasonic sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each ultrasonic sensor 9A to 9L from the ground surface is approximately the same.

[0027] In this embodiment, among the ultrasonic sensors 9A to 9L, the ultrasonic sensors 9A to 9D on the front of the vehicle 2 and the ultrasonic sensors 9I to 9L on the rear of the vehicle 2 are installed in positions where they can receive reflected waves as indirect waves from adjacent sensors. By receiving both direct and indirect waves, it is possible to determine not only the distance to the object but also the specific position of the object (relative position to the vehicle) using triangulation. The ultrasonic sensors 9E to 9H on the sides are installed spaced apart from each other and cannot receive indirect waves, but as the vehicle moves, it is possible to determine the specific position of the object (relative position to the vehicle) using triangulation with respect to the distance measured at the previous position, the distance measured at the current position, and the distance traveled in between.

[0028] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to various driver assistance functions, including automated driving assistance. For example, when performing automated driving assistance, it continuously detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drivetrain, and brakes, to travel along the generated driving trajectory at a speed according to the generated speed plan. The liquid crystal display 4 also displays a simulated image of the area around the vehicle, as well as bird's-eye and overhead images of the area around the vehicle generated from camera images. In particular, if there are objects that require attention from the vehicle, such as pedestrians or bicycles, around the vehicle 2, the ECU 10 superimposes an object image showing the appearance of the object onto the vehicle surrounding image, bird's-eye or overhead image, and also outputs a warning sound. The driver assistance ECU 10 is connected to the aforementioned operation unit 3, liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. Furthermore, it is connected to various sensors mounted on vehicle 2, such as GPS, vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as on-board devices such as the navigation system. The detailed configuration of the driver assistance ECU 10 will be described later.

[0029] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configuration related to the detection of objects of attention and the control of support for detected objects of attention, as well as the control related to said configuration, will be explained.

[0030] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0031] As shown in Figure 2, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1, and includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores driving trajectory data when the driving trajectory is calculated, a ROM 33 which stores control programs as well as driver assistance processing programs (see Figure 3) described later, and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 also executes various functions as a processing algorithm. For example, it has the function of displaying a vehicle surrounding image that simulates the area around the vehicle on the liquid crystal display 4, a function of displaying an image captured by a camera on the vehicle in an area corresponding to the camera's imaging range within the vehicle surrounding image, a function of placing and displaying an object image indicating an object of attention in the vehicle surrounding image according to the position of the object of attention when there is such an object around the vehicle, and a function of displaying an object image with a higher transparency when displaying an object image within the imaging area than when displaying an object image outside the imaging area.

[0032] Furthermore, the driver assistance ECU 10 is connected to various sensors 37 for detecting the vehicle's current position and behavior, such as GPS, vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units 38 of the vehicle, such as steering, brakes, accelerator, and transmission. When providing automatic driving assistance for vehicle 2, the ECU 10 controls each drive unit 38 while detecting the vehicle's current position and behavior based on the detection results of these sensors 37. Specifically, the automatic driving assistance may include, for example, continuously detecting the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, and controlling the vehicle, such as the steering, drive source, and brakes, so that it travels along the generated driving trajectory at a speed according to the generated speed plan. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes is done manually.

[0033] Furthermore, ROM33 includes vehicle information DB35 and object image DB36. Vehicle information DB35 stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) and detection axes (optical axis and field of view for cameras), overall length, vehicle width, wheelbase, and minimum turning radius of the cameras and ultrasonic sensors 9A-9L installed on vehicle 2. It also stores the imaging range (capable imaging range) of each camera: front camera 6, rear camera 7, and side cameras 8A and 8B. This information is to be entered in advance by the occupants or personnel from the vehicle manufacturer.

[0034] On the other hand, the object image DB36 stores various information used when rendering 3D object images showing the appearance of people and bicycles. In this embodiment, objects that can take on at least one of two states, namely a moving state or a stopped state, i.e., objects that move and stop rather than fixed objects such as walls and utility poles, are considered attention objects for vehicles. Anything that moves and stops may be considered an attention object, and preferably, specific objects may be chosen as attention objects, for example, people and bicycles. However, attention objects are not limited to people and bicycles; for example, vehicles and motorcycles may also be included. In that case, information used when rendering 3D object images showing the appearance of vehicles and motorcycles is also stored in the object image DB36.

[0035] Furthermore, the object image DB36 stores various information used when rendering 3D object images representing the appearance of people and bicycles, including information necessary for executing various processes for creating 3DCG such as modeling, scene layout settings, and rendering, as well as texture images to be applied to the created 3DCG models. The object image DB36 may also store object images representing the completed appearance of people and bicycles. In this case, there may be only one object image for each type of object, or multiple object images may be prepared for each type. For example, different object images may be prepared for children and adults, or for city bikes and mountain bikes. However, in the following explanation, only one object image will be used for each type of object.

[0036] On the other hand, an object image only needs to indicate the presence of the object of attention. It doesn't necessarily have to be an image showing the appearance of the object; it could be a simple geometric shape such as a circle or a rectangular prism.

[0037] Next, the driver assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the driver assistance processing program according to this embodiment. Here, the driver assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that provides user assistance with objects of attention located around the vehicle.

[0038] Furthermore, the above-mentioned driver assistance processing program may be executed only while the vehicle's automated driving assistance is in operation, or it may also be executed while the vehicle is being driven manually. Alternatively, it may be executed only while specific automated driving assistance, such as long-range parking assistance, is being performed. It may also be executed only while the vehicle is moving, or it may be executed while the vehicle is stopped. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the driver assistance device 1 and executed by the CPU 31.

[0039] First, in step 1 (hereinafter abbreviated as S), the CPU 31 generates a vehicle surroundings image 41 that simulates the area around the vehicle. The vehicle surroundings image 41 is not a real-world image captured by a camera, but a virtual image reproduced using computer graphics (CG), specifically a bird's-eye view looking down at the area around the vehicle from a virtual viewpoint above. In addition, a vehicle image 42, which schematically represents the vehicle, is superimposed on the vehicle's current position. The vehicle surroundings image 41 can be generated, for example, by acquiring 3D map information of the area around the vehicle's current position from an external server or a storage medium provided by the vehicle 2, and then extracting the area to be displayed (which varies depending on the position of the virtual viewpoint, the direction of the line of sight, and the scale) from the acquired 3D map information.

[0040] Furthermore, the size of the area where the vehicle surroundings image 41 is generated can be set as appropriate, but it should be at least larger than the area where the bird's-eye view image generated in S2 (described later) is displayed. While it is assumed that the 3D map information is generated in advance by the manufacturer, it may also be generated by the vehicle 2 based on information acquired by sensors and cameras. The objects to be reproduced in the vehicle surroundings image 41 may be limited to roads only, as shown in Figure 4, or they may also include structures such as buildings. If there are objects of attention such as pedestrians or cyclists, they are reproduced in the vehicle surroundings image 41 by placing object images as described later (S11).

[0041] Furthermore, the processes from S1 onward will be repeatedly executed at predetermined time intervals (for example, every 250ms) until the ACC power is turned off (S12: YES).

[0042] Next, in S2, the CPU 31 generates a bird's-eye view image of the area around the vehicle, looking diagonally downwards from above, based on the real-time images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. The method for generating the bird's-eye view image is described below. As shown in Figure 5, the real-time images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The images projected onto the virtual projection plane are then converted into images viewed from a virtual viewpoint looking diagonally downwards from above the vehicle 2 in the direction of travel, thereby generating the bird's-eye view image for each camera. The conversion to an image viewed from a virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate of the image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, to each coordinate of the ground coordinate system, which is set along the ground surface, and then to each coordinate of the bird's-eye view image coordinate system. The conversion formulas used for each coordinate conversion are already publicly known, so their explanation is omitted. Then, as shown in Figure 6, a bird's-eye view image is generated by combining the bird's-eye view image 51 obtained by transforming the viewpoint of the image captured by the front camera 6, the bird's-eye view image 52 obtained by transforming the viewpoint of the image captured by the rear camera 7, the bird's-eye view image 53 obtained by transforming the viewpoint of the image captured by the side camera 8A, and the bird's-eye view image 54 obtained by transforming the viewpoint of the image captured by the side camera 8B. The virtual viewpoint and line of sight of the bird's-eye view image generated in S2 are made to match the virtual viewpoint and line of sight of the vehicle surrounding image 41 generated in S1. In addition, when the vehicle is reversing, the bird's-eye view image generated will be a bird's-eye view image looking diagonally down at the rear of the vehicle.

[0043] In this embodiment, the vehicle surroundings image 41 generated in S1 is a bird's-eye view looking down at the area around the vehicle from above, so the image of the area around the vehicle generated in S2 is also a bird's-eye view. However, the vehicle surroundings image generated in S1 may also be an overhead view looking down vertically from a viewpoint above, in which case S2 may generate an overhead view image looking down vertically from the same viewpoint above. The generation of the overhead view image is basically the same process as the generation of the bird's-eye view image, with only the angle of the line of sight direction when the viewpoint is changed being different, so the explanation is omitted.

[0044] Subsequently, in S3, the CPU 31 combines the vehicle surroundings image 41, which is a simulated reproduction of the area around the vehicle generated in S1, with the bird's-eye view images 51-54 generated in S2, and displays them on the liquid crystal display 4. Specifically, as shown in Figure 7, the bird's-eye view images 51-54 generated in S2 are superimposed on the vehicle surroundings image 41 over the imaging areas corresponding to the imaging ranges of the front camera 6, rear camera 7, and side cameras 8A and 8B, and the vehicle surroundings image 41 is displayed on the liquid crystal display 4. As a result, as shown in Figure 7, the actual images captured by the cameras are displayed, especially in the area near the vehicle 2, while the simulated vehicle surroundings image 41 is displayed in other areas.

[0045] Furthermore, the imaging area over which the bird's-eye view images 51-54 are superimposed may be a narrower area than the imaging range of each camera. That is, instead of superimposing all of the bird's-eye view images 51-54 generated in S2, only the images within a predetermined distance (e.g., 2m) from the vehicle 2 may be superimposed. Also, in this embodiment, the bird's-eye view images 51-54 are superimposed and displayed on the vehicle surrounding image 41, but the imaging area may be such that the vehicle surrounding image 41 is not displayed and the bird's-eye view images 51-54 are displayed instead.

[0046] In Figure 7, only a bird's-eye view is displayed as the vehicle surroundings image 41. However, the display screen of the LCD display 4 may be divided into left and right sections to display both the bird's-eye view and the overhead view. Furthermore, the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B (the captured images themselves, without viewpoint transformation or compositing) may also be displayed together with the vehicle surroundings image 41.

[0047] Next, in S4, the CPU 31 performs image recognition processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B to detect objects of attention around the vehicle. As mentioned above, objects of attention can be in at least one of two states, moving or stationary, and are objects of attention for the vehicle; for example, people and bicycles fall into this category. For example, an object is detected as an object of attention if the category (type) of the object included in the captured image matches the category of an object of attention. Alternatively, instead of using categories, the movement of each object can be checked and judged individually whether or not it is an object of attention. In that case, detection by ultrasonic sensors 9A to 9L is also possible, as described later. Furthermore, for the detection of objects of attention, detection results from other vehicles may also be obtained via communication, not just from the vehicle itself. This makes it possible to detect objects of attention that are outside the detection range of the vehicle.

[0048] In step S4, the process for detecting an object of attention from the captured image can be, for example, to perform brightness correction based on the brightness difference between the road surface and the object of attention, then perform binarization to separate the object of attention from the image, geometric processing to correct distortion, smoothing processing to remove noise from the image, etc., to detect the boundary line between the road surface and the object of attention. Furthermore, detection may also be performed using known template matching processing or feature point detection processing. In addition, the image recognition processing on the captured image is not limited to the above example, and may be performed using machine learning, for example.

[0049] Furthermore, in addition to image recognition processing, ultrasonic sensors 9A to 9L may also be used in combination to detect objects of attention. While detection using ultrasonic sensors 9A to 9L can identify the location of objects around the vehicle, it is basically not possible to identify the category (type) of the detected object, that is, whether or not the detected object is a target of attention. However, if the object is moving, for example, it is possible to estimate that the object is a target of attention. Alternatively, by using a camera in combination, the type of object detected by ultrasonic sensors 9A to 9L may be identified.

[0050] Subsequently, in S5, the CPU 31 determines, based on the results of the object detection process in S4, whether or not there is an object of attention around the vehicle, or more specifically, within the display range of the vehicle surrounding image generated in S1.

[0051] If it is determined that there is an object of attention around the vehicle (S5: YES), the process proceeds to S6. Conversely, if it is determined that there is no object of attention around the vehicle (S5: NO), the process proceeds to S12.

[0052] In S6, the CPU 31 acquires an object image 60 showing the appearance of the attention objects detected around the vehicle. If there are multiple attention objects detected around the vehicle, an object image 60 is acquired for each of the multiple attention objects. Here, the object image 60 is a three-dimensional polygon image. Figure 8 shows an example of a human object image 60, which is a 3D image showing the appearance of a person. Note that the object image 60 may be, for example, a "wireframe model" that displays only edges, or a "surface model" that displays faces.

[0053] As shown in Figure 8, the human object image 60 is identifiable as a person, and its shape allows for identification of the person's orientation (which side is the front and which is the back). The person's orientation corresponds to the person's direction of movement, with the front facing the person being the direction of movement. Similarly, the shape allows for identification of left and right (which side is the right and which is the left). By using such a shape, when the object image 60 is superimposed on the vehicle surrounding image 41 as described later, the orientation of the actual person and the orientation of the object image 60 can be displayed in correspondence, making it possible to identify the orientation of the person who is the object of attention. On the other hand, it is also possible to intentionally make the orientation of the object image 60 unclear, in which case, even if the orientation of the actual person and the orientation of the object image 60 differ, it is possible to prevent it from causing a sense of incongruity.

[0054] Furthermore, various material settings and mapping processes may be applied to the object image 60. Mapping processes include texture mapping, which applies a texture image to the surface of the object, and bump mapping, which changes the direction of light reflection to create fine irregularities. For example, a texture image is an image that depicts effects such as shading, transparency, and reflection. When the object image 60 is placed in the vehicle surroundings image 41 in S11 described later, each effect of the texture image is rendered in a way that does not look unnatural and makes the object image 60 easily visible, taking into account the orientation of placement, the surrounding brightness, and the position of light sources (e.g., sunlight, streetlights). On the other hand, it is not necessary to set the light source to be the same as the real light source; a virtual light source may be set to make the object image 60 easily visible to the user. Furthermore, it is desirable that the display color of the texture image be a color (e.g., white, yellow, red) that has a brightness difference or contrast ratio above a threshold compared to the background color (road surface or soil color).

[0055] Furthermore, in S6, the CPU 31 may use the information stored in the object image DB 36 to perform processes such as modeling, scene layout setting, and rendering to create the object image 60 shown in Figure 8, or it may store pre-completed object images 60 in the object image DB 36 and retrieve them in S6 by reading them from the object image DB 36. When creating an object image 60, it is desirable to determine the type of detected object of attention using the image recognition process described above and create an object image 60 of the corresponding type. On the other hand, when reading an object image 60, it is desirable to store multiple object images 60 in the object image DB 36 in advance, separated by type of object of attention, and similarly determine the type of detected object of attention using the image recognition process described above and read the object image 60 of the corresponding type. On the other hand, the object image 60 does not necessarily have to be an image showing the appearance of the object of attention; for example, it may be a simple geometric shape such as a circle or a rectangular prism. In other words, it may be an object image 60 with a fixed appearance regardless of the type of object of attention.

[0056] Furthermore, the appearance of the object image 60 may be set arbitrarily by the user from among several options. In addition, the appearance of the object image 60 representing each type of object to be noticed may be set.

[0057] Subsequently, in S7, the CPU 31 obtains the specific location of the object of attention that was determined to be in the vicinity of the vehicle in S5. For example, it is identified in the coordinate system of the vehicle surroundings image 41. The location of the object of attention is determined using image recognition processing or ultrasonic sensors 9A to 9L.

[0058] Next, in S8, the CPU 31 determines whether the position of the object of attention acquired in S7 is within the imaging area where the bird's-eye view images 51-54 were displayed in S3.

[0059] Then, if it is determined that the location of the object of attention acquired in S7 is within the imaging area where the bird's-eye view images 51-54 were displayed in S3 (S8: YES), the process proceeds to S9. On the other hand, if it is determined that the location of the object of attention acquired in S7 is outside the imaging area (S8: NO), the process proceeds to S10.

[0060] In S9 and S10, the CPU 31 sets the transparency of the object image 60 acquired in S6 when displaying the object image 60 on the liquid crystal display 4. However, in S9, the transparency of the object image 60 is generally set higher than in S10. For example, if the transparency is set to 0% in S10, it is set to 50% in S9. If the transparency of the object image 60 is set higher in S9 than in S10, the value of the transparency can be changed as appropriate. The transparency of the object image 60 representing the object of attention may be set to a different value for each type of object of attention.

[0061] Furthermore, the transparency of object image 60 may be set by the user by specifying a concrete numerical value.

[0062] Furthermore, the object image 60 may be displayed with a uniformly increased transparency within the imaging area, or the transparency may remain unchanged immediately after entering the imaging area, but then be gradually increased thereafter.

[0063] For example, in step S9, the CPU 31 can measure the elapsed time since the object image 60 entered the imaging area and set the transmittance of the object image 60 to (elapsed time since the object image entered the imaging area [s] × 5)%. That is, the longer the elapsed time since the object image 60 entered the imaging area, the higher the transmittance of the object image 60 can be set. An upper limit may also be set for the transmittance (for example, 80%), or the object image 60 may be hidden (transmittance 100%) if the elapsed time since the object image 60 entered the imaging area exceeds a predetermined time.

[0064] As another example, in S9, the CPU 31 can calculate the distance from the vehicle to the object of attention within the imaging area and set the transmittance of the object image 60 to (100 - distance from the vehicle to the object of attention within the imaging area [m] × 10)%. In other words, the shorter the distance between the vehicle and the object of attention, the higher the transmittance of the object image 60 can be set. An upper limit may also be set for the transmittance (for example, 80%), or the object image 60 may be hidden (transmittance 100%) when the distance between the vehicle and the object of attention falls below a predetermined distance.

[0065] Next, in S11, the CPU 31 combines the object image 60 acquired in S6 with the vehicle surrounding image 41 to match the position of the object of attention included in the vehicle surrounding image 41, thereby placing and displaying the object image 60 in the vehicle surrounding image 41. In particular, if the object of attention is located within the imaging area, the object image 60 will be placed and displayed in the bird's-eye view images 51-54. In this case, since the bird's-eye view images 51-54 contain images of the actual object of attention, the images of the actual object of attention included in the bird's-eye view images 51-54 and the object image 60 will be displayed overlappingly. The transparency of the object image 60 is set to the transparency set in S9 or S10.

[0066] As described above, the processing from S1 onwards is repeatedly executed on the most recently captured image, and the vehicle surrounding image 41, bird's-eye view images 51-54, and object image 60 displayed on the liquid crystal display 4 are updated in real time. Therefore, if the object of attention moves, the position of the object image 60 will also move accordingly within the vehicle surrounding image 41. On the other hand, if the object of attention is stationary, the position of the object image 60 will remain fixed within the vehicle surrounding image 41 (however, if the vehicle is moving, the position of the virtual viewpoint will change, so the position on the screen will change).

[0067] In this embodiment, the vehicle surroundings image 41 overlays an object image 60 that mimics the object of attention onto the location of the object of attention, making it possible to clearly identify the presence and location of the object of attention to the user. In particular, it becomes possible to clearly identify the relative relationship between the vehicle's position and the object of attention, even in areas that are difficult to see directly, and to understand the environment around the vehicle. Furthermore, while images simulated using CG or the like often lack realism, making it difficult for the user to grasp the surrounding situation, this embodiment displays actual images captured by a camera, especially near the vehicle, allowing the user to clearly understand the surrounding situation.

[0068] Furthermore, when displaying a combination of a real-world image captured by a camera and a CG image that is simulated using computer graphics, even when displaying the same object of attention, the appearance of the CG image and the real-world image can differ significantly, making it difficult to match the content of the real-world image with the content of the CG image. In this embodiment, by superimposing an object image 60 indicating the object of attention in both the area where the real-world image is displayed (within the imaging area) and the area where the CG image is displayed (outside the imaging area), the correspondence can be easily grasped. However, when superimposing an object image 60 onto a real-world image, the actual image of the object of attention displayed in the real-world image becomes invisible, leading to a problem where the user's understanding of the surrounding situation becomes ambiguous. Nevertheless, in this embodiment, when superimposing an object image 60 onto a real-world image, increasing the transparency of the object image 60 prevents obstruction of the visibility of the object of attention in the real-world image, making it possible to effectively understand the situation around the vehicle using the real-world image.

[0069] Furthermore, regarding the setting of the transparency of the object image 60, as mentioned above, it is possible to display it with a uniformly increased transparency within the imaging area, or it is possible to display it with no change in transparency immediately after entering the imaging area, but then gradually increasing the transparency. For example, Figure 9 shows a configuration in which the transparency is uniformly increased within the imaging area. In the configuration shown in Figure 9, when a target object approaches the vehicle, the object image 60 is displayed with 0% transparency when the target object is outside the imaging area. Subsequently, when the target object enters the imaging area, the transparency changes from 0% to 50%, and the object image 60 continues to be displayed with 50% transparency until the target object is outside the imaging area.

[0070] On the other hand, Figure 10 shows a mode in which the transmittance remains unchanged immediately after entering the imaging area, but is gradually increased depending on the elapsed time since the object image 60 entered the imaging area, or the distance between the vehicle's position and the object of attention. In the mode shown in Figure 10, when the object of attention approaches the vehicle, if the object of attention is located outside the imaging area, the object image 60 is displayed with a transmittance of 0%. Subsequently, when the object of attention enters the imaging area, the transmittance of the object image 60 is maintained at 0%, but the longer the time the object image 60 remains in the imaging area thereafter, the more gradually the transmittance of the object image 60 increases. Alternatively, the shorter the distance between the vehicle and the object of attention, the more gradually the transmittance of the object image 60 increases. The transmittance can be set as high as 100% (not displayed), or an upper limit can be set to, for example, around 80%. Furthermore, regarding the transparency of object image 60, which has been increased in stages, the transparency may be returned to its original level once the conditions are met, or it may be displayed without reducing the transparency until the object of attention is located outside the imaging area.

[0071] Furthermore, if the distance between the vehicle and the object of attention becomes very short (for example, less than 1 meter), the display color of the corresponding object image 60 may be changed or it may be made to flash as a warning to the object of attention. Audio warnings may also be provided in conjunction with this.

[0072] Furthermore, if multiple objects requiring attention are located around the vehicle, the processes described in S6 to S11 above are performed for each object requiring attention.

[0073] Subsequently, in S12, the CPU 31 determines whether or not the ACC power supply has been turned off. If it is determined that the ACC power supply has been turned off (S12: YES), the driver assistance processing program is terminated. On the other hand, if it is determined that the ACC power supply has not been turned off (S12: NO), the program returns to S1.

[0074] Furthermore, when displaying bird's-eye view images 51-54 on the vehicle surrounding image 41, in this embodiment, bird's-eye view images 51-54 generated from real-time captured images are displayed on the imaging area corresponding to the current vehicle position. However, it is also possible to store previously generated bird's-eye view images 51-54 in a storage medium such as flash memory 34, and then superimpose and display the previously captured bird's-eye view images 51-54 on the imaging area corresponding to the vehicle's position in the past. As a result, as shown in Figure 11, previously generated bird's-eye view images 55 will be displayed not only in the vicinity of the current vehicle position but also in the vicinity of the vehicle's position in the past. This makes it possible to expand the display area of ​​the actual scenery image.

[0075] However, since the bird's-eye view image 55 displays a past image of the actual scene, not a current image, it is possible that the bird's-eye view image 55 may display an image of an object of attention that does not actually exist, or that an image of an object of attention that actually exists at that location may not be displayed. However, even if an image of an object of attention that exists is not displayed in the bird's-eye view image 55, the object image 60 is also displayed in the display area of ​​the bird's-eye view image 55, so it is possible to inform the user of the presence of the object of attention by displaying the object image 60.

[0076] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, a vehicle surrounding image 41 that simulates the area around the vehicle 2 is displayed on the liquid crystal display 4, and an image captured by a camera equipped on the vehicle 2 is displayed in the vehicle surrounding image 41 in an area corresponding to the camera's imaging range (S3). If there is an object of attention that the vehicle 2 needs to pay attention to in the area around the vehicle 2, an object image 60 indicating the object of attention is placed and displayed in the vehicle surrounding image 41 according to the position of the object of attention (S11). When the object image 60 is displayed within the imaging area, the transparency of the object image 60 is increased compared to when the object image 60 is displayed outside the imaging area, so that the simulated vehicle surrounding image 41 and the captured image can be linked. Furthermore, when the object image 60 is superimposed on the captured image, increasing the transparency of the object image 60 makes it possible to effectively grasp the situation around the vehicle using the captured image, which is a real-world image. Furthermore, when displaying the object image 60 within the imaging area, the transparency of the object image 60 is increased as the distance between the vehicle 2 and the object of attention decreases (S9, S10). Therefore, for close objects of attention that require particular attention from the vehicle 2, the captured image, which is a real-world image, is used preferentially to allow the user to see them, rather than the object image. Furthermore, when the object image 60 enters the imaging area from outside the imaging area, the transparency of the object image 60 is changed to a higher transparency than when the object image 60 is displayed outside the imaging area (S9, S10). This makes it possible to always prevent the object image 60 from interfering with the understanding of the situation around the vehicle using the captured image, which is a real-world image. Furthermore, when previously captured images are stored in a storage medium and displayed on the vehicle surroundings image 41, the previously captured images are superimposed onto the imaging area corresponding to the vehicle's position in the past, making it possible to display the captured images, which are actual scenery images, in a wider area.

[0077] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, pedestrians and bicycles are listed as objects of attention because they can be in at least one of two states, either moving or stationary, and are therefore objects of attention for the vehicle. However, other vehicles may also be included as objects of attention. Alternatively, stationary objects such as road markings or utility poles may also be included as objects of attention.

[0078] Furthermore, in this embodiment, the external shape of the object image 60 representing the object of attention is determined by the type of object of attention. However, even for the same object of attention, the shape of the object image 60 when displayed within the imaging area may be different from the shape of the object image 60 when displayed outside the imaging area. For example, the shape of the object image 60 when displayed within the imaging area may be simpler than the shape of the object image 60 when displayed outside the imaging area. As an example, if the object of attention is a pedestrian, when displaying the object image 60 outside the imaging area, it is possible to use a human-shaped object image 60 as shown in Figure 8, and when displaying the object image 60 within the imaging area, it is possible to use a circular or rectangular geometric object image 60.

[0079] In this embodiment, the vehicle surroundings image 41 is always displayed in combination with the bird's-eye view images 51-54, which are real-world images. However, the bird's-eye view images 51-54 may be displayed only when selected by the user. Furthermore, the vehicle surroundings image 41 (including the bird's-eye view images 51-54) may be made resizable by user operation.

[0080] Furthermore, in this embodiment, the images of the actual scenery displayed in combination with the vehicle surroundings image 41 are bird's-eye view images 51-54 generated by converting the viewpoint and combining the captured images taken by the front camera 6, rear camera 7, and side cameras 8A and 8B. However, the captured images themselves may also be displayed in combination with the vehicle surroundings image 41.

[0081] In this embodiment, a vehicle surroundings image 41, which simulates the area around the vehicle, is first displayed on the liquid crystal display 4, and the bird's-eye view images 51-54, which are actual scenery images, are superimposed on the imaging area of ​​the vehicle surroundings image 41. However, in the imaging area, the vehicle surroundings image 41 may not be displayed, and the image captured may be displayed instead.

[0082] Furthermore, the execution order of each step in the driver assistance processing program shown in Figure 3 is just an example, and the execution order can be changed as appropriate. For example, the detection process of the object to be noticed in S4 may be executed before S1 to S3. It is also possible to omit some of the steps included in the driver assistance processing program.

[0083] Furthermore, in this embodiment, the driver assistance processing program (Figure 3) is executed by the driver assistance ECU 10 of the driver assistance device 1, but the execution entity can be changed as appropriate. For example, it may be configured so that the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle devices execute the process. Alternatively, it can be executed by a communication terminal that is connected to the vehicle 2 in a communicative manner. Moreover, when executed by a communication terminal, it is also possible to execute part of the process on an external server.

[0084] [Note] The embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0085] (Invention A) The driving support device (1) according to claim 1, wherein when the object image (60) is displayed within the imaging area, the object image is superimposed on the image of the object to be noticed included in the imaging image.

[0086] According to this, even if the image of the object of attention included in the captured image is deformed or distorted, it is still possible to make the user aware of the presence of the object of attention through the object image.

[0087] (Invention B) The driving support device (1) according to claim 1, wherein when displaying the object image (60) within the imaging area, the transmittance of the object image is displayed at a higher level as the elapsed time since the object image entered the imaging area increases.

[0088] According to this, for objects that require attention and remain around the vehicle for an extended period, it becomes possible to prioritize the use of captured images of the actual scene rather than object images to allow the user to see them.

[0089] (Invention C) The driving assistance device (1) according to claim 2, wherein the object image (60) is hidden when the distance between the vehicle (2) and the object to be noticed becomes less than a predetermined distance.

[0090] According to this, for objects of attention that are particularly close to the vehicle, it becomes possible to accurately allow the user to understand their positional relationship with the vehicle using captured images of the actual scene, without displaying object images.

[0091] (Invention D) A driving support device (1) according to Invention B, which hides the object image when the elapsed time since the object image (60) entered the imaging area exceeds a predetermined time.

[0092] According to this, for objects of attention that remain around the vehicle for a particularly long time, it becomes possible to accurately allow the user to understand their positional relationship with the vehicle using captured images of the actual scenery, without displaying object images.

[0093] (Invention E) The driving support device (1) according to claim 1, wherein the shape of the object image (60) is a shape that shows the appearance of the object to be observed.

[0094] According to this, it becomes possible to inform the user of the type of object they are paying attention to through an object image.

[0095] (Invention F) The driving support device (1) according to claim 1, wherein the shape of the object image (60) when displayed within the imaging area is a simpler shape than the shape of the object image when displayed outside the imaging area.

[0096] According to this method, simplifying the shape of object images can prevent interference with the visibility of the object of attention in the captured image. [Explanation of Symbols]

[0097] 1…Driving assistance system, 2…Vehicle, 4…LCD display (display device), 6…Front camera (imaging device), 7…Rear camera (imaging device), 8A,8B…Side cameras (imaging devices), 10…Driving assistance ECU, 31…CPU, 41…Vehicle surroundings image, 51~54…Bird's-eye view image (imaging image), 60…Object image

Claims

1. The system displays an image of the area around the vehicle, which is a simulated reproduction of the area surrounding the vehicle. The image captured by the imaging device installed in the vehicle is displayed in the image area of ​​the vehicle surroundings that corresponds to the imaging range of the imaging device. If there is an object of attention that requires attention from the vehicle in the vicinity of the vehicle, an object image representing the object of attention is placed and displayed in the vehicle surroundings image according to the location of the object of attention. A driving assistance device that displays the object image within the imaging area with a higher transparency than when displaying the object image outside the imaging area.

2. The driving assistance device according to claim 1, wherein when displaying the object image within the imaging area, the transmittance of the object image is displayed higher as the distance between the vehicle and the object of attention decreases.

3. The driving support device according to claim 1, wherein, at the timing when the object image enters the imaging area from outside the imaging area, the transmittance of the object image is changed to a higher transmittance than when the object image is displayed outside the imaging area.

4. The captured images taken in the past are stored in a storage medium. The driver assistance device according to any one of claims 1 to 3, wherein when the captured image is displayed on the vehicle surroundings image, the previously captured image is superimposed on the imaging area corresponding to the past position of the vehicle.

Images