Driving assistance systems

Abstract

The present invention provides a driver assistance device that can make vehicle occupants aware of the presence of stationary objects that have moved outside the display screen of the display device. [Solution] The system is configured to display the surrounding environment in the direction of travel of the vehicle 2, specifically the surrounding environment within the imaging range captured by the front camera 6 and rear camera 7 installed on the vehicle 2, on the liquid crystal display 4. When an influencing stationary object that affects the movement of the vehicle 2 is detected in the direction of travel of the vehicle 2, and at least a portion of the area of ​​the influencing stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, the system is configured to notify the occupants of the vehicle 2 of the presence of the influencing stationary object.

Description

Technical Field

[0001] The present invention relates to a driving support device that performs driving support for a vehicle.

Background Art

[0002] Conventionally, various means have been used as information providing means for providing various types of information for vehicle driving support such as route guidance and obstacle warnings to vehicle occupants. For example, display on a liquid crystal display installed in the vehicle, voice output from a speaker, and the like. Here, there are areas around the vehicle that are blind spots that are difficult to visually recognize from the driver's position. In particular, when performing special operations such as parking operations, in order to allow the driver to grasp the situation of such blind spots, it has been conventionally done to display a peripheral image captured by a camera installed in the vehicle on a liquid crystal display.

[0003] For example, Japanese Patent Application Laid-Open No. 2004-240480 discloses a technique in which the surrounding situation of a vehicle is imaged by an in-vehicle camera, the captured peripheral image is displayed on an in-vehicle monitor, and when an obstacle is included in the peripheral image displayed on the in-vehicle monitor, an image of a three-dimensional object is superimposed on the position of the obstacle and displayed to allow the user to grasp the presence of the obstacle in advance.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 discloses that when a vehicle approaches an obstacle, the display color of a 3D object may change or the display may flash, and these are basically intended to ensure that the user is aware of the presence of the obstacle without overlooking it. However, there was a problem in that it was difficult to make the presence of the obstacle known depending on the positional relationship between the vehicle and the obstacle. For example, in the case of a sign 101 supported by a thin pillar as shown in Figure 13, if the vehicle 102 approaches the sign 101 too closely, the sign 101 will move out of the imaging range of the in-vehicle camera 103 mounted on the vehicle 102 and will move outside the display screen of the in-vehicle monitor. In that case, even if a 3D object is displayed, it becomes difficult to recognize the presence of the sign 101 on the display screen of the in-vehicle monitor, and the recognition of the presence of the sign 101 will be diminished.

[0006] The present invention has been made to solve the aforementioned problems of the conventional invention, and aims to provide a driving assistance device that can recognize the presence of a stationary object by notifying the vehicle occupant of the presence of the object, even when at least a portion of the area of ​​the stationary object displayed on the display device moves from within the display screen to outside the display screen. [Means for solving the problem]

[0007] To achieve the above objective, the driving assistance device according to the present invention displays the surrounding environment in the direction of travel of the vehicle, specifically the surrounding environment within the imaging range captured by an imaging device installed on the vehicle, on a display device, and when a stationary object affecting the movement of the vehicle is detected in the direction of travel of the vehicle, and at least a portion of the area of ​​the stationary object displayed on the display device moves from within the display screen of the display device to outside the display screen, the presence of the stationary object is notified to the occupant of the vehicle. Furthermore, "displaying the surrounding environment within the imaging range captured by the imaging device installed on the vehicle, within the surrounding environment in the direction of travel of the vehicle, on the display device" means that the captured image taken by the imaging device installed on the vehicle may be displayed as is, a part of the captured image taken by the imaging device may be cropped and displayed, an image in which the viewpoint has been virtually transformed from the captured image taken by the imaging device may be displayed, or an image in which captured images taken by multiple imaging devices have been combined. Furthermore, "detecting stationary objects that affect the movement of the vehicle" may mean detecting stationary objects that are expected to approach or come into contact with when the vehicle moves while maintaining its current steering, or stationary objects that are expected to approach or come into contact with when the vehicle moves under autonomous driving conditions, or stationary objects within a predetermined area as seen from the current position of the vehicle, or stationary objects located within the imaging range captured by an imaging device installed on the vehicle. Furthermore, the "stationary objects that affect vehicle movement" that are subject to detection include stationary objects that vehicles must avoid in order to move, such as structures like signs, walls, and utility poles, as well as natural objects like trees and rocks. [Effects of the Invention]

[0008] According to the driver assistance device of the present invention having the above configuration, even if at least a portion of the area of ​​a stationary object displayed on the display device moves from within the display screen of the display device to outside the display screen, it is possible to notify the occupants of the vehicle and allow them to recognize the presence of the stationary object. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This is a block diagram showing the configuration of the driver assistance system according to this embodiment. [Figure 3] This is a flowchart of the driver assistance processing program according to this embodiment. [Figure 4] This is a diagram illustrating the influence of stationary objects. [Figure 5]This diagram shows the change in the relative positions of a vehicle and an influencing stationary object when the vehicle is moving in reverse. [Figure 6] This diagram illustrates the conditions under which it is determined that an influencing stationary object has moved from inside the display screen of a liquid crystal display to outside the display screen. [Figure 7] This diagram illustrates the conditions under which it is determined that an influencing stationary object has moved from inside the display screen of a liquid crystal display to outside the display screen. [Figure 8] This figure illustrates an example of extracting cut-in images from previously captured images. [Figure 9] This figure shows an example of displaying a cut-in image. [Figure 10] This figure shows an example of displaying an object image instead of a cut-in image. [Figure 11] This diagram shows an example of displaying text instead of an in-image cut-in. [Figure 12] This diagram shows an example of displaying a cut-in image and an object image overlapping. [Figure 13] This diagram illustrates the problems with conventional technology. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the driver assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the driver assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0012] In addition to manual driving in which the vehicle 2 travels based on the user's driving operation, the vehicle 2 may also be a vehicle capable of assisted driving in which the vehicle automatically travels without depending on the user's driving operation.

[0013] In addition, the vehicle 2 performs driving assistance to display the surrounding environment within the imaging range captured by the front camera 6 or the rear camera 7, which is an imaging device installed in the vehicle 2, on the liquid crystal display 4 inside the vehicle among the surrounding environments in the traveling direction. Note that the display of the above surrounding environment may be performed only under specific situations such as when parking or leaving the warehouse, or may be always performed during traveling. Alternatively, it may be performed only when it is selected by the user to display the surrounding environment.

[0014] More specifically, in the above driving assistance, the surrounding environment within the imaging range captured by the camera corresponding to the traveling direction of the vehicle 2 (the front camera 6 when moving forward and the rear camera 7 when moving backward) is displayed on the liquid crystal display 4. In addition to directly visually observing the periphery of the vehicle, the user can confirm the safety in the traveling direction of the vehicle 2 by visually recognizing the display screen of the liquid crystal display 4. In particular, when moving backward, there is a problem that it is difficult to directly visually confirm the safety behind the vehicle, but by visually recognizing the liquid crystal display 4, it becomes easy to confirm the safety behind the vehicle. Further, in the present embodiment, as will be described later, when a stationary object that affects the movement of the vehicle 2 is detected in the traveling direction of the vehicle 2, and at least a partial area of the stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, the presence of the stationary object is also notified to the passengers of the vehicle 2. Thereby, even if the stationary object moves outside the display screen of the liquid crystal display 4, it is possible to surely grasp its presence.

[0015] In addition, as shown in FIG. 1, the vehicle 2 includes a liquid crystal display 4 that displays an imaging image of the surrounding environment in the traveling direction of the vehicle for the occupant and other information related to driving support, a speaker 5 that outputs voice guidance related to driving support, a front camera 6, a rear camera 7, side cameras 8A and 8B for imaging the periphery of the vehicle, ultrasonic sensors 9A to 9L for detecting obstacles around the vehicle, and a driving support ECU (Electronic Control Unit) 10 that performs various arithmetic processes based on the input information. Here, the driving support device 1 includes the driving support ECU 10.

[0016] Each component included in the vehicle 2 will be described below. First, the liquid crystal display 4 is a type of display device, which is installed on the instrument panel of the vehicle 2. During the running of the vehicle 2, among the surrounding environment in the traveling direction of the vehicle 2, the surrounding environment within the imaging range captured by a camera corresponding to the traveling direction installed on the vehicle 2 (the front camera 6 when moving forward and the rear camera 7 when moving backward) is displayed on the liquid crystal display 4. In addition, particularly when a stationary object that affects the movement of the vehicle 2 in the traveling direction of the vehicle 2 is detected, and at least a partial area of the stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, past camera imaging images, object images, etc. are also additionally displayed to notify the occupant of the vehicle 2 of the presence of the stationary object. Note that the liquid crystal display 4 may be used in combination with a navigation device.

[0017] In addition, the speaker 5 is installed on the instrument panel of the vehicle 2 and outputs guidance voices, warning sounds, etc. related to driving support. Note that the speaker 5 may be used in combination with a navigation device.

[0018] In addition, the front camera 6 is an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror with the optical axis direction facing forward in the traveling direction of the vehicle.

[0019] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0020] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0021] As mentioned above, the images captured by the front camera 6 and the rear camera 7 are displayed on the LCD display 4 as needed while the vehicle is in motion. The driver assistance ECU 10 can also generate bird's-eye and overhead views of the area around the vehicle by performing viewpoint transformation and synthesis processing on the images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B, and display these on the LCD display 4 as well. Furthermore, by performing image recognition processing on the captured images, it is possible to detect lane markings and obstacles around the vehicle. Obstacle detection using image recognition processing can also identify the type of obstacle, and the obstacles to be detected include stationary objects that affect the movement of the vehicle 2 (for example, structures such as signs, walls, and utility poles, and natural objects such as trees and rocks).

[0022] The ultrasonic sensors 9A to 9L are positioned at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, the ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by the ultrasonic sensors 9A to 9L include, for example, people, bicycles, other vehicles, walls, and other obstacles that the vehicle 2 needs to avoid while driving. In addition, millimeter-wave sensors or laser sensors may be used as distance measuring sensors instead of ultrasonic sensors. Furthermore, in this embodiment, as described above, it is also possible to detect obstacles by performing image recognition on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Therefore, if obstacle detection is performed using the above cameras, the ultrasonic sensors 9A to 9L may be omitted.

[0023] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. It performs various controls, calculations, and management in driver assistance. The driver assistance ECU 10 is connected to the aforementioned liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, and steering sensor, as well as in-vehicle devices such as the navigation system. The detailed configuration of the driver assistance ECU 10 will be described later.

[0024] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as a vehicle; however, only the configuration related to the control of driver assistance using the liquid crystal display 4, and the control related to said configuration will be explained.

[0025] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0026] As shown in Figure 2, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1, and includes an internal storage device such as a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores generated driving trajectory data, a ROM 33 which stores control programs as well as driver assistance processing programs (see Figure 3) described later, and a flash memory 34 which is a storage medium that stores programs read from the ROM 33. The driver assistance ECU 10 also has various functions as a processing algorithm. For example, it has a function to display the surrounding environment within the imaging range captured by the front camera 6 or rear camera 7 installed on the vehicle 2 on the liquid crystal display 4, and a function to notify the occupants of the vehicle 2 of the presence of a stationary object that affects the movement of the vehicle 2 when a stationary object that affects the movement of the vehicle 2 is detected in the direction of travel of the vehicle 2 and at least a part of the area of ​​the stationary object displayed on the liquid crystal display 4 moves from inside the display screen of the liquid crystal display 4 to outside the display screen of the liquid crystal display 4.

[0027] Furthermore, the driver assistance ECU 10 is connected to various sensors 37 for detecting the vehicle's behavior, such as a vehicle speed sensor, acceleration sensor, and steering sensor, as well as to various drive units 38 of the vehicle, such as the steering, brakes, and accelerator. Based on the detection results of these sensors 37, it detects the vehicle's current behavior. It is also possible to perform various driver assistance functions for the vehicle 2 by controlling each drive unit 38. Specific examples of driver assistance include, for example, controlling the deceleration or stopping of the vehicle 2 when an obstacle is detected within a predetermined distance from the vehicle 2, and automatic parking assistance that automatically parks the vehicle in a parking space.

[0028] Furthermore, ROM33 includes vehicle information DB35, which stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position), detection axis X, overall length, vehicle width, wheelbase, and minimum turning radius of cameras and ultrasonic sensors 9A-9L installed on vehicle 2. In particular, information regarding the optical axis and field of view of the cameras is also stored. This information is to be entered in advance by the occupants or personnel from the vehicle manufacturer.

[0029] Meanwhile, the flash memory 34 includes an image database 36, which stores images previously captured by the front camera 6 and the rear camera 7. Furthermore, if a stationary object affecting the movement of the vehicle 2 is detected in the direction of travel of the vehicle 2, as described later, and at least a portion of the area of ​​the stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, past images from the cameras stored in the image database 36 are added to the display on the liquid crystal display 4 in order to inform the occupants of the vehicle 2 of the presence of the stationary object. Details will be described later.

[0030] Next, the driver assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the driver assistance processing program according to this embodiment. Here, the driver assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that assists the user by displaying the surrounding environment in the direction of travel of the vehicle on the liquid crystal display 4.

[0031] In the following embodiment, the explanation will be based on the case where the vehicle 2 is reversing for the purpose of parking, and the surrounding environment in the direction of travel of the vehicle, i.e., the area behind the vehicle, is displayed on the liquid crystal display 4. However, it is not necessary to display the surrounding environment in the direction of travel of the vehicle only while reversing; it may also be displayed when moving forward or when stopped. For example, when the vehicle is moving forward, the surrounding environment in front of the vehicle is displayed on the liquid crystal display 4. When the vehicle is stopped, the surrounding environment in the direction of travel of the vehicle corresponding to the direction of travel of the vehicle immediately before or the current shift position is displayed on the liquid crystal display 4. Furthermore, it is not limited to when driving manually, but the surrounding environment in the direction of travel of the vehicle may also be displayed on the liquid crystal display 4 when driving automatically. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the driver assistance device 1 and executed by the CPU 31.

[0032] First, in step 1 (hereinafter abbreviated as S), the CPU 31 acquires a real-time image showing the surrounding environment behind the vehicle 2, which is the direction of travel of the vehicle, within the imaging range captured by the rear camera 7 installed on the vehicle 2, and begins displaying it on the liquid crystal display 4. The image displayed on the liquid crystal display 4 in S1 should be an image that allows the occupants of the vehicle 2 to confirm the environment behind the vehicle, which is the direction of travel of the vehicle 2. This may be the image captured by the rear camera 7 itself, a cropped portion of the image captured by the rear camera 7, an image with a virtually transformed viewpoint of the image captured by the rear camera 7, or an image created by combining images captured by multiple imaging devices, for example, a bird's-eye view looking down at the direction of travel of the vehicle from above. The display of the surrounding environment on the liquid crystal display 4 will continue until this driving support processing program is terminated (S10: YES).

[0033] Next, in S2, the CPU 31 starts storing the images captured by the rear camera 7 in the image DB 36 of the flash memory 34 in parallel with displaying them on the liquid crystal display 4. As with S1, the storage of images captured by the rear camera 7 continues until this driving support processing program ends (S10: YES). If the amount of stored data reaches the upper limit, the oldest data is deleted in order and new images are sequentially overwritten and stored.

[0034] Next, in S3, the CPU 31 determines whether or not it has detected a stationary object (hereinafter referred to as an "influencing stationary object") that affects the vehicle's movement in the direction of travel. An "influencing stationary object" is a stationary object that the vehicle must avoid in order to move, and includes structures such as signs, walls, and utility poles, as well as natural objects such as trees and rocks. On the other hand, moving objects such as pedestrians and other vehicles (however, parked vehicles may be considered stationary objects), and stationary objects that do not affect the vehicle's movement, such as small steps or signs that are higher than the vehicle's height, are excluded from being influencing stationary objects. Furthermore, influencing stationary objects may be limited to stationary objects specifically installed on the road surface. In that case, movable objects such as vehicles are excluded from being influencing stationary objects, even if they are stopped at the time of detection. Furthermore, in S3, stationary objects that are located within the imaging range captured by the rear camera 7 are detected as stationary objects in the direction of travel of the vehicle. However, the detection range may be narrower. For example, stationary objects located within a predetermined area as seen from the current vehicle 2 may be detected as stationary objects in the direction of travel of the vehicle, or only stationary objects that are expected to approach or come into contact with the vehicle when it moves while maintaining its current steering may be detected.

[0035] For example, Figure 4 shows a vehicle 2 and stationary objects 42 and 43 when parking in an area enclosed by parking space lines 41 as the parking target position. As shown in Figure 4, both stationary objects 42 and 43 are located in the direction of the vehicle's movement, but stationary object 42 is located at a height greater than the overall height of vehicle 2, so even if vehicle 2 reverses to the parking target position, there is no possibility of physical contact with vehicle 2. Therefore, stationary object 42 does not affect the movement of vehicle 2 and is excluded from the list of influencing stationary objects. On the other hand, stationary object 43 is located at a height less than the overall height of vehicle 2, so if vehicle 2 reverses to the parking target position, there is a possibility of contact with vehicle 2. Therefore, stationary object 43 affects the movement of vehicle 2 and is considered an influencing stationary object. Furthermore, even if stationary objects are located in the direction of vehicle 2's movement and at a height less than the overall height of vehicle 2, it is desirable to exclude stationary objects that are far from vehicle 2 or that are located far from the expected path of vehicle 2 from the list of influencing stationary objects.

[0036] Furthermore, the detection of the influencing stationary objects described above may be performed by image recognition on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or by ultrasonic sensors 9A to 9L, or by using a combination of both. The process for detecting influencing stationary objects from the captured images may involve, for example, brightness correction based on the brightness difference between the road surface and the influencing stationary objects on the road surface, followed by binarization to separate the influencing stationary objects from the image, geometric processing to correct distortion, and smoothing to remove image noise, thereby detecting the boundary line between the road surface and the influencing stationary objects. If an influencing stationary object is detected, its position and shape are also identified. If multiple influencing stationary objects are detected, the position and shape are identified for each object. When determining whether a detected object is an influencing stationary object, known template matching or feature point detection processes can be used. Furthermore, the image recognition processing on the captured images is not limited to the above example; for example, machine learning may be used.

[0037] If it is determined that a stationary object affecting the vehicle's movement has been detected in the direction of travel (S3:YES), the process proceeds to S4. Conversely, if it is determined that no stationary object affecting the vehicle's movement has been detected in the direction of travel (S3:NO), the process proceeds to S10.

[0038] In S4, the CPU 31 stores information about the stationary objects in the direction of travel of the vehicle 2 that were detected in S3, including their location and shape, in a storage medium such as RAM 32.

[0039] Next, in S5, the CPU 31 begins to store the vehicle 2's driving record. In other words, the CPU 31 continues to store information obtained from various sensors 37 and various drive units 38 (for example, vehicle position, vehicle speed, acceleration, steering angle, etc.) in the RAM 12 until this driving support processing program ends (S10: YES).

[0040] Subsequently, in S6, the CPU 31 begins monitoring the positional relationship between vehicle 2 and the influencing stationary object. Specifically, the positional relationship between the influencing stationary object detected in S3 and recorded in S4 and vehicle 2 is calculated based on the vehicle's own driving record, which was started in S5. While known and conventional techniques can be used for this monitoring, a brief explanation will be given of the case where it is implemented using SLAM (Simultaneous Localization and Mapping) as an example.

[0041] The CPU 31 extracts feature points from the image area captured by the rear camera 7 and creates a map using these feature points. Based on this, the CPU 31 estimates the current position of vehicle 2 on the map based on the vehicle's own driving record, which was started in S5, and simultaneously updates the map as new feature points are obtained. Furthermore, by using the stationary object information stored in the RAM 32 in S4, the location of influencing stationary objects within the map can also be identified. Based on the position of vehicle 2 and the positions of the influencing stationary objects on the map, the positional relationship between vehicle 2 and the influencing stationary objects can be calculated.

[0042] Here, as shown in Figure 5(4) below, if the influencing stationary object approaches vehicle 2, the rear camera 7, other cameras, and ultrasonic sensors 9A~9L may not be able to detect the influencing stationary object. Even in such cases, by comparing the current position of vehicle 2 on the map as described above with the position of the influencing stationary object, it is possible to identify (track) the positional relationship between vehicle 2 and the undetectable influencing stationary object. Since the position of the influencing stationary object is fixed, once the position of the influencing stationary object is identified on the map, the positional relationship between vehicle 2 and the influencing stationary object can be identified even if the position of the influencing stationary object becomes undetectable afterward.

[0043] Next, in S7, the CPU 31 determines whether a portion of the influencing stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen. Specifically, if the entire influencing stationary object is within the imaging range of the rear camera 7, the CPU 31 identifies the external shape of the influencing stationary object displayed on the liquid crystal display 4 through image recognition processing, and further determines whether the external shape of the influencing stationary object has reached the edge 50 of the display screen of the liquid crystal display 4 by performing image recognition processing on the image displayed on the liquid crystal display 4. If the external shape of the influencing stationary object has reached the edge 50 of the display screen of the liquid crystal display 4, the CPU 31 determines that a portion of the influencing stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen (S7: YES).

[0044] For example, Figure 5 shows the positional relationship between vehicle 2 and the influencing stationary object 43 when vehicle 2 is reversing. As vehicle 2 moves in reverse, the position transitions sequentially from (1) to (4), and the distance between vehicle 2 and the influencing stationary object 43 gradually decreases. Furthermore, Figure 6 shows the display screen shown on the liquid crystal display 4 near Figure 5(2). In Figure 6, the influencing stationary object 43 has reached the edge 50 of the liquid crystal display 4. As a result, the CPU 31 determines that a portion of the influencing stationary object 43 has moved from inside the display screen of the liquid crystal display 4 to outside the display screen. Information to determine the position of the edge 50 of the display screen of the liquid crystal display 4 is stored in the ROM 33 or the like beforehand.

[0045] Alternatively, in S7, the CPU 31 may determine whether a portion of the influencing stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen by calculating the positional relationship between the vehicle 2 and the influencing stationary object using that positional relationship. Specifically, the CPU 31 acquires the positional relationship between the vehicle 2 and the influencing stationary object identified in S6, the shape of the influencing stationary object, and the parameters of the rear camera 7. The parameters of the rear camera 7 include, for example, the installation position relative to the vehicle 2, the optical axis, and the field of view. The CPU 31 then uses this information to calculate and determine (estimate) whether a portion of the influencing stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen.

[0046] On the other hand, in cases where the influencing stationary object is relatively large compared to the imaging range of the rear camera 7, only a portion of the influencing stationary object may be included within the imaging range of the rear camera 7. That is, when the execution of the driving assistance processing program is started and the influencing stationary object is detected, a portion of the detected influencing stationary object may already be located outside the display screen of the liquid crystal display 4. In such cases, in S7, the CPU 31 determines whether at least a portion of the influencing stationary object that was included within the display screen of the liquid crystal display 4 at the time of detection has moved from within the display screen of the liquid crystal display 4 to outside the display screen, and whether the vehicle 2 and the influencing stationary object are approaching each other. For example, if the display screen was as shown in Figure 7 when the execution of the driving assistance processing program was started, the CPU 31 monitors the influencing stationary object 43 by image recognition processing to determine whether at least a portion of the influencing stationary object 43 has moved from within the display screen of the liquid crystal display 4 to outside the display screen, and also determines whether the vehicle 2 and the influencing stationary object 43 are approaching each other by monitoring the positional relationship between the vehicle 2 and the influencing stationary object 43, which was started in S6. Furthermore, if at least a portion of the area of ​​the influencing stationary object that was included within the display screen of the liquid crystal display 4 at the time the influencing stationary object was detected moves from within the display screen of the liquid crystal display 4 to outside the display screen, and the vehicle 2 and the influencing stationary object are approaching each other, then it is determined that a portion of the area of ​​the influencing stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen (S7:YES).

[0047] If it is determined that a portion of the stationary object within the liquid crystal display 4 has moved from within the display screen to outside the display screen (S7:YES), the process proceeds to S8. Conversely, if it is determined that a portion of the stationary object within the liquid crystal display 4 has not moved from within the display screen to outside the display screen (S7:NO), the process proceeds to S10.

[0048] In S8, the CPU 31 generates information to notify the vehicle occupants of the presence of an influencing stationary object that has moved from within the display screen to outside the display screen. First, the CPU 31 extracts an image range from the past captured images that were started to be stored in S2, which includes the influencing stationary object that was determined to have moved from within the display screen to outside the display screen in S6. The image extracted in S6 will be referred to as the cut-in image below. Here, when extracting the cut-in image, if it is possible to extract an image range that includes the entire shape of the influencing stationary object, the image range that includes the entire influencing stationary object will be given priority. For example, as shown in Figure 8, if there is a past captured image 51 that includes the entire influencing stationary object 43, it is desirable to extract the narrowest possible image range that includes the entire influencing stationary object 43 from the captured image 51 as the cut-in image 52. Furthermore, if it is possible to extract multiple image ranges that include the entire shape of the influencing stationary object (when there are many past captured images that capture the entire influencing stationary object), it is desirable to select the image range that makes it easiest for the occupants to recognize the influencing stationary object and extract it as the cut-in image.

[0049] For example, let's consider the case where, as shown in Figure 5, the vehicle 2 moves backward, causing the influencing stationary object 43 to move from inside the display screen of the liquid crystal display 4 to outside the display screen. In S2, the CPU 31 extracts an image range containing the influencing stationary object 43 as a cut-in image from the previously captured images that were started to be stored. At this time, the CPU 31 can extract an image range that includes the entire shape of the influencing stationary object 43 during the transition from Figure 5(1) to (2), but the positional relationship near Figure 5(2) allows for the extraction of an image range that captures a larger portion of the entire influencing stationary object 43 than the positional relationship near Figure 5(1). Therefore, it is desirable to extract an image range containing the influencing stationary object as a cut-in image from the captured image taken at the positional relationship near Figure 5(2).

[0050] On the other hand, if, when the driver assistance processing program is started and the influencing stationary object is detected, a portion of the detected influencing stationary object is located outside the display screen of the liquid crystal display 4, for example, if the display screen of the liquid crystal display 4 was as shown in Figure 7 at the time the influencing stationary object was detected, then in S8, it is not possible to extract an image range that includes the entire influencing stationary object. However, there may be images captured by the rear camera 7 before the influencing stationary object is detected that include the entire influencing stationary object (for example, images captured when the influencing stationary object is located far from the vehicle). Therefore, in cases where, when the influencing stationary object is detected, a portion of the detected influencing stationary object is located outside the display screen of the liquid crystal display 4, if it is possible to extract an image range that includes the entire influencing stationary object from images captured by the rear camera 7 before the influencing stationary object was detected, the image range that includes the entire influencing stationary object is extracted as a cut-in image from those images. On the other hand, if it is not possible to extract an image range containing the entire influencing stationary object from the image captured by the rear camera 7 before the influencing stationary object is detected, the image captured by the rear camera 7 at the time the influencing stationary object is detected is expected to contain the influencing stationary object that is closest to its overall shape. Therefore, the image range containing the influencing stationary object is extracted from that image as a cut-in image.

[0051] Furthermore, in S8, the CPU 31 sets the display position and size of the extracted cut-in image. The display position of the cut-in image is set to a position corresponding to the influencing stationary object that has moved outside the display screen, that is, a position that suggests the position of the influencing stationary object. For example, as shown in Figure 6, if the influencing stationary object is displayed in the upper right of the display screen of the liquid crystal display 4, and then the influencing stationary object gradually moves to the right, causing a portion of the area to move from inside to outside the display screen, the display position of the cut-in image is set to the right-hand area of ​​the display screen. This suggests that the influencing stationary object is located outside the display screen to the right of the liquid crystal display 4. Although not shown in the illustration, if the influencing stationary object gradually moves to the left, causing a portion of the area to move from inside to outside the display screen, the display position of the cut-in image is set to the left-hand area of ​​the display screen. However, if a portion of the influencing stationary object remains within the display screen, it is desirable to set the display position of the cut-in image so as not to overlap with the influencing stationary object. On the other hand, if the image of the influencing stationary object remaining within the display screen is sufficiently larger than the cut-in image, the display position may be set to overlap with the influencing stationary object. On the other hand, the display size of the cut-in image can be fixed, or it can be changed depending on the size and type of the stationary object influencing it. As described later, the cut-in image is displayed superimposed on the real-time image captured by the rear camera 7, so it is desirable to set it to a size that does not obstruct the view of the real-time image.

[0052] Subsequently, in S9, the CPU 31 displays the cut-in image extracted in S8 on the liquid crystal display 4 at the same position and size set in S8 (cut-in display). This notifies the occupants of the vehicle 2 of the presence of a stationary object that has moved outside the display screen. For example, Figure 9 shows an example of the display screen of the liquid crystal display 4 displayed in S9.

[0053] As shown in Figure 9, the liquid crystal display 4 displays the real-time captured image 51 taken by the rear camera 7 in S1, while a cut-in image 52 is also displayed superimposed on the captured image 51. Figure 9 shows the case where the influencing stationary object 43 moves to the right, and a portion of it moves from within the display screen to outside the display screen. The display position of the cut-in image 52 is the right-hand region within the display screen of the liquid crystal display 4. This suggests that the influencing stationary object 43 is located to the right of the display screen of the liquid crystal display 4. By viewing the display screen shown in Figure 9, the occupants of the vehicle 2 can recognize the existence of the influencing stationary object 43, as well as its location and external shape. Specifically, they can estimate the approximate location of the influencing stationary object 43 outside the display screen based on the display position of the cut-in image 52. In addition, although only a portion of the influencing stationary object 43 is displayed in the captured image 51, the entire external shape of the influencing stationary object 43 is displayed in the cut-in image 52, making it easy to understand what the external shape of the influencing stationary object 43 is.

[0054] In addition, although Figure 9 shows a case where a part of the influencing stationary object 43 is included in the captured image 51, even if the entire area of ​​the influencing stationary object 43 subsequently moves from within the display screen to outside the display screen, the occupants of vehicle 2 can continue to recognize the existence of the influencing stationary object 43, its position, and its external shape by continuing to display the cut-in image 52. Furthermore, in this embodiment, as described above, the positional relationship between the current position of vehicle 2 and the position of the influencing stationary object is monitored on the map (S6), so even if the influencing stationary object moves outside the imaging range of the rear camera 7, it is possible to identify (track) its position relative to vehicle 2 (in which direction the influencing stationary object is located outside the display screen of the liquid crystal display 4).

[0055] Furthermore, the cut-in image 52 may be displayed semi-transparently (for example, with a transmittance of 30%) rather than being opaque. This prevents a decrease in the visibility of the real-time captured image 51 due to the display of the cut-in image 52.

[0056] In the example described above, the image range containing the influencing stationary object is extracted as a cut-in image from past captured images stored in the captured image DB36, and the extracted cut-in image is added to the surrounding environment displayed on the liquid crystal display 4 to provide notification of the influencing stationary object. However, instead of a cut-in image, the above notification may be provided by adding an object image or text that suggests the presence of the influencing stationary object.

[0057] In this case, the CPU 31 sets the display position, size, shape, and text content for displaying an object image or text that suggests the presence of the influencing stationary object in S8. For example, when displaying an object image, the shape of the object image may be fixed, or it may be a shape that mimics the influencing stationary object that has been determined to have moved from inside the display screen to outside the display screen (for example, an object image of the shape of a sign if it is a sign). When displaying text, the text may describe the type, shape, or name of the influencing stationary object, or it may simply be a text indicating that the influencing stationary object is outside the display screen. The display position and size of the object image and text are set in the same way as when displaying a cut-in image as described earlier.

[0058] For example, Figure 10 shows an example of the display screen of the liquid crystal display 4 when an object image 61 is displayed instead of a cut-in image. In the example shown in Figure 10, the object image 61 is a rectangular parallelepiped with a wall shape. By displaying an object image 61 with a wall shape, the occupants can recognize that there is some factor outside the display screen that is hindering the movement of the vehicle 2, and in particular, it is possible to prompt the occupants of the vehicle to change course or stop moving in order to avoid contact with the object image 61. On the other hand, Figure 11 shows an example of the display screen of the liquid crystal display 4 when text 62 is displayed instead of a cut-in image. By displaying text 62, it is possible to more directly make the occupants of the vehicle 2 aware that there is a stationary object affecting them located outside the display screen.

[0059] Furthermore, the cut-in image 52, object image 61, and text 62 described above may be displayed redundantly. For example, Figure 12 shows an example of the display screen of the liquid crystal display 4 when the cut-in image 52 and object image 61 are displayed redundantly. In the example shown in Figure 12, an object image 61 that suggests the presence of an influencing stationary object is added and displayed at a position corresponding to the location of the influencing stationary object 43 that has moved outside the display screen, and the cut-in image 52 is displayed superimposed on the object image 61 to provide notification. This allows the presence and external shape of the influencing stationary object that has moved outside the display screen to be recognized, and prompts the vehicle occupants to change their course or stop moving to avoid contact with the object image 61. Although not shown in the illustration, the object image 61 and text 62 may also be displayed redundantly, or the cut-in image 52 and text 62 may also be displayed redundantly. Furthermore, voice guidance may also be provided in parallel.

[0060] Subsequently, in S10, the CPU 31 determines whether or not to terminate the driver assistance. Here, the termination of the driver assistance may be conditional on, for example, the user performing a predetermined termination operation, or on the shift position being moved to "P" or the ACC power being turned off.

[0061] Then, if it is determined that the driver assistance has ended (S10:YES), the driver assistance processing program is terminated. On the other hand, if it is determined that the driver assistance should be continued (S10:NO), the process returns to S3, and the display of captured images and cut-in images on the liquid crystal display 4 continues.

[0062] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, the surrounding environment in the direction of travel of the vehicle 2, within the imaging range captured by the front camera 6 and rear camera 7 installed on the vehicle 2, is displayed on the liquid crystal display 4 (S1). When an influencing stationary object that affects the movement of the vehicle 2 is detected in the direction of travel of the vehicle 2, and at least a portion of the influencing stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, the presence of the influencing stationary object is notified to the occupants of the vehicle 2 (S9), thereby making it possible to recognize the presence of the influencing stationary object. Furthermore, the images captured by the front camera 6 and the rear camera 7 are stored in a storage medium such as memory (S2). When at least a portion of the area of ​​the influencing stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, the image range including the influencing stationary object is extracted from past captured images stored in the storage medium (S8). The extracted image range is then added to the surrounding environment displayed on the liquid crystal display 4 (S9) to provide notification. Therefore, even if the influencing stationary object is not displayed on the liquid crystal display 4 or only partially displayed, the vehicle occupants can easily grasp the presence, location, and external shape of the influencing stationary object by viewing the additionally displayed image. Furthermore, if at least a portion of the area of ​​the affected stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen, notification is provided by adding an object image or text suggesting the presence of the affected stationary object to the surrounding environment displayed on the liquid crystal display 4 at a position corresponding to the location of the stationary object that has moved outside the display screen. Therefore, even if the affected stationary object is not displayed on the liquid crystal display 4 or only partially displayed, the vehicle occupants can easily grasp the presence and location of the affected stationary object by visually inspecting the additionally displayed object image or text. Furthermore, if an influencing stationary object is detected in the direction of travel of vehicle 2, and a portion of the detected object is already located outside the display screen of the liquid crystal display 4 at the time of detection, notification will be given when at least a portion of the object that was included within the display screen of the liquid crystal display 4 at the time of detection moves from within the display screen of the liquid crystal display 4 to outside the display screen, and vehicle 2 approaches the influencing object. This makes it possible to make the vehicle occupants aware of the presence of an influencing object that is already partially outside the display screen at the time of detection.

[0063] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, the situation in which vehicle 2 is reversing for the purpose of parking was used as an example for explanation, but the driving assistance processing program shown in Figure 3 does not necessarily have to be executed only while reversing, but may also be executed when moving forward or when stopped. In that case, in S1, for example, when the vehicle is moving forward, the surrounding environment in front of the vehicle captured by the front camera 6 is displayed on the liquid crystal display 4. When the vehicle is stopped, the surrounding environment in the direction of travel of the vehicle immediately before or the direction of travel corresponding to the current shift position is displayed on the liquid crystal display 4. In addition to the front camera 6 and rear camera 7, images captured by side cameras 8A and 8B may also be displayed on the liquid crystal display 4.

[0064] Furthermore, when displaying the image captured by the rear camera 7 in S1, the image captured by the rear camera 7 itself may be displayed, or a part of the image captured by the rear camera 7 may be cropped and displayed, or an image in which the viewpoint has been virtually transformed from the image captured by the rear camera 7 may be displayed, or an image created by combining images captured by multiple imaging devices may be displayed, for example, a bird's-eye view image looking down at the direction of travel of the vehicle from above at an angle downwards.

[0065] Furthermore, in this embodiment, when at least a portion of the area of ​​the influencing stationary object displayed on the liquid crystal display 4 moves from within the display screen of the liquid crystal display 4 to outside the display screen (S7:YES), a cut-in image 52 is displayed to inform the occupants of the vehicle 2 of the presence of the influencing stationary object. However, the cut-in image 52 may be displayed to inform the occupants of the vehicle 2 of the presence of the influencing stationary object when the entire area of ​​the influencing stationary object moves from within the display screen of the liquid crystal display 4 to outside the display screen, that is, when the influencing stationary object completely disappears from the liquid crystal display 4.

[0066] Furthermore, the execution order of each step in the driver assistance processing program shown in Figure 3 is just an example, and the execution order can be changed as appropriate. For example, although the recording of stationary object information in S4 is followed by the recording of the vehicle's driving record in S5, the recording of stationary object information may be performed after the recording of the vehicle's driving record. It is also possible to omit some of the steps included in the driver assistance processing program; for example, steps S4 to S6 may be omitted.

[0067] Furthermore, in this embodiment, the driver assistance ECU 10 of the driver assistance device 1 executes the processing of the driver assistance processing program (Figure 3), but the execution entity can be changed as appropriate. For example, the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle devices may be used to execute the processing.

[0068] [Note] The embodiments described above also illustrate the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0069] (Invention A) When the entire stationary object (43) is included within the imaging range, the external shape of the stationary object displayed on the display device (4) is identified. The driving assistance device (1) according to claim 1, wherein when the external shape of the stationary object reaches the edge (50) of the display screen of the display device (5), it is determined that at least a portion of the area of ​​the stationary object (43) displayed on the display device (4) has moved from inside the display screen of the display device (4) to outside the display screen.

[0070] According to this, it is possible to start notifying the vehicle occupants at the same time that at least a portion of a stationary object begins to move off-screen.

[0071] (Invention B) Information about the stationary object (43) is obtained, including information that identifies the position and shape of the stationary object. The driving support device (1) according to claim 1, which determines whether at least a portion of the area of ​​the stationary object (43) displayed on the display device (4) has moved from within the display screen of the display device (4) to outside the display screen, based on the parameters of the imaging device (6, 7) and the stationary object information.

[0072] According to this, it becomes possible to determine, through calculation, whether at least a portion of a stationary object has moved from within the display screen to outside the display screen of the display device, without performing image recognition processing on the image displayed on the screen.

[0073] (Invention C) The captured images taken by the imaging devices (6, 7) are stored in the storage medium (36). When at least a portion of the stationary object (43) displayed on the display device (4) moves from within the display screen of the display device to outside the display screen, an image range (52) including the stationary object is extracted from past captured images stored in the storage medium. The driving support device (1) according to claim 1, which provides notification by additionally displaying an object image (61) that suggests the presence of the stationary object on the display device (4) at a position corresponding to the position of the stationary object that has moved outside the display screen within the surrounding environment displayed on the display device (4), and by displaying the image range extracted and superimposed on the object image (61).

[0074] According to this, it becomes possible to make the vehicle occupants recognize the presence and external shape of a stationary object whose area has moved from within the display screen to outside the display screen, and to prompt the vehicle occupants to stop moving in order to avoid contact with the object image.

[0075] (Invention D) The driving support device according to claim 2, wherein when extracting the image range (52), if it is possible to extract an image range that includes the entire stationary object (43), the image range that includes the entire stationary object is given priority in extraction.

[0076] According to this, even if only a portion of a stationary object is displayed within the display range of the display device, the overall external shape of the stationary object can be displayed by the additionally displayed image, making it easy for vehicle occupants to understand the external shape of the stationary object.

[0077] (Invention E) If the stationary object (43) is detected in the direction of travel of the vehicle (2), and a portion of the stationary object detected at the time of detection is located outside the display screen of the display device (4), The driving support device (1) according to claim 2, in which case, when extracting the image range (52), the driving support device (1) extracts the image range including the stationary object from the captured image captured by the imaging device (7) at the time the stationary object is detected, or, if it is possible to extract an image range including the entire stationary object from the captured image captured before the stationary object is detected, the driving support device (1) extracts the image range including the entire stationary object.

[0078] According to this, even if a portion of a stationary object is already located outside the display screen of the device at the time of detection, it is possible to select and extract the image range that is easiest for the occupant to recognize the stationary object. [Explanation of symbols]

[0079] 1...Driving assistance system, 2...Vehicle, 4...Liquid crystal display (display device), 6...Front camera (imaging device), 7...Rear camera (imaging device), 10...Driving assistance ECU, 36...Image database, 43...Influencing stationary object (stationary object), 50...Edge of display screen, 52...Cut-in image (image range including stationary object), 61...Object image, 62...Text

Claims

1. The surrounding environment in the direction of travel of the vehicle, within the imaging range captured by the imaging device installed on the vehicle, is displayed on the display device. A driving assistance device that, when it detects a stationary object in the direction of travel of the vehicle that affects the movement of the vehicle, and when at least a portion of the area of ​​the stationary object displayed on the display device moves from within the display screen of the display device to outside the display screen, notifies the occupant of the vehicle of the presence of the stationary object.

2. The captured image taken by the aforementioned imaging device is stored in a storage medium. When at least a portion of the area of ​​the stationary object displayed on the display device moves from within the display screen of the display device to outside the display screen, the image range including the stationary object is extracted from the past captured images stored in the storage medium. The driving support device according to claim 1, which provides notification by adding the extracted image range to the surrounding environment displayed on the display device.

3. When at least a portion of the area of ​​the stationary object displayed on the display device moves from within the display screen of the display device to outside the display screen, The driving support device according to claim 1, which provides notification by additionally displaying an object image or text suggesting the presence of the stationary object at a position corresponding to the position of the stationary object that has moved outside the display screen, within the surrounding environment displayed on the display device.

4. When a stationary object is detected in the direction of travel of the vehicle, and a portion of the stationary object detected at the time of detection is located outside the display screen of the display device, The driving assistance device according to claim 1, which provides notification when at least a portion of the stationary object that was included in the display screen of the display device at the time the stationary object was detected moves from within the display screen of the display device to outside the display screen, and the vehicle and the stationary object are approaching each other.

Images