Driving assistance device, driving assistance method, and program

Abstract

The system appropriately notifies the user of the presence of an approaching object through a visual display. [Solution] A driver assistance device comprising: an image display control unit that displays surrounding images taken of the area around the vehicle by an in-vehicle camera on a display device and superimposes detection information of objects detected by an object detection sensor onto the surrounding images; and an alarm control unit that issues an alarm sound when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, further comprising: an enlarged display control unit that displays an enlarged image on the display device when it receives an enlarged display request from the user; and an enlarged display cancellation control unit that prohibits or cancels the display of the enlarged image if a predetermined enlarged display cancellation condition is met, even if the enlarged display control unit has received an enlarged display request, including cases where an alarm has been issued for the enlarged display cancellation condition.

Description

Technical Field

[0001] The present disclosure relates to a driving support device, a driving support method, and a program.

Background Art

[0002] For example, in Patent Document 1, in a driving support device that superimposes the detection range of an in-vehicle object detection sensor on a captured image captured by an in-vehicle camera, when the object detection sensor detects an obstacle, the display indicating the detection range is stopped, and a detection display of the obstacle is displayed, thereby making it easier for the occupant to recognize that there is an obstacle in the detection range. A technique is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] In a system that superimposes a detection display of an obstacle on a captured image as disclosed in Patent Document 1, there is a system having a function of enlarging and displaying a part of the captured image in response to a request from a user (mainly, a driver). In such a system, for example, when an approaching object is detected in a direction that has disappeared during the enlarged display, there is a problem that the user cannot be appropriately notified of the presence of the approaching object by display.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to appropriately notify a user of the presence of an approaching object by display.

[0006] The technology of the present disclosure is An image display control unit that displays surrounding images taken of the area around the vehicle by a camera mounted on the vehicle on a display device installed inside the vehicle, and superimposes detection information of objects detected by an object detection sensor mounted on the vehicle onto the surrounding image in correspondence with the position of the detected object, A driving assistance device comprising: an alarm control unit that issues an alarm sound when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, Upon receiving a request from a user for enlarged display, the system includes an enlarged display control unit that displays an enlarged image of the surrounding image on the display device, The system includes an enlargement display cancellation control unit that, even if the enlargement display control unit receives the enlargement display request, prohibits the display of the enlarged image on the display device or cancels the display of the enlarged image currently displayed on the display device if a predetermined enlargement display cancellation condition is met, The aforementioned enlarged display cancellation condition includes the case in which the alarm is issued by the alarm control unit. It is characterized by the following: [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram showing the hardware configuration of the vehicle according to this embodiment. [Figure 2] This is a schematic diagram showing the software configuration of the control device according to this embodiment. [Figure 3] (A) is a schematic diagram showing an example of a peripheral image according to this embodiment. (B) is a schematic diagram showing an example of a magnified image according to this embodiment. [Figure 4] This flowchart illustrates the routine for canceling the magnification control according to this embodiment. [Modes for carrying out the invention]

[0008] The driver assistance device, driver assistance method, and program according to this embodiment will be described below with reference to the drawings.

[0009] [Hardware configuration] Figure 1 is a schematic diagram showing the hardware configuration of vehicle VH according to this embodiment. Hereafter, vehicle VH may be referred to as "our vehicle" when it is necessary to distinguish it from other vehicles, etc.

[0010] Vehicle VH has an ECU (Electronic Control Unit) 10. The ECU 10 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, and an interface device 14, etc. The CPU 11 is a processor that executes various programs stored in the ROM 12. The ROM 12 is a non-volatile memory that stores data necessary for the CPU 11 to execute various programs. The RAM 13 is a volatile memory that provides a work area that is expanded when various programs are executed by the CPU 11. The interface device 14 is a communication device for communicating with external devices.

[0011] The ECU10 is a central control unit that provides various driving assistance functions. Driving assistance is a concept that includes autonomous driving. The ECU10 is connected to the drive unit 20, steering unit 21, braking unit 22, internal sensor device 30, external sensor device 40, HMI (Human Machine Interface) 60, etc., in a communication manner.

[0012] The drive unit 20 generates driving force that is transmitted to the drive wheels of the vehicle VH. Examples of the drive unit 20 include an electric motor and an engine. The steering unit 21 applies steering force to the wheels of the vehicle VH. The braking unit 22 applies braking force to the wheels of the vehicle VH.

[0013] The internal sensor device 30 consists of sensors that detect the state of the vehicle VH. Specifically, the internal sensor device 30 includes a vehicle speed sensor 31, an accelerator sensor 32, a brake sensor 33, a steering angle sensor 34, etc. The vehicle speed sensor 31 detects the vehicle speed, i.e., the vehicle speed of the vehicle VH. The accelerator sensor 32 detects the amount of accelerator pedal operation by the driver (not shown). The brake sensor 33 detects the amount of brake pedal operation by the driver (not shown). The steering angle sensor 34 detects the rotation angle of the steering wheel or steering shaft (not shown), i.e., the steering angle. The internal sensor device 30 transmits the state of the vehicle VH detected by each sensor 31 to 34 to the ECU 10 at a predetermined interval.

[0014] The external sensor device 40 consists of sensors that recognize target information regarding targets in the vicinity of the vehicle VH. Specifically, the external sensor device 40 includes a radar sensor 41, a sonar sensor 42, a camera sensor 43, and the like.

[0015] The radar sensor 41 includes a forward radar sensor that recognizes the area in front of the vehicle VH, a rear radar sensor that recognizes the area behind the vehicle VH, a left-side radar sensor that recognizes the left side of the vehicle VH, a right-side radar sensor that recognizes the right side of the vehicle VH, and the like. The radar sensor 41 includes a millimeter-wave radar and / or a lidar. The millimeter-wave radar emits radio waves in the millimeter-wave band and receives reflected waves reflected by targets within its radiation range to acquire the relative distance and relative speed between the vehicle VH and the target. The lidar sequentially scans pulsed laser light with a shorter wavelength than millimeter waves in multiple directions and receives reflected light reflected by the target to acquire the shape of the target, the relative distance and relative speed between the vehicle VH and the target.

[0016] The sonar sensor 42 includes a front sonar sensor that recognizes the front of the vehicle VH, a rear sonar sensor that recognizes the rear of the vehicle VH, a left-side sonar sensor that recognizes the left side of the vehicle VH, a right-side sonar sensor that recognizes the right side of the vehicle VH, and the like. The sonar sensor 42 emits ultrasonic waves and receives reflected waves reflected by a target existing within the emission range, thereby obtaining the relative distance, relative speed, etc. between the vehicle VH and the target.

[0017] The camera sensor 43 includes a front camera sensor that photographs the front of the vehicle VH, a rear camera sensor that photographs the rear of the vehicle VH, a left-side camera sensor that photographs the left side of the vehicle VH, a right-side camera sensor that photographs the right side of the vehicle VH, and the like. The camera sensor 43 is, for example, a stereo camera or a monocular camera, and a digital camera having an imaging element such as a CMOS or a CCD can be used. The camera sensor 43 obtains the shape of the target, the relative distance between the vehicle VH and the target, the relative speed, etc. by processing the photographed image data.

[0018] The external sensor device 40 transmits the acquired target information to the ECU 10 at a predetermined cycle. Note that the external sensor device 40 does not necessarily need to include all of the radar sensor 41, the sonar sensor 42, and the camera sensor 43, and it may include at least the sonar sensor 42 and the camera sensor 43.

[0019] The HMI 60 is an interface for performing information input / output between the ECU 10 and the driver, and includes an input device and an output device. Examples of the input device include a touch panel type liquid crystal display, a switch, a voice pickup microphone, etc. Examples of the output device include a display device 61, a speaker 62, etc. The display device 61 is, for example, a center display, a multi-information display, etc. The speaker 62 is, for example, a speaker of an audio system or a navigation system.

[0020] [Software Configuration] FIG. 2 is a schematic diagram showing the software configuration of the ECU 10 according to the present embodiment. As shown in FIG. 2, the ECU 10 includes, as functional elements, an object recognition unit 100, an alarm control unit 110, an image display control unit 120, an enlarged display control unit 130, an enlarged display cancel processing unit 140, and the like. Each of these functional elements 100 to 140 is realized by the CPU 11 of the ECU 10 reading out a program stored in the ROM 12 and executing it in the RAM 13. Note that all or part of each of the functional elements 100 to 140 can also be provided in another ECU separate from the ECU 10 or in an information processing device of a facility (such as a management center) capable of communicating with the vehicle VH.

[0021] The object recognition unit 100 recognizes surrounding objects such as stationary objects and moving objects existing around the host vehicle VH based on the detection results of the external sensor device 40. Specifically, the object recognition unit 100 recognizes the distance between the host vehicle VH and the surrounding objects existing around the host vehicle VH and the direction of the surrounding objects with respect to the host vehicle VH based on the detection results of the external sensor device 40 (for example, the sonar sensor 43). The object recognition unit 100 transmits the recognition results (distance, direction, etc.) of the surrounding objects to the alarm control unit 110 at a predetermined cycle.

[0022] The alarm control unit 110 performs alarm control to alert the user when there is an object approaching the vehicle VH in the vicinity of the vehicle VH. Here, an approaching object refers to an object that could become an obstacle when the vehicle VH is moving, an object that approaches the vehicle VH when the vehicle VH is stopped, etc. When the distance between the vehicle VH recognized by the object recognition unit 100 and the surrounding object falls below a predetermined threshold distance Dv, the alarm control unit 110 determines that the risk of collision between the vehicle 1 and the surrounding object has reached a predetermined risk level. When the alarm control unit 110 determines that the predetermined risk level has been reached, it performs alarm control to alert the user by sounding an alarm from the speaker 62 or buzzer 63. Furthermore, when the alarm control unit 110 receives a cancellation request from the user (for example, a mute operation), it cancels the alarm control, that is, it does not perform alarm control even if the risk of collision between the vehicle VH and the surrounding object has reached a predetermined risk level. A user's cancellation request can be received, for example, by touching the screen of the display device 61, turning a physical switch on or off, or by voice recognition using a sound collector.

[0023] The image display control unit 120 generates an overhead view image from a virtual viewpoint based on image data of the area around the vehicle VH captured by the camera sensor 41, and performs image display control to display the generated overhead view image as a surrounding image on the display device 61. Specifically, the image display control unit 120 projects the image data onto a predetermined projection surface (for example, a hemisphere), and takes the image projected onto the projection surface as the overhead view image, which is the area included in a predetermined field of view when viewed from an arbitrary virtual viewpoint (for example, vertically above the vehicle VH). The image display control unit 120 also generates a surrounding image by superimposing the vehicle VH at approximately the center position of the overhead view image and superimposing surrounding targets detected by the external sensor device 40 at corresponding positions in the overhead view image.

[0024] Figure 3A is a schematic diagram showing an example of a surrounding image P displayed on the display device 61 by the image display control unit 120. In Figure 3A, the symbol VH represents the vehicle VH (for example, a polygon schematically showing the shape of the vehicle VH) superimposed on the overhead view image, and the symbol OJ represents a surrounding object superimposed on the overhead view image. Note that the surrounding image P is not limited to an overhead view image of the vehicle VH viewed from vertically above, but may also be a front view image showing the front side of the vehicle VH, etc.

[0025] When the magnification control unit 130 receives a magnification request from the user, it performs magnification control to enlarge and display on the display device 61 a portion of the surrounding image P corresponding to any position selected by the user. The magnification request can be received, for example, by the user tapping on any position on the surrounding image P displayed on the display device 61. As an example, the surrounding image P shown in Figure 3A is divided into four regions (right front RF, left front LF, right rear RR, left rear LR) by dashed lines. If the user taps, for example, the right front RF, the magnification control unit 130 displays an enlarged image PE of the right front side of the vehicle VH on the display device 61, as shown in Figure 3B.

[0026] Incidentally, when alarm control is being executed, displaying an enlarged image PE on the display device 61 in response to a user request for enlarged display presents a problem in that it becomes difficult for the user to recognize the presence of an approaching object. Also, in a scenario where alarm control is executed targeting a new approaching object while the enlarged image PE is being displayed on the display device 61 in response to a user request, continuing the enlarged display presents a problem in that the presence of the approaching object cannot be properly notified to the user. Furthermore, suppose the user recognizes an approaching object due to the execution of alarm control and requests cancellation of the alarm sound (mute operation), and then also requests enlarged display. In such a scenario, for example, if an approaching object in a new direction is detected outside the field of view of the enlarged image PE, or if the approach of an already detected approaching object is detected again outside the field of view of the enlarged image PE, there is a problem in that the presence of the approaching object cannot be properly notified to the user. The enlarged display cancellation processing unit 140 performs a cancellation process that appropriately notifies the user of the presence of an approaching object by displaying it, by prohibiting or canceling the execution of the enlarged display control depending on the situation. The details of the cancellation process will be explained below.

[0027] <Scene 1> Suppose the alarm control unit 110 is performing alarm control when the magnified display control unit 130 receives a request for magnified display from the user. In such a scenario, if the magnified image PE is displayed on the display device 61, the nearby object may not be within the field of view of the magnified image PE, making it highly likely that the user will not be able to recognize the presence of the nearby object. In this case, the magnified display cancellation processing unit 140 prohibits the magnified display control unit 130 from performing magnified display control. That is, the display device 61 displays the surrounding image P, which shows the entire area around the vehicle VH. This makes it possible to appropriately notify the user of the presence of nearby objects through the display.

[0028] <Scene 2> Suppose the magnification control unit 130 receives a magnification display request from the user and displays the magnified image PE on the display device 61, at which point the alarm control unit 110 detects a new nearby object and executes alarm control. In such a scenario, if the magnified image PE is continuously displayed on the display device 61, the nearby object may not be within the field of view of the magnified image PE, making it highly likely that the user will not be able to recognize the presence of the nearby object. In this case, the magnification display cancellation processing unit 140 cancels the execution of the magnification display control by the magnification display control unit 130. That is, the display on the display device 61 is switched from the magnified image PE to the surrounding image P which displays the entire surroundings of the vehicle VH. This makes it possible to appropriately notify the user of the presence of a new nearby object through the display.

[0029] <Scene 3> Suppose the alarm control unit 110 has executed an alarm control, allowing the user to recognize a nearby object, then request to cancel the alarm sound (mute operation), and further request to enlarge the display. In such a scenario, since the user is aware of nearby objects in their surroundings, prohibiting the enlargement display would reduce the user's usability. In this case, the enlargement display cancellation processing unit 140 does not prohibit the execution of the enlargement display control unit 130. That is, it allows the execution of the enlargement display control.

[0030] <Scene 4> Suppose that after the alarm control unit 110 executes alarm control, the user recognizes a nearby object, requests to cancel the alarm sound (mute operation), and then requests an enlarged display, resulting in the enlarged image PE being displayed on the display device 61. In such a scenario, for example, if the target recognition unit 100 detects a new nearby object from a new direction that is not within the field of view of the enlarged image PE, continuing to display the enlarged image PE on the display device 61 would increase the likelihood that the user would not be able to recognize the presence of the new nearby object. In this case, the enlarged display cancellation processing unit 140 cancels the execution of the enlarged display control unit 130. That is, the display on the display device 61 is switched from the enlarged image PE to the surrounding image P which displays the entire surroundings of the vehicle VH. This makes it possible to appropriately notify the user of the presence of a new nearby object that has appeared in a new direction through the display.

[0031] <Scene 5> Suppose that after the alarm control unit 110 executes alarm control, the user recognizes an approaching object, requests to cancel the alarm sound (mute operation), and then requests an enlarged display, resulting in the enlarged image PE being displayed on the display device 61. In such a scenario, for example, if the target recognition unit 100 detects the approaching object again, which was originally detected but is not displayed within the field of view of the enlarged image PE, continuing to display the enlarged image PE on the display device 61 would increase the likelihood that the user would not be able to recognize the presence of the approaching object again. In this case, the enlarged display cancellation processing unit 140 cancels the execution of the enlarged display control unit 130. That is, the display on the display device 61 is switched from the enlarged image PE to the surrounding image P which displays the entire surroundings of the vehicle VH. This makes it possible to appropriately notify the user of the presence of the approaching object again through the display.

[0032] Next, the routine for canceling the magnification control executed by the CPU 11 of the ECU 10 will be explained based on the flowchart shown in Figure 4. This routine is started, for example, when a peripheral image P or a magnified image PE is displayed on the display device 61.

[0033] In step S100, the ECU 10 determines whether or not alarm control is being performed to sound an alarm from the speaker 62 or buzzer 63. If alarm control is being performed (Yes), the ECU 10 proceeds to step S180. On the other hand, if alarm control is not being performed (No), the ECU 10 proceeds to step S110.

[0034] In step S180, the ECU 10 prohibits or cancels the execution of the magnification control that causes the display device 61 to display the magnified image PE, and returns to this routine. Specifically, if the peripheral image P is displayed on the display device 61, even if the user requests magnification, the ECU 10 prohibits the display of the magnified image PE. That is, the peripheral image P continues to be displayed on the display device 61. Also, if the magnified image PE is displayed on the display device 61, the ECU 10 cancels the display of the magnified image PE. That is, the display on the display device 61 is switched from the magnified image PE to the peripheral image P.

[0035] In step S110, the ECU 10 determines whether the user has requested the alarm sound to be canceled (mute operation). If the user has requested the alarm sound to be canceled (Yes), the ECU 10 proceeds to the process in step S130. On the other hand, if the user has not requested the alarm sound to be canceled (No), the ECU 10 proceeds to the process in step S120.

[0036] In step S120, the ECU 10 authorizes the execution of magnification control to display the magnified image PE on the display device 61 and returns to this routine. Specifically, if the peripheral image P is displayed on the display device 61, and the user requests magnification, the ECU 10 causes the display device 61 to display the magnified image PE. Furthermore, if the magnified image PE is already displayed on the display device 61, the ECU 10 causes the display device 61 to continue displaying the magnified image PE.

[0037] In step S130, the ECU 10 determines whether it has detected a new nearby object from a new direction that is not within the field of view of the enlarged image PE. If a new nearby object is detected from the new direction (Yes), the ECU 10 proceeds to step S180. That is, it prohibits or cancels the execution of the enlarged display control and then returns to this routine. On the other hand, if no new nearby object is detected from the new direction (No), the ECU 10 proceeds to step S140. Note that the processing in steps S130 and S140 can be performed in any order.

[0038] In step S140, the ECU 10 determines whether it has detected the reappearance of a previously detected nearby object that is not displayed within the field of view of the enlarged image PE. If it has detected the reappearance of a nearby object (Yes), the ECU 10 proceeds to step S180. That is, it prohibits or cancels the execution of the enlarged display control and then returns to this routine. On the other hand, if it has not detected the reappearance of a nearby object (No), the ECU 10 proceeds to step S120. That is, it permits the execution of the enlarged display control and returns to this routine.

[0039] This disclosure is not limited to the embodiments described above, and various modifications are possible as long as they do not depart from the purpose of the present invention.

[0040] For example, the technology disclosed herein can also be applied to autonomous vehicles that perform some or all of the driving operations automatically. In the case of autonomous vehicles, the disclosure should be applied when the driving operation is switched from fully automatic to manual.

Claims

1. An image display control unit that displays surrounding images taken of the area around the vehicle by a camera mounted on the vehicle on a display device installed inside the vehicle, and superimposes detection information of objects detected by an object detection sensor mounted on the vehicle onto the surrounding image in correspondence with the position of the detected object, A driving assistance device comprising: an alarm control unit that issues an alarm sound when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, Upon receiving a request from a user for enlarged display, the system includes an enlargement display control unit that displays an enlarged image of the surrounding image on the display device, The system includes an enlargement display cancellation control unit that, even if the enlargement display control unit receives the enlargement display request, prohibits the display of the enlarged image on the display device or cancels the display of the enlarged image currently displayed on the display device if a predetermined enlargement display cancellation condition is met, The aforementioned enlarged display cancellation condition includes the case in which the alarm is issued by the alarm control unit. A driving assistance device characterized by the following features.

2. A driving support device according to claim 1, The alarm control unit is configured to cancel the alarm when it receives a cancellation request from the user. The aforementioned conditions for canceling the enlarged display include: The first specific condition is met when the alarm has been canceled, and after the magnification control unit receives the magnification request, the object detection sensor detects a new object different from the object that the canceled alarm was targeting; and the second specific condition is met when the alarm has been canceled, and after the magnification control unit receives the magnification request, the object detection sensor detects that an object outside the field of view of the magnified image is approaching the vehicle. A driving assistance device characterized by the following features.

3. A camera mounted on the vehicle captures images of the area around the vehicle, which are displayed on a display device located inside the vehicle's cabin. Furthermore, detection information of objects detected by an object detection sensor mounted on the vehicle is superimposed on the surrounding image, corresponding to the position of the detected object. A driving assistance method that provides a warning by sound when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, Upon receiving a request from the user for enlarged display, the enlarged image obtained by enlarging the surrounding image is displayed on the display device. Even if the aforementioned enlargement display request is received, if the predetermined enlargement display cancellation conditions are met, the display of the enlarged image on the display device is prohibited, or the display of the enlarged image currently displayed on the display device is canceled. The aforementioned conditions for canceling the enlarged display include cases where the aforementioned alarm has been issued. A driving assistance method characterized by the following features.

4. A camera mounted on the vehicle captures images of the area around the vehicle, which are displayed on a display device located inside the vehicle's cabin. Furthermore, detection information of objects detected by an object detection sensor mounted on the vehicle is superimposed on the surrounding image, corresponding to the position of the detected object. When the relative relationship between the object detected by the object detection sensor and the vehicle reaches a predetermined risk level, the computer of the driver assistance system that issues an audible warning is instructed to: Upon receiving a request from the user for enlarged display, the process involves displaying an enlarged image, which is an enlarged version of the surrounding image, on the display device. Even if the aforementioned enlargement display request is received, if the predetermined enlargement display cancellation conditions are met, the system will perform the following processes: prohibit the display of the enlarged image on the display device, or cancel the display of the enlarged image currently displayed on the display device. The aforementioned conditions for canceling the enlarged display include cases where the aforementioned alarm has been issued. A program characterized by the following features.

Images