Driving assistance device, driving assistance method, and program product

Abstract

The driving assist device of the present disclosure appropriately notifies a user of the presence of an approaching object. A driving assist device including: an image display control section that causes a display device to display a surrounding image obtained by a vehicle-mounted camera capturing a surrounding of a vehicle, and causes detection information of an object detected by an object detection sensor to be superimposed and displayed on the surrounding image; and an alarm control section that implements an alarm based on an alarm sound in a case where a relative relationship of the object detected by the object detection sensor to the vehicle reaches a prescribed risk level, characterized by including: an enlarged display control section that causes the display device to display an enlarged image when an enlarged display request from a user is received; and an enlarged display cancel control section that prohibits or cancels the display of the enlarged image in a case where a prescribed enlarged display cancel condition is satisfied even when the enlarged display control section receives the enlarged display request, the prescribed enlarged display cancel condition including a case where the alarm is implemented.

Description

Technical Field

[0001] This disclosure relates to driving assistance devices, driving assistance methods, and program products. Background Technology

[0002] For example, Patent Document 1 discloses the following technology: In a driving assistance device that overlays the detection range of an onboard object detection sensor onto an image captured by an onboard camera, when the object detection sensor detects an obstacle, the display of the detection range is stopped and the obstacle detection display is displayed, thereby making it easy for the occupant to identify the presence of an obstacle within the detection range.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-186132 Summary of the Invention

[0006] There exists a system, as disclosed in Patent Document 1, that overlays obstacle detection and display onto a captured image, and has the function of magnifying and displaying a portion of the captured image upon request from a user (primarily a driver). In such a system, there is a problem: for example, when an approaching object is detected in a direction no longer displayed during magnification, the user cannot be adequately notified by displaying the presence of that object.

[0007] This disclosure was made to solve the aforementioned problems, and its purpose is to appropriately notify the user by indicating the presence of an approaching object.

[0008] The technology disclosed herein is a driving assistance device, comprising:

[0009] An image display control unit causes a display device installed inside the vehicle's passenger compartment to display a surrounding image obtained by a camera mounted on the vehicle capturing images of the vehicle's surroundings, and causes detection information of objects detected by an object detection sensor mounted on the vehicle to be superimposed onto the surrounding image, correspondingly displaying the position of the detected object; and

[0010] The alarm control unit, when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, implements an alarm based on an alarm tone. The driving assistance device is characterized by comprising:

[0011] The magnification display control unit, upon receiving a magnification display request from a user, causes the display device to display a magnified image obtained by magnifying the surrounding image; and

[0012] Even if the magnification display cancellation control unit accepts the magnification display request, it will still prohibit the display of the magnified image on the display device or de-display the magnified image on the display device if the predetermined magnification display cancellation conditions are met.

[0013] The conditions for canceling the magnified display include situations where the alarm is being issued by the alarm control unit. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating the hardware structure of the vehicle according to this embodiment.

[0015] Figure 2 This is a schematic diagram illustrating the software structure of the control device in this embodiment.

[0016] Figure 3 A is a schematic diagram showing an example of the peripheral image of this embodiment. Figure 3 B is a schematic diagram showing an example of an enlarged image of this embodiment.

[0017] Figure 4 This is a flowchart illustrating the routine for canceling the magnification display control in this embodiment. Detailed Implementation

[0018] Hereinafter, the driving assistance device, driving assistance method, and program product of this embodiment will be described with reference to the accompanying drawings.

[0019] [Hardware Structure]

[0020] Figure 1 This is a schematic diagram illustrating the hardware structure of the vehicle VH according to this embodiment. Hereinafter, when it is necessary to distinguish it from other vehicles, the vehicle VH will sometimes be referred to as "this vehicle".

[0021] The vehicle (VH) has an ECU (Electronic Control Unit) 10. The ECU 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, and an interface device 14. The CPU 11 is a processor that executes various program products stored in the ROM 12. The ROM 12 is non-volatile memory that stores data required by the CPU 11 to execute various program products. The RAM 13 is volatile memory that provides the operating area when the CPU 11 executes various program products. The interface device 14 is a communication device for communicating with external devices.

[0022] ECU10 is the central control device for various driving assistance functions. Driving assistance includes the concept of autonomous driving. ECU10 is communicatively connected to drive unit 20, steering unit 21, braking unit 22, internal sensor unit 30, external sensor unit 40, HMI (Human Machine Interface) 60, etc.

[0023] The drive unit 20 generates a driving force that is transmitted to the drive wheels of the vehicle VH. Examples of drive units 20 include electric motors and engines. The steering unit 21 applies steering force to the wheels of the vehicle VH. The braking unit 22 applies braking force to the wheels of the vehicle VH.

[0024] The internal sensor unit 30 is a type of sensor that detects the vehicle's VH (vehicle speed, throttle position, and humidity) state. Specifically, the internal sensor unit 30 includes a vehicle speed sensor 31, a throttle sensor 32, a brake sensor 33, and a steering angle sensor 34. The vehicle speed sensor 31 detects the vehicle's VH speed, i.e., vehicle speed. The throttle sensor 32 detects the amount of input from the driver to the accelerator pedal (not shown). The brake sensor 33 detects the amount of input from the driver to the brake pedal (not shown). The steering angle sensor 34 detects the rotation angle of the steering wheel or steering shaft (not shown), i.e., the steering angle. The internal sensor unit 30 sends the vehicle VH state detected by each sensor 31-34 to the ECU 10 at a predetermined period.

[0025] The external sensor device 40 is a sensor that identifies target information related to targets around the vehicle (VH). Specifically, the external sensor device 40 includes a radar sensor 41, a sonar sensor 42, and a camera sensor 43, etc.

[0026] Radar sensor 41 includes a front radar sensor for identifying the front of the vehicle VH, a rear radar sensor for identifying the rear of the vehicle VH, a left-side radar sensor for identifying the left side of the vehicle VH, and a right-side radar sensor for identifying the right side of the vehicle VH. Radar sensor 41 includes millimeter-wave radar and / or lidar. Millimeter-wave radar emits millimeter-wave radio waves and receives reflected waves from targets within its emission range, thereby obtaining the relative distance and relative speed between the vehicle VH and the target. Lidar sequentially scans pulsed laser light with wavelengths shorter than millimeter waves in multiple directions and receives reflected light from targets, thereby obtaining the shape of the target, the relative distance and relative speed between the vehicle VH and the target.

[0027] The sonar sensor 42 includes a front sonar sensor for identifying the front of the vehicle VH, a rear sonar sensor for identifying the rear of the vehicle VH, a left-side sonar sensor for identifying the left side of the vehicle VH, and a right-side sonar sensor for identifying the right side of the vehicle VH. The sonar sensor 42 emits ultrasonic waves and receives reflected waves from targets within its emission range, thereby obtaining the relative distance and relative speed between the vehicle VH and the target.

[0028] Camera sensor 43 includes a front camera sensor for capturing images of the front of the vehicle VH, a rear camera sensor for capturing images of the rear of the vehicle VH, a left-side camera sensor for capturing images of the left side of the vehicle VH, and a right-side camera sensor for capturing images of the right side of the vehicle VH. Camera sensor 43 can be, for example, a stereo camera or a single-lens reflex camera, and can utilize a digital camera with imaging elements such as CMOS or CCD. Camera sensor 43 processes the captured image data to obtain the shape of the target object, the relative distance between the vehicle VH and the target object, and the relative speed, etc.

[0029] The external sensor device 40 sends the acquired target information to the ECU 10 at a predetermined period. In addition, the external sensor device 40 does not necessarily need to have all of the radar sensor 41, sonar sensor 42 and camera sensor 43, as long as it has at least the sonar sensor 42 and camera sensor 43.

[0030] HMI60 is an interface for inputting and outputting information between ECU10 and the driver, and includes input and output devices. Input devices may include a touch panel-type LCD display, a switch, and a voice microphone. Output devices may include a display device 61 and a speaker 62. The display device 61 may be a central display or a multi-function information display. The speaker 62 may be a speaker from an audio system or navigation system.

[0031] [Software Structure]

[0032] Figure 2 This is a schematic diagram illustrating the software structure of the ECU10 in this embodiment. For example... Figure 2 As shown, the ECU 10 includes a target recognition unit 100, an alarm control unit 110, an image display control unit 120, a magnified display control unit 130, and a magnified display cancellation processing unit 140 as functional elements. These functional elements 100-140 are implemented by the CPU 11 of the ECU 10 reading the program products stored in the ROM 12 into the RAM 13 and executing them. Furthermore, all or part of each functional element 100-140 can also be installed in other ECUs separate from the ECU 10, or in an information processing device of a facility capable of communicating with the vehicle's VH (such as a management center).

[0033] The target object recognition unit 100 identifies surrounding targets, such as stationary and moving objects, existing around the vehicle VH, based on the detection results of the external sensor device 40. Specifically, the target object recognition unit 100 identifies the distance between the vehicle VH and surrounding targets existing around the vehicle VH, as well as the direction of the surrounding targets relative to the vehicle VH, based on the detection results of the external sensor device 40 (e.g., sonar sensor 43). The target object recognition unit 100 sends the identification results (distance, direction, etc.) of the surrounding targets to the alarm controller 110 at a predetermined period.

[0034] When an object approaching the vehicle VH is present in the vicinity of the vehicle, the alarm control unit 110 performs alarm control to alert the user. Here, an object approaching refers to an object that may become an obstacle while the vehicle VH is moving, or an object that approaches the vehicle VH when the vehicle VH is stationary. When the distance between the vehicle VH and a surrounding object, as identified by the object recognition unit 100, is less than or equal to a predetermined threshold distance Dv, the alarm controller 110 determines that the collision risk between the vehicle VH and the surrounding object has reached a predetermined risk level. When the predetermined risk level is reached, the alarm control unit 110 performs alarm control to alert the user by sounding an alarm tone through the speaker 62 and the buzzer 63. Furthermore, when a cancellation request from the user (e.g., a mute operation) is received, the alarm control unit 110 deactivates the alarm control; that is, even when the collision risk between the vehicle VH and a surrounding object reaches the predetermined risk level, alarm control is not performed. The user's cancellation request can be received, for example, through touch operation of the display device 61 screen, on / off operation of a physical switch, or voice recognition based on a microphone.

[0035] The image display control unit 120 generates an overhead image viewed from a virtual viewpoint based on image data of the area surrounding the vehicle VH captured by the camera sensor 43, and performs image display control to display the generated overhead image as a peripheral image on the display device 61. Specifically, the image display control unit 120 projects image data onto a predetermined projection surface (e.g., a hemisphere), and sets the area within a predetermined field of view when viewed from any virtual viewpoint (e.g., above the vertical direction of the vehicle VH) in the image projected onto the projection surface as the overhead image. In addition, the image display control unit 120 generates a peripheral image by overlaying the vehicle VH at approximately the center position of the overhead image and overlaying peripheral objects detected by the external sensor device 40 at corresponding positions of the overhead image.

[0036] Figure 3 A is a schematic diagram illustrating an example of a peripheral image P displayed on the display device 61 via the image display control unit 120. Figure 3In Figure A, reference numeral VH indicates the vehicle VH superimposed on the top-view image (e.g., a polygon schematically representing the shape of the vehicle VH), and reference numeral OJ indicates a surrounding object superimposed on the top-view image. Furthermore, the surrounding image P is not limited to a top-view image obtained by observing the vehicle VH from a vertical direction; it can also be a front view image showing the front side of the vehicle VH, etc.

[0037] When a user requests magnification, the magnification control unit 130 executes magnification control, which magnifies the portion of the peripheral image P corresponding to any position selected by the user and displays it on the display device 61. The magnification request can be processed, for example, by the user clicking on any position on the peripheral image P displayed on the display device 61. As an example, in... Figure 3 In the surrounding image P shown in Figure A, four regions are divided by dashed lines (Right Front RF, Left Front LF, Right Rear RR, and Left Rear LR). For example, if the user clicks on the Right Front RF, such as... Figure 3 As shown in Figure B, the magnified display control unit 130 displays the magnified image PE obtained by magnifying the right front side of the vehicle VH on the display device 61.

[0038] However, during alarm control execution, if a magnified image PE is displayed on the display device 61 according to a user's request for magnified display, it becomes difficult for the user to identify the presence of an approaching object. Furthermore, if the magnified image PE is continuously displayed on the display device 61 during alarm control of a new approaching object, it becomes impossible to properly notify the user of the object's presence. Additionally, suppose the user identifies an approaching object due to alarm control execution and requests to mute the alarm (silence operation), and then requests further magnification. In such scenarios, for example, if an approaching object is detected in a new direction outside the magnified image PE's field of view, or if a previously detected approaching object is detected approaching again outside the magnified image PE's field of view, it becomes impossible to properly notify the user of the object's presence. The magnified display cancellation processing unit 140 performs cancellation processing, which disables or deactivates the magnified display control depending on the situation, thereby appropriately notifying the user of the presence of the approaching object through display. The details of the cancellation processing will be explained below.

[0039] <Scene 1>

[0040] Suppose that during alarm control by alarm control unit 110, magnification display control unit 130 receives a magnification display request from the user. In this scenario, if a magnified image PE is displayed on display device 61, the possibility that the approaching object is not within the field of view of magnified image PE increases. In this case, magnification display cancellation processing unit 140 prohibits the execution of magnification display control by magnification display control unit 130. That is, a peripheral image P displaying the entire surroundings of the vehicle VH is displayed on display device 61. Thus, the presence of the approaching object can be appropriately notified to the user through the display.

[0041] Scene 2

[0042] Suppose that while the magnified display control unit 130 receives a magnified display request from a user and displays a magnified image PE on the display device 61, the alarm control unit 110 executes alarm control due to the detection of a new approaching object. In this scenario, if the magnified image PE continues to be displayed on the display device 61, the possibility that the user cannot recognize the presence of the approaching object increases if it does not enter the field of view of the magnified image PE. In this case, the magnified display cancellation processing unit 140 cancels the execution of the magnified display control performed by the magnified display control unit 130. That is, the display on the display device 61 switches from the magnified image PE to displaying a peripheral image P of the entire surrounding area of ​​the vehicle VH. Thus, the presence of a new approaching object can be appropriately notified to the user through the display.

[0043] Scene 3

[0044] Suppose that after a user identifies an approaching object by performing alarm control through alarm control unit 110, the user requests to cancel the alarm sound (mute operation) and further requests to zoom in on the display. In this scenario, since the user has identified an approaching object, disabling the zoom in display would reduce user usability. In this case, the zoom in display cancellation processing unit 140 does not disable the zoom in display control unit 130 from performing zoom in display control. That is, zoom in display control is allowed.

[0045] Scene 4

[0046] Suppose that after a user identifies an approaching object by performing alarm control through alarm control unit 110, the user requests to cancel the alarm sound (mute operation) and further requests a magnified display, thereby displaying a magnified image PE on display device 61. In this scenario, for example, if the target object recognition unit 100 detects a new approaching object in a new direction that has never entered the view of magnified image PE, and the magnified image PE continues to be displayed on display device 61, the possibility that the user cannot recognize the presence of the new approaching object increases. In this case, magnified display cancellation processing unit 140 cancels the execution of magnified display control performed by magnified display control unit 130. That is, the display on display device 61 switches from magnified image PE to displaying a peripheral image P of the entire surrounding area of ​​the vehicle VH. Thus, the presence of a new approaching object appearing in a new direction can be appropriately notified to the user through the display.

[0047] Scene 5

[0048] Suppose that after a user identifies an approaching object by executing alarm control through alarm control unit 110, the user requests to cancel the alarm sound (mute operation) and further requests a magnified display, thereby displaying a magnified image PE on display device 61. In such a scenario, for example, if the target object recognition unit 100 detects the reappearance of a previously detected approaching object that is not displayed within the view of magnified image PE, the possibility that the user will not be able to recognize the presence of the reappearing approaching object increases if magnified image PE is continuously displayed on display device 61. In this case, magnified display cancellation processing unit 140 cancels the execution of magnified display control performed by magnified display control unit 130. That is, the display on display device 61 switches from magnified image PE to displaying a peripheral image P of the entire surrounding area of ​​the vehicle VH. Thus, the presence of a reappearing approaching object can be appropriately notified to the user through the display.

[0049] Next, based on Figure 4 The flowchart shown illustrates a routine for canceling the magnified display control executed by the CPU 11 of ECU 10. This routine begins, for example, when a peripheral image P or a magnified image PE is displayed on the display device 61.

[0050] In step S100, ECU10 determines whether to execute alarm control that causes speaker 62 and buzzer 63 to sound an alarm tone. If alarm control is being executed (yes), ECU10 proceeds to step S180. On the other hand, if alarm control is not executed (no), ECU10 proceeds to step S110.

[0051] In step S180, ECU 10 disables or deactivates the magnification display control that causes display device 61 to display the magnified image PE, and returns to the present routine. Specifically, when the peripheral image P is displayed on display device 61, ECU 10 disables the display of the magnified image PE even if the user requests magnification. That is, the peripheral image P is continuously displayed on display device 61. Furthermore, when display device 61 displays the magnified image PE, ECU 10 deactivates the display of the magnified image PE. That is, the display on display device 61 switches from the magnified image PE to the peripheral image P.

[0052] In step S110, ECU10 determines whether the user has requested to cancel the alarm sound (mute operation). If the user has requested to cancel the alarm sound (yes), ECU10 proceeds to step S130. On the other hand, if the user has not requested to cancel the alarm sound (no), ECU10 proceeds to step S120.

[0053] In step S120, ECU10 allows the execution of magnified display control to cause display device 61 to display magnified image PE, and returns to the current routine. Specifically, if the user requests magnification while display device 61 is displaying peripheral image P, ECU10 causes display device 61 to display magnified image PE. Furthermore, when display device 61 is displaying magnified image PE, ECU10 continuously displays magnified image PE on display device 61.

[0054] In step S130, ECU10 determines whether a new approaching object is detected from a new direction that has never entered the view of the magnified image PE. If a new approaching object is detected from the new direction (yes), ECU10 proceeds to step S180. That is, after disabling or deactivating the magnification display control, it returns to the current routine. On the other hand, if no new approaching object is detected from the new direction (no), ECU10 proceeds to step S140. Furthermore, the processing order of steps S130 and S140 is different.

[0055] In step S140, ECU10 determines whether a previously detected approaching object that is not displayed within the view of the magnified image PE is detected approaching again. If a reappearance of the approaching object is detected (yes), ECU10 proceeds to step S180. That is, after disabling or deactivating the magnification display control, it returns to the current routine. On the other hand, if no reappearance of the approaching object is detected (no), ECU10 proceeds to step S120. That is, it allows the magnification display control to be executed and returns to the current routine.

[0056] Furthermore, this disclosure is not limited to the above-described embodiments; various modifications can be made without departing from the purpose of the invention.

[0057] For example, the technology disclosed herein can also be applied to fully or partially autonomous vehicles that perform driving operations automatically. In the case of autonomous vehicles, this disclosure can be applied when the driving operation switches from fully automatic to manual.

Claims

1. A driving assistance device, comprising: An image display control unit causes a display device installed inside the vehicle's passenger compartment to display a surrounding image obtained by a camera mounted on the vehicle capturing images of the vehicle's surroundings, and causes detection information of objects detected by an object detection sensor mounted on the vehicle to be superimposed onto the surrounding image, correspondingly displaying the position of the detected object; and The alarm control unit, when the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, implements an alarm based on an alarm tone. The driving assistance device is characterized by having: The magnification display control unit, upon receiving a magnification display request from a user, causes the display device to display a magnified image obtained by magnifying the surrounding image; and Even if the magnification display cancellation control unit accepts the magnification display request, it will still prohibit the display of the magnified image on the display device or de-display the magnified image on the display device if the predetermined magnification display cancellation conditions are met. The conditions for canceling the magnified display include situations where the alarm is being issued by the alarm control unit.

2. The driving assistance device according to claim 1, characterized in that, The alarm control unit is configured to cancel the execution of the alarm when it receives a cancellation request from the user. The conditions for canceling the magnified display include at least one of a first specific condition and a second specific condition. The first specific condition is that, if the alarm has been canceled, after the magnified display control unit accepts the magnified display request, the object detection sensor detects a new object that is different from the object that was the object of the canceled alarm. The second specific condition is that, if the alarm has been canceled, after the magnified display control unit accepts the magnified display request, the object detection sensor detects an object outside the field of view of the magnified image approaching the vehicle.

3. A driving assistance method, The display device installed inside the vehicle's passenger compartment displays a surrounding image obtained by a camera mounted on the vehicle capturing images of the vehicle's surroundings. Furthermore, the detection information of objects detected by an object detection sensor mounted on the vehicle is correspondingly superimposed onto the surrounding image to display the position of the detected object. If the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, an alarm based on an alarm tone will be activated. The driving assistance method is characterized in that... When a user requests a magnified display, the display device displays a magnified image obtained by enlarging the surrounding image. Even if the request for magnified display is accepted, if the specified conditions for canceling magnified display are met, the display of the magnified image on the display device shall be prohibited, or the display of the magnified image on the display device shall be deactivated. The conditions for canceling the magnified display include situations where the alarm is in effect.

4. A program product, characterized in that, The computer of the driver assistance device performs the processing. The driving assistance device displays a surrounding image of the vehicle's perimeter, captured by a camera mounted on the vehicle, on a display device located inside the vehicle's cabin. It also overlays the detection information of objects detected by an object detection sensor mounted on the vehicle onto the surrounding image, showing the positions of the detected objects. If the relative relationship between an object detected by the object detection sensor and the vehicle reaches a predetermined risk level, an alarm based on an alarm sound is activated. The process is as follows: When a user requests magnified display, the display device displays a magnified image obtained by magnifying the surrounding image; and Even if the request for magnified display is accepted, the display of the magnified image on the display device shall be prohibited or the display of the magnified image on the display device shall be deactivated if the specified conditions for canceling magnified display are met. The conditions for canceling the magnified display include situations where the alarm is in effect.

Images