Driving assistance system

Abstract

A driving assistance system is provided with: a vehicle-mounted outside recognition device that recognizes traffic participants around a vehicle as surrounding traffic participants; an alert object recognition unit that recognizes, as an alert object, a traffic participant, among the traffic participants recognized as surrounding traffic participants, which has a large degree of impact on safe driving; a display object setting unit that sets the preceding vehicle 2a recognized as the alert object as a display object; a display control unit that performs image enhancement processing on a windshield (51) provided in a front display unit (50) at a position visible to the driver of the host vehicle, the image enhancement processing being to emphasize the presence of a preceding vehicle (2a) set as a display target; and the driver identification and detection device is used for detecting whether the driver identifies the surrounding traffic participants or not. The display object setting unit cancels the setting of the display object of the preceding vehicle 2a when the recognition of the driver on the preceding vehicle 2a set as the display object is detected, and resets the preceding vehicle 2a as the display object when the degree of influence on the preceding vehicle 2a is increased thereafter.

Description

Technical Field

[0001] The present invention relates to a driving assistance system. More specifically, the system identifies surrounding traffic participants that have a significant impact on the safe driving of a vehicle as warning targets, and prompts the driver of the vehicle to identify the warning targets. Background Art

[0002] In recent years, efforts have been actively underway to provide access to sustainable transportation systems that also take into account vulnerable road users. To achieve this goal, efforts are underway to further improve traffic safety and convenience through the research and development of driver assistance technologies that assist drivers in understanding their surroundings.

[0003] In the driving assistance technology shown in Patent Document 1, when a dangerous object that poses a risk of collision with the vehicle is detected based on an image of the vehicle's surroundings, a first warning image emphasizing the dangerous object is displayed on a display unit to encourage the driver of the vehicle to identify the dangerous object.

[0004] [Prior Art Literature]

[0005] (Patent Document)

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2010-191793 Summary of the Invention

[0007] [Problems to be solved by the invention]

[0008] Furthermore, in the driving assistance technology disclosed in Patent Document 1, after displaying a first warning image, if the driver confirms the dangerous object, the first warning image is discontinued and a second warning image, with less emphasis than the first, is displayed. In this way, conventional driving assistance technology discontinues the warning image upon driver confirmation, thereby preventing the continued display of the warning image from causing annoyance to the driver. However, there are cases where, after the driver confirms the dangerous object, its behavior changes, further increasing the risk of collision with the vehicle. However, Patent Document 1 does not contemplate this scenario.

[0009] An object of the present invention is to provide a driving assistance system that can help a driver recognize the presence of a warning object that may affect the safe driving of the vehicle without causing annoyance, thereby promoting the development of a sustainable transportation system.

[0010] [Technical means to solve the problem]

[0011] (1) The driving assistance system of the present invention comprises: an external recognition device for recognizing traffic participants around the vehicle as surrounding traffic participants; a warning object recognition unit for recognizing traffic participants that have a greater degree of influence on the safe driving of the vehicle as warning objects among the traffic participants recognized as the surrounding traffic participants; a display object setting unit for setting the traffic participant recognized as the warning object as a display object; a display control unit for performing an emphasis image processing on a display unit provided at a position where the driver of the vehicle can visually recognize the traffic participant, wherein the emphasis image processing emphasizes the presence of the traffic participant set as the display object; and a driver recognition detection unit for detecting whether the driver recognizes the surrounding traffic participants. The display object setting unit is characterized in that, when it is detected that the driver recognizes a specific traffic participant set as the display object, the display object setting for the specific traffic participant is canceled, and when the degree of influence on the specific traffic participant increases, the specific traffic participant is set as the display object again.

[0012] (2) At this time, it is preferred that, after the setting of the display object for the specific traffic participant is released, if the degree of influence on the specific traffic participant increases from the time point when the setting of the display object is released or the time point when the identification of the driver is detected, the display object setting unit sets the specific traffic participant as the display object again.

[0013] (3) At this time, it is preferred that the aforementioned external recognition device obtains movement state information related to the movement state of the aforementioned surrounding traffic participants, and the aforementioned warning object recognition unit or the aforementioned display object setting unit calculates the aforementioned influence degree based on the aforementioned movement state information.

[0014] (4) In this case, it is preferred that the driver recognition detection device has a camera device for photographing the driver's face, and detects whether the driver recognizes the surrounding traffic participants based on image data obtained by the camera device.

[0015] (5) At this time, it is preferable that the warning target recognition unit or the display target setting unit detects or predicts the behavior of the surrounding traffic participants based on the movement status information, and calculates the influence degree based on the results of these detections or predictions.

[0016] (6) At this time, it is preferred that the aforementioned external recognition device obtains traffic environment information related to the traffic environment around the aforementioned vehicle, and the aforementioned warning object recognition unit or the aforementioned display object setting unit predicts the behavior of the aforementioned surrounding traffic participants based on the aforementioned traffic environment information.

[0017] (7) At this time, it is preferred that the aforementioned driving assistance system also has a driver state acquisition device, which acquires driver state information related to the driving ability of the aforementioned driver, and the aforementioned warning object recognition unit or the aforementioned display object setting unit calculates the aforementioned impact level based on the aforementioned movement state information and the aforementioned driver state information.

[0018] (8) At this time, it is preferred that after the aforementioned specific traffic participant is set as the aforementioned display object, the aforementioned display object setting unit cancels the setting of the aforementioned display object for the aforementioned specific traffic participant after a prescribed emphasis display period or when the aforementioned driver's recognition of the aforementioned display object is detected, and the aforementioned emphasis display period is set based on the aforementioned degree of influence on the aforementioned specific traffic participant.

[0019] (9) At this time, it is preferred that, when multiple traffic participants are identified as the aforementioned warning objects, the aforementioned display object setting unit will set the traffic participants extracted in sequence from the one with the greater influence as the aforementioned display objects within a range not exceeding a prescribed set upper limit.

[0020] (10) At this time, it is preferred that the aforementioned driving assistance system further includes a driver state acquisition device, which acquires driver state information related to the driving ability of the aforementioned driver, and the aforementioned warning object identification unit identifies the traffic participants whose aforementioned influence degree is greater than a prescribed influence degree threshold among the traffic participants identified as the aforementioned surrounding traffic participants as the aforementioned warning objects, and the aforementioned influence degree threshold is set based on the aforementioned driver state information.

[0021] (11) At this time, it is preferred that the aforementioned display unit includes a front display unit, and the front display unit projects the image in front of the aforementioned vehicle. In the aforementioned emphasized image processing, the aforementioned display control unit uses the part of the aforementioned front display unit that projects the image of the aforementioned display object as a front display area, and overlaps and displays a highlight image in the front display area, thereby emphasizing the existence of the aforementioned display object.

[0022] (12) At this time, it is preferred that the aforementioned display unit includes a rear display unit, and the rear display unit projects an image of the rear of the aforementioned vehicle. In the aforementioned emphasized image processing, the aforementioned display control unit uses the portion of the aforementioned rear display unit that projects the image of the aforementioned display object as a rear display area, and displays a highlight image in an overlapping manner in the rear display area, thereby emphasizing the existence of the aforementioned display object.

[0023] (Effects of the Invention)

[0024] (1) In the present invention, the external recognition device identifies traffic participants around the vehicle as surrounding traffic participants, the alert object recognition unit identifies traffic participants that have a greater impact on the safe driving of the vehicle among the traffic participants identified as surrounding traffic participants as alert objects, the display object setting unit sets the traffic participant identified as the alert object as a display object, and the display control unit performs emphasis image processing on a display unit that can be visually recognized by the driver, wherein the emphasis image processing emphasizes the presence of the traffic participant set as the display object. Therefore, according to the present invention, when there is a warning object that has a greater impact on the safe driving of the vehicle around the vehicle and the warning object is set as a display object, the presence of the warning object (display object) in the display unit is emphasized by performing emphasis image processing, so that the driver can easily recognize the presence of the warning object. In addition, in the present invention, when the driver detects that the driver has recognized a specific traffic participant set as a display object, the display object setting unit cancels the display object setting for the specific traffic participant. Accordingly, the display control unit ends the emphasis image processing for the specific traffic participant whose recognition by the driver was detected. Thus, according to the present invention, it is possible to prevent a situation in which the driver is annoyed by the emphasized image processing being performed even after the driver has made an identification. In addition, in the present invention, as described above, after the setting of the display object for a specific traffic participant is released, in a case where the degree of influence for the specific traffic participant increases, the display object setting unit sets the specific traffic participant as the display object again. Accordingly, the display control unit starts the emphasized image processing for the specific traffic participant whose degree of influence has increased again. Thus, according to the present invention, even after the driver has made an identification of a specific traffic participant, emphasized image processing is still performed for the same specific traffic participant according to subsequent changes in the situation. Thus, it is possible to avoid the driver being annoyed by the presence of a specific traffic participant that may affect the safe driving of the vehicle, thereby assisting the driver in driving safely. In addition, it is possible to promote the development of a sustainable transportation system.

[0025] (2) In the present invention, as described above, after the display object setting for a specific traffic participant is released based on the detection of the driver's recognition, if the degree of influence for the specific traffic participant increases from the time the display object setting is released or the time the driver's recognition is detected, the display object setting unit sets the specific traffic participant as a display object again and restarts the emphasized image processing for the specific traffic participant. In this way, according to the present invention, by using the time when the display object setting is released or the time when the driver's recognition is detected as a reference to determine whether the degree of influence has increased, it is possible to reasonably determine whether the driver should be prompted to recognize the specific traffic participant again, thereby assisting the driver in driving safely.

[0026] (3) In the present invention, the alert target identification unit or the display target setting unit calculates the degree of influence based on the movement state information related to the movement state of surrounding traffic participants. Thus, according to the present invention, the degree of influence on the safe driving of the host vehicle can be calculated with high accuracy, taking into account the movement state of surrounding traffic participants. Therefore, surrounding traffic participants that the driver should reliably identify can be set as display targets.

[0027] (4) In the present invention, the driver recognition detection device detects whether the driver recognizes surrounding traffic participants based on the image data of the driver's face, thereby detecting whether the driver recognizes surrounding traffic participants by visual observation.

[0028] (5) In the present invention, the alert target identification unit or display target setting unit detects or predicts the behavior of surrounding traffic participants based on their movement status information, and calculates the degree of influence for each surrounding traffic participant based on the results of these detections or predictions. Thus, according to the present invention, the degree of influence can be calculated with high accuracy, reflecting the results of the detection or prediction of the behavior of surrounding traffic participants. Therefore, surrounding traffic participants that the driver should reliably identify can be set as display targets.

[0029] (6) In the present invention, the alert target identification unit or display target setting unit predicts the behavior of surrounding traffic participants based on traffic environment information related to the traffic environment surrounding the vehicle. This allows the behavior of surrounding traffic participants to be predicted with high accuracy, and the degree of their influence to be calculated with high accuracy. Thus, according to the present invention, surrounding traffic participants that the driver should reliably identify can be set as display targets.

[0030] (7) In the present invention, the alert object recognition unit or display object setting unit calculates the degree of influence based on the movement state information and the driver state information. This allows the degree of influence to be calculated with high accuracy, reflecting the current driving ability of the driver of the vehicle. Thus, according to the present invention, surrounding traffic participants that the driver should reliably identify can be set as display objects.

[0031] (8) In the present invention, after a specific traffic participant identified as a warning target is set as a display object, the display object setting unit cancels the setting of the display object for the specific traffic participant after a prescribed emphasized display period has elapsed or when the driver's recognition of the display object is detected, and ends the emphasized image processing. Thus, according to the present invention, it is possible to prevent the driver from feeling annoyed by the emphasized image processing being performed for a long time. In addition, in the present invention, by setting the emphasized display period based on the degree of influence on the specific traffic participant, the emphasized display period can be set to a reasonable length according to the degree of influence of the specific traffic participant.

[0032] (9) In the present invention, when multiple traffic participants are identified as alert targets, the display target setting unit sets as display targets traffic participants sequentially extracted, starting with the one with the greatest impact, within a range not exceeding a predetermined upper limit. Thus, according to the present invention, even when multiple traffic participants are identified as alert targets, the image processing for traffic participants exceeding the set upper limit is not simultaneously emphasized, thereby preventing the driver from becoming annoyed or confused.

[0033] (10) In the present invention, the alert object recognition unit identifies, among the traffic participants identified as surrounding traffic participants, traffic participants whose influence level is greater than a threshold level of influence as alert objects, and sets the threshold level of influence based on driver status information. Thus, the frequency of performing the emphasized image processing can be changed according to the driver's current state, thereby assisting the driver in driving safely.

[0034] (11) In the present invention, the display control unit, during the emphasized image processing, uses the portion of the front display unit where the image of the display object is projected as the front display area, and superimposes the highlighted image on the front display area to emphasize the presence of the display object. This allows the driver to easily recognize the presence of the display object in front of the vehicle.

[0035] (12) In the present invention, the display control unit, during the image emphasis processing, uses the portion of the rear display unit where the image of the display object is projected as the rear display area, and superimposes a highlight image on the rear display area to emphasize the presence of the display object. This allows the driver to easily recognize the presence of the display object behind the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram schematically showing the structure of a driving assistance system and a vehicle equipped with the driving assistance system according to one embodiment of the present invention.

[0037] Figure 2This is a diagram illustrating an example of image enhancement processing for the front display unit.

[0038] Figure 3 This diagram illustrates an example of image enhancement processing for an electronic room mirror on a rear display unit.

[0039] Figure 4 This is a flowchart (part 1) illustrating a specific flow of display / non-display setting processing in a display object setting unit.

[0040] Figure 5 This is a flowchart (part 2) illustrating the specific flow of the display / non-display setting process in the display object setting unit.

[0041] Figure 6 This is a flowchart (part 3) illustrating the specific flow of the display / non-display setting process in the display object setting unit.

[0042] Figure 7 This is a flowchart (part 4) illustrating the specific flow of the display / non-display setting process in the display object setting unit.

[0043] Figure 8 This is a diagram showing an example of an alert target list stored in the storage area of the display target setting unit.

[0044] Figure 9 This is a flowchart showing the specific process of updating the canceling flag. DETAILED DESCRIPTION

[0045] Hereinafter, a driving assistance system according to an embodiment of the present invention will be described with reference to the drawings.

[0046] Figure 1 This figure schematically illustrates the configuration of a driving assistance system 1 according to this embodiment and a vehicle V (hereinafter referred to simply as "vehicle V") equipped with the driving assistance system 1. The following description assumes that vehicle V is a so-called right-hand-drive four-wheeled vehicle, meaning that the driver's seat is located on the right side in the vehicle width direction as viewed in the direction of travel. However, the present invention is not limited to this configuration. The vehicle may also be a so-called left-hand-drive four-wheeled vehicle, meaning that the driver's seat is located on the left side in the vehicle width direction as viewed in the direction of travel.

[0047] The driving assistance system 1 identifies traffic participants around the vehicle V as surrounding traffic participants, and identifies one or more traffic participants among these identified surrounding traffic participants who are judged to have a greater impact on the safe driving of the vehicle V as warning objects, and then performs emphasized image processing on a display unit set at a position where the driver of the vehicle V can visually recognize the existence of these one or more warning objects, thereby assisting the driver in recognizing the existence of the warning objects. In addition, the following is a description of the installation of the driving assistance system 1 on an image Figure 1 While this description will focus on a four-wheeled vehicle, i.e., vehicle V, as shown in FIG. , the present invention is not limited thereto. The present invention also encompasses a driving assistance system mounted on vehicles other than four-wheeled vehicles, such as trucks or straddle-type vehicles. Furthermore, when mounted on a vehicle, particularly a straddle-type vehicle, at least a portion of the driving assistance system may be mounted on a helmet worn by the rider. Furthermore, in this embodiment, traffic participants refer to individuals, such as vehicles or pedestrians, who move voluntarily.

[0048] The driving assistance system 1 includes: an on-board external world recognition device 2 for recognizing traffic participants (hereinafter also referred to as "surrounding traffic participants") existing around the vehicle V (including in front of, behind, and on both sides of the vehicle V) and the traffic environment around the vehicle V; a driver state acquisition device 3 for acquiring driver state information related to the driver's driving ability; a driver recognition detection device 4 for detecting whether the driver is aware of the surrounding traffic participants; a front display unit 50 for projecting an image in front of the vehicle V; a rear display unit 53 for projecting an image behind the vehicle V; and an image processing control device 6 for performing emphasized image processing on the above-mentioned display units 50, 53 based on the recognition results of the on-board external world recognition device 2, the driver state information acquired by the driver state acquisition device 3, and the detection results of the driver recognition detection device 4.

[0049] The vehicle-mounted environment recognition device 2 includes camera units 21 a and 21 b , a plurality of (e.g., five) laser radar units 22 a , 22 b , 22 c , 22 d , and 22 e , a plurality of (e.g., five) radar units 23 a , 23 b , 23 c , 23 d , and 23 e , a surrounding recognition device 24 , and a navigation device 25 .

[0050] The front camera unit 21a and the rear camera unit 21b are digital cameras using solid-state imaging elements such as charge-coupled devices (CCDs) or complementary metal oxide semiconductors (CMOSs), and capture images of the vehicle V in the forward and rearward directions of travel. These camera units 21a and 21b are mounted within the vehicle V, for example, on the roof of the vehicle, facing forward and rearward, respectively. Images captured by these camera units 21a and 21b are transmitted to the surrounding recognition device 24.

[0051] The LiDAR units 22a through 22e are each LiDAR (Light Detection and Ranging) units that detect objects around the vehicle V by measuring scattered light from pulsed laser light emitted by an object. The first LiDAR unit 22a is located at the right corner of the front of the vehicle V as viewed in the direction of travel and detects objects slightly to the right and forward of the vehicle V. The second LiDAR unit 22b is located at the left corner of the front of the vehicle V as viewed in the direction of travel and detects objects slightly to the left and forward of the vehicle V. The third LiDAR unit 22c is located at the center of the rear of the vehicle V in the vehicle width direction and detects objects behind the vehicle V. The fourth LiDAR unit 22d is located at the rear of the right side of the vehicle V and detects objects slightly to the right and rear of the vehicle V. The fifth LiDAR unit 22e is located at the rear of the left side of the vehicle V and detects objects slightly to the left and rear of the vehicle V. The detection signals from these LiDAR units 22a through 22e are transmitted to the surrounding recognition device 24.

[0052] Radar units 23a through 23e are millimeter-wave radars that detect objects around the vehicle V by measuring the reflected waves from millimeter-wave radiation emitted by objects. The first radar unit 23a is located at the right corner of the front of the vehicle V as viewed in the direction of travel and detects objects slightly to the right and forward of the vehicle V. The second radar unit 23b is located at the left corner of the front of the vehicle V as viewed in the direction of travel and detects objects slightly to the left and forward of the vehicle V. The third radar unit 23c is located at the center of the front of the vehicle V in the vehicle width direction and detects objects slightly to the front of the vehicle V. The fourth radar unit 23d is located at the right corner of the rear of the vehicle V as viewed in the direction of travel and detects objects slightly to the right and rear of the vehicle V. The fifth radar unit 23e is located at the left corner of the rear of the vehicle V as viewed in the direction of travel and detects objects slightly to the left and rear of the vehicle V. Detection signals from these radar units 23a through 23e are transmitted to the surrounding recognition device 24.

[0053] The surrounding recognition device 24 performs sensor fusion processing on the detection results from the camera units 21a, 21b, the lidar units 22a-22e, and the radar units 23a-23e to identify surrounding traffic participants and the traffic environment surrounding the vehicle V (i.e., obstacles, roads, lanes, signs, and signals). It also obtains movement state information related to the type and movement state of each surrounding traffic participant (i.e., position, movement direction, movement speed, movement acceleration, relative position and relative speed relative to the vehicle V), as well as traffic environment information related to the traffic environment. Furthermore, the surrounding recognition device 24 preferably assigns a unique identification number to each identified surrounding traffic participant so that the image processing control device 6 (described later) can identify each surrounding traffic participant, and the movement state information is associated with the identification number. The surrounding recognition device 24 transmits the obtained movement state information or traffic environment information to the driver recognition detection device 4 or the image processing control device 6.

[0054] The navigation device 25 includes, for example, a Global Navigation Satellite System (GNSS) receiver that determines the current position of the vehicle V based on signals received from GNSS satellites, and a storage device that stores map information. The navigation device 25 transmits traffic environment information related to the current position of the vehicle V and the types of surrounding roads, etc., acquired using the GNSS receiver and storage device, to the image processing control device 6.

[0055] The driver status acquisition device 3 is composed of various devices that acquire driver status information related to the driver's driving ability. More specifically, the driver status acquisition device 3 includes, for example, an in-vehicle camera that detects the driver's gaze direction and whether their eyes are open; a seatbelt sensor attached to the driver's seatbelt that detects the driver's pulse and breathing; a steering wheel sensor attached to the driver's steering wheel that detects the driver's skin potential; an in-vehicle microphone that detects conversations between the driver and fellow passengers; a wearable device that measures the driver's biological information, such as heart rate, blood pressure, and blood oxygen saturation; and a smartphone that stores the driver's schedule information. The driver status acquisition device 3 transmits the acquired driver status information to the image processing control device 6.

[0056] The driver recognition detection device 4 detects whether the driver has recognized surrounding traffic participants. Specifically, the driver recognition detection device 4 detects, for each surrounding traffic participant, whether the driver has visually confirmed the surrounding traffic participants identified by the onboard external recognition device 2. More specifically, the driver recognition detection device 4 includes the aforementioned in-vehicle camera that captures the driver's face. Based on the image data of the driver's face captured by the in-vehicle camera, the driver's line of sight is determined. The driver's line of sight is compared with the positions of surrounding traffic participants determined based on the movement status information to detect whether the driver has recognized surrounding traffic participants.

[0057] The front display unit 50 includes a windshield 51 located in front of the driver, and a heads-up display unit 52 that projects an image onto the windshield 51. The image in front of the vehicle V is projected on the windshield 51. The heads-up display unit 52 displays a predetermined image in a superimposed manner in a designated front display area on the windshield 51 in response to a command from the image processing control device 6, which will be described later.

[0058] The rear display unit 53 includes: an electronic interior rearview mirror 54 that projects an image behind the vehicle V; a left exterior rearview mirror unit 55 that projects an image behind the left side of the vehicle V; and a right exterior rearview mirror unit 56 that projects an image behind the right side of the vehicle V.

[0059] The electronic rearview mirror 54 is a display that projects an image of the rear of the vehicle V, for example, captured by the rear camera unit 21 b, on its image display surface. The electronic rearview mirror 54 is provided in a position slightly above the driver's view within the cabin of the vehicle V. The electronic rearview mirror 54 displays a predetermined image superimposed on a designated rear display area on the image display surface in response to a command from the image processing control device 6 described later.

[0060] The left and right exterior mirror units 55 and 56 are respectively provided on the left and right sides of the exterior of the vehicle V as viewed from the driver. These exterior mirror units 55 and 56 include mirror surfaces for projecting images of the left and right rearward directions of the vehicle V, respectively, and image display devices for superimposing and displaying a predetermined image in a designated rearward display area on these mirror surfaces in response to a command from an image processing control device 6 described later.

[0061] The image processing control device 6 includes: a warning object recognition unit 61 for identifying, among the traffic participants identified as surrounding traffic participants by the onboard external recognition device 2, traffic participants that have a significant impact on the safe driving of the vehicle V as warning objects; a display object setting unit 62 for setting one or more traffic participants identified as warning objects by the warning object recognition unit 61 as either display objects or non-display objects; and a display control unit 63 for performing emphasis image processing on the front display unit 50 and the rear display unit 53, wherein the emphasis image processing emphasizes the presence of the one or more traffic participants set as display objects. The functions of the warning object recognition unit 61, the display control unit 63, and the display object setting unit 62 are described in order below.

[0062] The warning object identification unit 61 identifies one or more traffic participants identified as surrounding traffic participants by the on-board external recognition device 2 as a warning object, based on the movement status information, traffic status information and driver status information, and identifies the traffic participant who has a greater impact on the safe driving of the vehicle V.

[0063] The alert target recognition unit 61 first calculates the degree of impact on the safe driving of the vehicle V for each of the traffic participants identified as surrounding traffic participants by the onboard external environment recognition device 2 based on the movement status information, traffic status information, and driver status information. In this embodiment, the degree of impact on the safe driving of the vehicle V is set as a parameter that increases or decreases based on the probability of contact between the target traffic participant and the vehicle V and the extent of damage to the vehicle V caused by the target traffic participant's actions.

[0064] More specifically, the alert target identification unit 61 detects or predicts dangerous behaviors of surrounding traffic participants that may threaten the safe driving of the vehicle V based on information such as movement status information, traffic environment information, and driver status information. Based on the results of the detection or prediction of these dangerous behaviors, the alert target identification unit 61 calculates the degree of influence of the surrounding traffic participants. Examples of dangerous behaviors that may threaten the safe driving of the vehicle V include: behaviors of surrounding traffic participants suddenly approaching the vehicle V; behaviors of surrounding traffic participants suddenly blocking or crossing the planned driving route of the vehicle V; behaviors of surrounding traffic participants suddenly slowing down while on the planned driving route of the vehicle V; behaviors of surrounding traffic participants passing close to the vehicle V at a faster speed than the vehicle V; behaviors of surrounding traffic participants following the vehicle V at a short distance; and behaviors of surrounding traffic participants provoking the driver of the vehicle V.

[0065] Furthermore, these surrounding traffic participants may also engage in dangerous behaviors as described above due to changes in their surrounding traffic environment (e.g., a reduction in the number of lanes or the presence of a merging point). Therefore, it is preferable that the alert target recognition unit 61 predicts dangerous behaviors of the surrounding traffic participants based not only on the movement state information but also on the traffic environment information.

[0066] Furthermore, the impact of dangerous behavior by surrounding traffic participants, such as those described above, on the safe driving of the vehicle V varies depending on the state of the driver of the vehicle V. Specifically, if the driver of the vehicle V is prone to delay in recognizing surrounding traffic participants, the impact on the safe driving of the vehicle V will be greater due to this delay in recognition by the driver. Therefore, it is preferable that the alert target recognition unit 61 calculates the degree of impact on the targeted surrounding traffic participants based not only on the movement state information and traffic environment information but also on the driver's state information.

[0067] After calculating the impact level for each surrounding traffic participant according to the above process, the alert target identification unit 61 compares these calculated impact levels with a predetermined impact level threshold to identify surrounding traffic participants to be identified as alert targets. Specifically, the alert target identification unit 61 identifies one or more surrounding traffic participants whose impact level exceeds the threshold as an alert target.

[0068] Here, the alert object recognition unit 61 may set the impact threshold as a fixed value or as a variable value that is sequentially set based on either or both of the traffic environment information and the driver's state information. More specifically, for example, if the alert object recognition unit 61 determines based on the driver's state information that the driver of the vehicle V is anxious, the impact threshold may be set to a lower value compared to if the driver is not anxious. In this way, by changing the impact threshold to a lower value as the driver's anxiety increases, the alert object recognition unit 61 can increase the frequency of performing the emphasized image processing (described later) and frequently encourage the driver to be aware of their surroundings. Furthermore, for example, if the alert object recognition unit 61 determines based on the traffic environment information that the surrounding traffic environment is prone to accidents (e.g., areas with narrow or narrow roads or areas near a merging point on a major road), the impact threshold may be set to a lower value compared to if the environment is less prone to accidents. In this way, the warning target recognition unit 61 changes the impact level threshold to a smaller value as the traffic environment around the vehicle V becomes more accident-prone, thereby increasing the frequency of the emphasized image processing described later and frequently prompting the driver to recognize the surroundings.

[0069] The alert object identification unit 61 sends relevant information such as the alert object identification number corresponding to the number of surrounding traffic participants identified as alert objects, the identification numbers unique to the surrounding traffic participants identified as alert objects, and the degree of influence on the surrounding traffic participants identified as alert objects to the display object setting unit 62 and the display control unit 63.

[0070] The display control unit 63 performs emphasis image processing on the front display unit 50 that projects the image in front of the vehicle V and the rear display unit 53 that projects the image behind and to the rear side of the vehicle V. The emphasis image processing emphasizes the presence of surrounding traffic participants set as display objects in the front display unit 50 and the rear display unit 53 through the processing described later in the display object setting unit 62.

[0071] First, the display control unit 63 calculates the position of the surrounding traffic participant set as the display object by the display object setting unit 62 based on the movement state information. Based on the calculated position of the display object, the display control unit 63 determines whether the display object is located in front of the vehicle V, that is, whether the display object is projected on the windshield 51 of the front display unit 50. If the display object is located in front of the vehicle V, the display control unit 63 performs emphasis image processing on the heads-up display unit 52 of the front display unit 50 to emphasize the presence of the display object projected on the windshield 51.

[0072] Figure 2: is a diagram showing an example of image enhancement processing for the front display unit 50. Figure 2 3 shows a case where a preceding vehicle 2a traveling to the left in front of the vehicle V is set as a display target among the surrounding traffic participants of the vehicle V. The alert target recognition unit 61 recognizes the preceding vehicle 2a as an alert target based on a determination that the preceding vehicle 2a may change lanes so as to block the lane in front of the vehicle V.

[0073] The display control unit 63 displays the portion of the preceding vehicle 2a set as the display target on the windshield 51 in front of the driver in the emphasis image processing of the front display unit 50. Figure 2 In the example, the portion of the windshield 51 where the driver can see the real image of the preceding vehicle 2a is used as the front display area, and a highlight image 631 having a specified shape is overlapped and displayed in the front display area to emphasize the existence of the preceding vehicle 2a.

[0074] Figure 2 The highlight image 631 is shown as a circular ring, but the present invention is not limited thereto. The highlight image 631 may also be a rectangular or other geometric shape as long as it attracts the driver's attention. Figure 2 The highlight image 631 is shown to be slightly smaller than the preceding vehicle 2a, but the present invention is not limited thereto. The highlight image 631 can be enlarged within a range that does not hinder the driver's visual confirmation, or can be changed according to the size of the image of the display object to be projected. Figure 2 While the figure illustrates a case where a translucent highlight image 631 is displayed superimposed on the preceding vehicle 2a, the present invention is not limited thereto. A non-translucent highlight image may be provided as long as the shape of the highlight image does not obstruct the driver's visual recognition. Furthermore, the color of the highlight image 631 is preferably set to a color that is different from the preceding vehicle 2a and the surrounding background to attract the driver's attention (e.g., green, yellow, or red). Furthermore, the display control unit 63 may flash the highlight image 631 at a predetermined period as needed.

[0075] Back to Figure 1If the display object is not located in front of the vehicle V, the display control unit 63 determines, based on the calculated position of the display object, whether the display object is located outside the blind spot behind or to the rear side of the vehicle V, that is, whether the display object is projected on either the image display surface of the electronic interior rearview mirror 54 of the rear display unit 53 or the mirror surface of the exterior rearview mirror units 55, 56. If the display object is located outside the blind spot behind or to the rear side of the vehicle V, the display control unit 63 performs emphasis image processing on either the electronic interior rearview mirror 54 or the exterior rearview mirror units 55, 56 of the rear display unit 53, wherein the emphasis image processing emphasizes the presence of the display object projected on the image display surface and the mirror surface.

[0076] Figure 3 : is a diagram showing an example of image enhancement processing of the electronic rearview mirror 54 of the rear display unit 53. Figure 3 3 shows a case where a following vehicle 3a traveling behind the vehicle V is set as a display target among the surrounding traffic participants of the vehicle V. Since the following vehicle 3a follows the vehicle V at a very short distance, the warning target recognition unit 61 recognizes the following vehicle 3a as a warning target.

[0077] The display control unit 63 displays the portion of the following vehicle 3a set as the display target on the image display surface of the electronic interior rearview mirror 54 and the mirror surfaces of the exterior rearview mirror units 55, 56 in the emphasis image processing of the rear display unit 53. Figure 3 In the example, the portion of the image display surface of the electronic interior rearview mirror 54 where the driver can see the image of the following vehicle 3a is used as the rear display area, and a highlight image 631 having the above-mentioned form is overlapped and displayed in the rear display area to emphasize the existence of the following vehicle 3a.

[0078] The display control unit 63 performs highlight image processing on the front display unit 50 and the rear display unit 53 according to the above process. This highlight image processing emphasizes the presence of surrounding traffic participants designated as display targets. Furthermore, if a surrounding traffic participant previously designated as a display target is now designated as a non-display target after being identified as a warning target, i.e., if the display target setting for the surrounding traffic participant is canceled, the display control unit 63 terminates the previously executed highlight image processing. Furthermore, if a surrounding traffic participant previously designated as a display target is no longer identified as a warning target by the warning target recognition unit 61, the display control unit 63 also terminates the previously executed highlight image processing. Upon terminating the highlight image processing, the display control unit 63 may either immediately erase the highlight image 631 previously displayed on the front display unit 50 and the rear display unit 53, or perform grayed-out image processing for a predetermined period of time. This grayed-out image processing gradually changes the morphology (i.e., shape, size, transparency, color, etc.) of the highlight image 631, thereby making the highlight image 631 less noticeable.

[0079] The display target setting unit 62 sets one or more surrounding traffic participants identified as alert targets by the alert target recognition unit 61 as either display targets or non-display targets. When the display target setting unit 62 sets a surrounding traffic participant identified as an alert target as a display target, the display control unit 63 performs emphasized image processing on the surrounding traffic participant set as a display target, as described above. Furthermore, when the display target setting unit 62 sets a surrounding traffic participant as a non-display target, emphasized image processing is not performed on the surrounding traffic participant, even if the surrounding traffic participant is identified as an alert target.

[0080] Specifically, the upper limit of the number of surrounding traffic participants simultaneously set as display targets in the display target setting unit 62 (hereinafter also referred to as the "upper limit of display targets") is equal to the number of times the emphasized image processing is simultaneously executed in the display control unit 63, that is, the number of highlight images 631 displayed on the front display unit 50 and the rear display unit 53. Furthermore, the upper limit of the display target setting is set to "2," expressed as "Nmax." Specifically, the following describes a case where the display target setting unit 62 sets as display targets a number of surrounding traffic participants, extracted within a range not exceeding the upper limit of the display target setting Nmax, from among the multiple surrounding traffic participants identified as alert targets, and sets all remaining traffic participants as non-display targets. However, the present invention is not limited to this. The upper limit of the display target setting Nmax can be set to "1" or a value greater than "2."

[0081] Figures 4 to 7 This is a flowchart illustrating the specific flow of the display / non-display setting process performed by the display target setting unit 62. The display / non-display setting process, as used herein, refers to the process of setting surrounding traffic participants identified as alert targets as either display targets or non-display targets. This process is repeatedly executed within a predetermined control cycle by the display target setting unit 62. Hereinafter, the process of the display target setting unit 62 setting surrounding traffic participants previously designated as display targets as non-display targets is also referred to as canceling the display target setting.

[0082] First, in step ST1 , the display target setting unit 62 updates a predetermined alert target list based on information sent from the alert target recognition unit 61 , and the process proceeds to step ST2 .

[0083] Figure 8 FIG. 1 is a diagram showing an example of a warning target list stored in the storage area of the display target setting unit 62. Figure 8 As shown, the alert target list records, in addition to the number of alert target recognitions in the current control cycle, various information associated with identification numbers unique to each surrounding traffic participant recognized as an alert target by the alert target recognition unit 61 in the current control cycle.

[0084] Here, in Figure 8 In the alert target list, the identification number of each surrounding traffic participant identified as an alert target by the alert target identification unit 61 in the current control cycle is recorded in the "surrounding traffic participant identification number" column.

[0085] exist Figure 8 In the alert target list, the “recognition time” column records the time when the surrounding traffic participant (i.e., the surrounding traffic participant associated with the identification number, the same below) as the target is recognized as the alert target by the alert target recognition unit 61.

[0086] exist Figure 8 In the alert target list, the "Influence Level (Latest)" column records the latest value of the influence level of the targeted surrounding traffic participants (i.e., the influence level value calculated by the alert target identification unit 61 during the current control cycle). While this embodiment describes the calculation of the influence level for each surrounding traffic participant by the alert target identification unit 61 per control cycle, the present invention is not limited thereto. The influence level for each surrounding traffic participant identified as an alert target by the alert target identification unit 61 may also be calculated per control cycle by the display target setting unit 62 using the same process as the alert target identification unit 61 described above.

[0087] exist Figure 8In the alert object list, the status of the display object flag for each surrounding traffic participant as an object is recorded in the "Display Object Flag" column. The display object flag is a flag indicating that the display object setting unit 62 sets the surrounding traffic participant as an object as a display object, and is initially set to "OFF" when it is identified as an alert object by the alert object recognition unit 61. As will be described later, when the display object setting unit 62 sets the surrounding traffic participant identified as an alert object as a display object, it sets the status of the display object flag associated with the surrounding traffic participant to "ON". In addition, when the display object setting unit 62 sets the surrounding traffic participant identified as an alert object as a non-display object, it sets the status of the display object flag associated with the surrounding traffic participant to "OFF".

[0088] exist Figure 8 In the alert target list, the "Set Start Time" column records the time at which the display target setting unit 62 sets the surrounding traffic participants identified as alert targets as display targets. As will be described later, when the display target setting unit 62 sets the surrounding traffic participants identified as alert targets as display targets, it sets the time at that time as the setting start time.

[0089] exist Figure 8 The "Highlight Display Period" column in the alert target list contains the highlighted display period, which corresponds to the maximum period during which surrounding road users are continuously displayed. As described later, the display target setting unit 62 sets the highlighted display period based on the latest impact level during the control cycle.

[0090] exist Figure 8 In the alert object list, the status of the release flag for each surrounding traffic participant as an object is recorded in the "Release Flag" column. The release flag is a flag indicating that the driver has identified the surrounding traffic participant as an object, and is set to "OFF" when it is first identified as an alert object by the alert object recognition unit 61. In addition, as described later, the display object setting unit 62 sets the release flag from "OFF" to "ON" when the display object setting for the surrounding traffic participant is released. In addition, as described later, after the display object setting unit 62 sets the release flag to "ON", if a specified condition is met, the display object setting unit 62 resets the release flag from "ON" to "OFF". In addition, as described later, the display object setting unit 62 will not set a surrounding traffic participant whose release flag status is set to "ON" as a display object.

[0091] In addition, Figure 8In the alert target list, the "Impact Level (When Released)" column records the impact level value at the time the display target setting for the target surrounding traffic participants was released. As described later, when the display target setting unit 62 sets the release flag from "OFF" to "ON," it stores the impact level value at that time in the alert target list.

[0092] Back to Figure 4 In step ST1, the display object setting unit 62 updates the alert object list based on the information sent from the alert object recognition unit 61 and then moves on to step ST2. More specifically, the display object setting unit 62 updates the "Level of Influence (Latest)" column in the alert object list to the latest value. Furthermore, if the alert object recognition unit 61 identifies a new surrounding traffic participant as an alert target, the display object setting unit 62 adds information such as the identification number of the new surrounding traffic participant and the latest level of influence value to the alert object list and increments the alert object recognition count by one. Furthermore, if a surrounding traffic participant previously identified as an alert target by the alert object recognition unit 61 is no longer identified as an alert target, the display object setting unit 62 deletes all information related to the no longer identified surrounding traffic participant from the alert object list and decrements the alert object recognition count by one.

[0093] Next, in step ST2, the display target setting unit 62 refers to the alert target list and determines whether the number of alert target recognitions is 0. If the display target setting unit 62 determines YES in step ST2, that is, if the number of alert target recognitions is 0, the display / non-display setting process ends without executing the following process. Alternatively, if the display target setting unit 62 determines NO in step ST2, that is, if the number of alert target recognitions is greater than 0, the process proceeds to step ST3.

[0094] In step ST3, the display target setting unit 62 executes a cancellation flag update process and transfers to step ST11. The cancellation flag update process is a process for updating the status of the cancellation flag defined in the alert target list. The details will be referred to below. Figure 9 To explain.

[0095] In step ST11, the display object setting unit 62 refers to the alert object list and determines whether the number of identified alert objects is equal to "1," which is less than the upper limit Nmax of the number of display objects set. If the determination result in step ST11 is YES, that is, if the number of identified alert objects is "1," the display object setting unit 62 proceeds to step ST12. Furthermore, if only one surrounding traffic participant is identified as an alert object, the display object setting unit 62 performs the processing of steps ST12 to ST26 described below. Therefore, in the following description of the processing of steps ST12 to ST26, for ease of understanding, the surrounding traffic participant identified as an alert object is referred to as "traffic participant X."

[0096] First, in step ST12, the display target setting unit 62 refers to the alert target list and determines whether the display target flag for road user X is "OFF." Specifically, in step ST12, the display target setting unit 62 determines whether road user X was not set as a display target in the previous control cycle. In other words, whether emphasized image processing for road user X was not performed in the previous control cycle. If the result of the determination in step ST12 is YES, the display target setting unit 62 proceeds to step ST13.

[0097] In step ST13 , the display object setting unit 62 refers to the alert object list and determines whether the state of the cancellation flag for the road user X is “OFF”. If the determination result in step ST13 is YES, the display object setting unit 62 proceeds to step ST14 .

[0098] In step ST14, the display target setting unit 62 sets road user X as a display target, and the process proceeds to step ST15. Thus, based on the setting of road user X as a display target by the display target setting unit 62, the display control unit 63 begins highlighting the image of road user X. In step ST15, the alert target list is updated, and the display / non-display setting process ends. More specifically, based on the new setting of road user X as a display target in step ST14, the display target setting unit 62 sets the display target flag for road user X in the alert target list from "OFF" to "ON" and then records the current time in the "Setting Start Time" column for road user X in the alert target list.

[0099] As described above, the state of the cancel flag for road user X is initially set to "OFF" when the alert target recognition unit 61 identifies road user X as an alert target. Therefore, if only one person (i.e., road user X) is identified as an alert target by the alert target recognition unit 61, the display target setting unit 62 immediately sets road user X as a display target after the alert target recognition unit 61 identifies road user X as an alert target and begins highlighting the image of road user X.

[0100] On the other hand, if the result of the determination in step ST13 is NO, that is, if the state of the cancellation flag for road user X is "ON," the display target setting unit 62 completes the display-non-display setting process, setting road user X to non-display. Thus, the display target setting unit 62 does not set surrounding road users whose cancellation flags are set to "ON" as display targets.

[0101] On the other hand, if the result of determination in step ST12 is NO, that is, if the traffic participant X has already been set as a display target in the previous control cycle, the display target setting unit 62 proceeds to step ST21 (see Figure 5 ).

[0102] In step ST21, the display target setting unit 62 refers to the detection results of the driver recognition detection device 4 and the alert target list to determine whether the driver of the vehicle V has recognized the road participant X set as the display target from the time the setting was started to the present. If the determination result of step ST21 is NO, the display target setting unit 62 proceeds to step ST22.

[0103] In step ST22, the display target setting unit 62 refers to the alert target list to obtain the latest impact level for traffic participant X, which has been designated as a display target. Based on the obtained impact level, it calculates a highlighted display period for traffic participant X, and then proceeds to step ST23. In step ST22, the display target setting unit 62 preferably calculates a longer highlighted display period for traffic participant X as the impact level increases. In step ST23, the display target setting unit 62 updates the alert target list, and then proceeds to step ST24. More specifically, based on the newly calculated highlighted display period for traffic participant X in step ST22, the display target setting unit 62 records the newly calculated highlighted display period in the "Highlighted Display Period" column for traffic participant X in the alert target list.

[0104] In step ST24, the display target setting unit 62 refers to the alert list and determines whether the highlighted display period has elapsed since the start of the setting for road user X. If the display target setting unit 62 determines "NO" in step ST24, it terminates the display / non-display setting process while maintaining the display target setting for road user X. Specifically, if the display target setting unit 62 determines "NO" in step ST24, it terminates the display / non-display setting process while maintaining the display target flag for road user X in the alert list "ON."

[0105] In addition, when the judgment result of step ST21 is YES or the judgment result of step ST24 is YES, that is, after the traffic participant X is set as the display object, the display object setting unit 62 transfers to step ST25 after the highlighted display period for the traffic participant X or when the driver's recognition of the traffic participant X is detected.

[0106] In step ST25, the display target setting unit 62 sets the road user X, previously set as a display target, to a non-display target, and the process then moves to step ST26. In other words, the display target setting unit 62 cancels the display target setting for road user X, and the process then moves to step ST26. Thus, in response to the cancellation of the display target setting for road user X by the display target setting unit 62, the display control unit 63 terminates the emphasized image processing for road user X. In step ST26, the display target setting unit 62 updates the alert target list and terminates the display / non-display setting process. More specifically, in response to the cancellation of the display target setting for road user X in step ST25, the display target setting unit 62 changes the state of the display target flag for road user X in the alert target list from "ON" to "OFF," and also changes the state of the cancellation flag from "OFF" to "ON." Furthermore, the display control unit 63 records the latest impact level in the "Influence Level (at cancellation)" column.

[0107] As described above, after canceling the display target setting for a surrounding traffic participant, the display target setting unit 62 sets the state of the canceling flag for the surrounding traffic participant from "OFF" to "ON". On the other hand, as described above, the display target setting unit 62 does not set the surrounding traffic participant whose canceling flag state is set to "ON" as a display target (refer to the above-mentioned step T13). Therefore, unless the following reference is made Figure 9 The canceling flag update process described above resets the canceling flag from "ON" to "OFF", and the display target setting unit 62 does not set the surrounding road users as display targets again.

[0108] Figure 9 This is a flowchart showing the specific process of updating the canceling flag.

[0109] First, in step ST31, the display target setting unit 62 refers to the alert target list and identifies all of one or more surrounding traffic participants whose disabling flags are set to "ON," and then moves on to step ST32. Hereinafter, the surrounding traffic participants in the alert target list whose disabling flags are set to "ON" will also be referred to as "disabling participants."

[0110] In step ST32 , the display target setting unit 62 refers to the alert target list, acquires the influence level at the time of release and the latest influence level for each releaser, and then moves to step ST33 .

[0111] In step ST33, the display target setting unit 62 calculates the degree of influence increase for each releaser and then moves on to step ST34. More specifically, the display target setting unit 62 subtracts the degree of influence at the time of release from the latest degree of influence obtained in step ST32 to calculate the degree of influence increase for each releaser (degree of influence increase = latest degree of influence - degree of influence at the time of release).

[0112] In step ST34, the display target setting unit 62 determines whether there is a person whose influence degree increase is greater than a predetermined positive value threshold value among the one or more releasers determined in step ST31. If the result of the determination in step ST34 is NO, the display target setting unit 62 does not update the alert target list and moves to step ST11 (see Figure 4 ), if it is YES, go to step ST35.

[0113] In step ST35, the display target setting unit 62 updates the alert target list and proceeds to step ST11. More specifically, the display target setting unit 62 resets the state of the release flag from "ON" to "OFF" for the releaser whose influence degree increase exceeds the amplitude threshold in the alert target list.

[0114] Here, the function of the cancellation sign as described above will be described with reference to a case where a certain road user Y is recognized as a warning target by the warning target recognition unit 61 .

[0115] In this case, when road user Y is identified as an alert target, the display target setting unit 62 sets road user Y as a display target (see step ST14), thereby performing emphasized image processing on road user Y. Subsequently, if the driver recognizes road user Y as a display target (see step ST21) or if the emphasized display time has elapsed since the setting start time (see step ST24), the display target setting unit 62 cancels the display target setting for road user Y (see step ST25) and sets the status of the cancellation flag for road user Y from "OFF" to "ON" (see step ST26). Furthermore, as long as the cancellation flag is set to "ON," the display target setting unit 62 will not set road user Y as a display target again (see step ST13). Thereafter, if the degree of influence of road user Y, whose cancellation flag was set to "ON," has increased by a predetermined magnitude threshold or more since the time that road user Y was set as a non-display target or the time that the driver detected road user Y (see step ST34), the display target setting unit 62 resets the cancellation flag for road user Y from "ON" to "OFF" (see step ST35). This allows road user Y to be set as a display target. Therefore, as long as the total number of surrounding road users set as display targets (hereinafter referred to as the "number of display targets") at the time the cancellation flag for road user Y was reset from "ON" to "OFF" is less than the upper limit Nmax of display target settings, the display target setting unit 62 immediately sets road user Y as a display target again (see step ST13), thereby resuming the emphasized image processing for road user Y.

[0116] On the other hand, when the result of determination in step ST11 is NO, that is, when the number of warning target recognitions is "2" or more, the display target setting unit 62 proceeds to step ST41 (see Figure 6 ). In addition, when two or more surrounding road users are identified as alert targets, the display target setting unit 62 executes steps ST41 to ST59 described below (see Figure 6 and Figure 7 Therefore, in the following description of the processing of steps ST41 to ST59, for ease of understanding, the surrounding traffic participants identified as alert targets, that is, the surrounding traffic participants recorded in the alert target list, are expressed as "traffic participants A, B, ..."

[0117] First, in step ST41, the display target setting unit 62 refers to the alert target list and determines whether any of the multiple road users A, B, ... have been set as display targets. In other words, the display target setting unit 62 determines whether any of the multiple road users A, B, ... have a display target flag set to "ON" in the alert target list. If the result of the determination in step ST41 is YES, the display target setting unit 62 proceeds to step ST42.

[0118] Furthermore, in the following description of the processing of steps ST42 to ST47, the one or more traffic participants among the plurality of traffic participants A, B, ... that have been set as display targets in the processing of step ST41 will also be referred to as traffic participants X1, .... Furthermore, in the following processing of steps ST42 to ST47, the display target setting unit 62 determines whether to release the display target setting for the one or more traffic participants X1, ... that have been set as display targets.

[0119] First, in step ST42, the display target setting unit 62 refers to the detection results of the driver recognition detection device 4 and the alert target list to determine whether any of the one or more road users X1,... set as display targets has been recognized by the driver since the setting start time. If the determination result of step ST42 is NO, the display target setting unit 62 proceeds to step ST43.

[0120] In step ST43, the display object setting unit 62 refers to the alert object list to obtain the latest impact level for one or more road users X1,... designated as display targets. Based on the obtained impact level, it calculates a highlighted display period for each road user X1,..., and then proceeds to step ST44. In step ST43, similar to step ST22 described above, the display object setting unit 62 preferably calculates a longer highlighted display period for a greater impact level. In step ST44, the display object setting unit 62 updates the alert object list and then proceeds to step ST45. More specifically, based on the newly calculated highlighted display period for each road user X1,... in step ST43, the display object setting unit 62 records the newly calculated highlighted display period in the "Highlighted Display Period" column for each road user X1,... in the alert object list.

[0121] In step ST45 , the display target setting unit 62 refers to the warning list and determines whether there is a road user whose emphasis display period has passed since the setting start time among the one or more road users X1 , . . . set as display targets.

[0122] In addition, when the judgment result of step ST42 is YES or the judgment result of step ST45 is YES, that is, when there is a traffic participant who has passed the emphasized display period or a traffic participant who has been identified by the driver among the one or more traffic participants X1,... set as display objects, the display object setting unit 62 transfers to step ST46.

[0123] In step ST46, the display object setting unit 62 cancels the display object setting for the traffic participant X1, ..., which has passed the highlighted display period or the traffic participant identified by the driver, among the one or more traffic participants X1, ..., which have been set as display objects, and then transfers to step ST47. In this way, according to the cancellation of the display object setting in the display object setting unit 62, the display control unit 63 ends the emphasized image processing for the person whose display object setting has been cancelled. In step ST47, the display object setting unit 62 updates the alert object list and transfers to step ST51 (see Figure 7 More specifically, in response to the display object setting being released in step ST46, the display object setting unit 62 sets the state of the display object flag for the object whose display object setting has been released in the alert object list from "ON" to "OFF", and sets the state of the release flag from "OFF" to "ON", and further records the latest impact level in the "Impact Level (when released)" column.

[0124] If the result of determination in step ST41 is NO or if the result of determination in step ST45 is NO, the display object setting unit 62 proceeds to step ST51 (see Figure 7 ). In the following processing of steps ST51 to ST59 , the display target setting unit 62 determines whether there is a road user newly set as a display target from among the plurality of road users A, B, . . . identified as alert targets.

[0125] First, in step ST51, the display target setting unit 62 refers to the alert target list and obtains the total number of traffic participants currently designated as display targets, i.e., the number of display targets N. The process then proceeds to step ST52. More specifically, the display target setting unit 62 obtains the number of display targets N by counting the number of traffic participants in the alert target list whose display target flags are set to "ON."

[0126] In step ST52, the display target setting unit 62 determines whether the number of display target persons N obtained in step ST51 is "0." If the determination result in step ST52 is YES, that is, if none of the multiple road users A, B, ... identified as alert targets are set as display targets, the display target setting unit 62 proceeds to step ST53.

[0127] In step ST53, the display target setting unit 62 refers to the alert target list and extracts a road user whose release flag is set to "OFF" from the plurality of road users A, B, ... as a new candidate, and then proceeds to step ST54. If the display target setting unit 62 is unable to extract even a single new candidate from the plurality of road users A, B, ... during the processing of step ST53, the display / non-display setting process ends without executing steps ST54 and ST55.

[0128] In step ST54, the display target setting unit 62 references the alert target list to obtain the latest influence level for each new candidate extracted in step ST53. It then sets the new candidates extracted, starting with the one with the highest influence level, as display targets, within a range not exceeding the upper limit Nmax of display targets. The process then proceeds to step ST55. Thus, based on the one or more new candidates extracted within a range not exceeding the upper limit Nmax of display targets being set as display targets by the display target setting unit 62 (hereinafter referred to as "newly set candidates"), the display control unit 63 begins highlighting the image of these new candidates. In step ST55, the display target setting unit 62 updates the alert target list and terminates the display / non-display setting process. More specifically, the display target setting unit 62 sets the display target flag for each new candidate in the alert target list from "OFF" to "ON" and records the current time in the "Setting Start Time" column for each new candidate in the alert target list.

[0129] On the other hand, when the determination result of step ST52 is NO, that is, when at least one of the plurality of road users A, B, . . . identified as alert targets is set as a display target, the display target setting unit 62 proceeds to step ST56 .

[0130] In step ST56, the display target setting unit 62 determines whether the number N of display target users obtained in step ST51 is less than the upper limit Nmax of display target settings. If the determination result in step ST56 is NO, that is, if the alert targets exceeding the upper limit Nmax of display target settings have already been set as display targets, the display / non-display setting process ends. In other words, the display target setting unit 62 does not set more alert targets than the upper limit Nmax of display target settings as display targets.

[0131] If the result of step ST56 is YES, that is, if the number of display subjects N is less than the upper limit Nmax of display subjects and there is space available for new display subjects, the display target setting unit 62 proceeds to step ST57. Furthermore, in the following steps ST57 to ST59, the number Nres (= Nmax - N), obtained by subtracting the number of display subjects N from the upper limit Nmax of display subjects, is referred to as the remaining upper limit.

[0132] In step ST57, the display target setting unit 62 refers to the alert target list and selects, as additional candidates, those road users whose display target flags and the release flags are both set to "OFF" from among the multiple road users A, B, ..., and then proceeds to step ST58. If the display target setting unit 62 is unable to select even one additional candidate from among the multiple road users A, B, ... during step ST57, the display / non-display setting process ends without executing steps ST58 and ST59.

[0133] In step ST58, the display target setting unit 62 references the alert target list to obtain the latest influence level of each additional candidate extracted in step ST57. It then sets the additional candidates extracted, starting with the one with the highest influence level, as display targets within the remaining set upper limit Nres, and proceeds to step ST59. Thus, based on the display target setting unit 62 setting one or more additional candidates extracted within the remaining set upper limit Nres as display targets (hereinafter, the one or more additional candidates newly set as display targets in step ST58 will also be referred to as "additionally set candidates"), the display control unit 63 begins highlighting the image of these additional candidates. Furthermore, by setting the additional candidates as display targets within the remaining set upper limit Nres in step ST58, the display target setting unit 62 ensures that the number of display candidates, N, does not exceed the display target set upper limit Nmax, even when combined with the candidates already set as display targets. In step ST59, the display target setting unit 62 updates the alert target list and terminates the display / non-display setting process. More specifically, the display target setting unit 62 sets the display target flag for each additional setter in the alert target list from OFF to ON, and further records the current time in the “setting start time” column for each additional setter in the alert target list.

[0134] According to the driving assistance system 1 of this embodiment, the following effects are achieved.

[0135] (1) The on-vehicle external recognition device 2 identifies traffic participants around the vehicle V as surrounding traffic participants. The warning object recognition unit 61 identifies traffic participants that have a greater impact on the safe driving of the vehicle V among the traffic participants identified as surrounding traffic participants as warning objects. The display object setting unit 62 sets the traffic participant identified as the warning object as a display object. The display control unit 63 performs emphasis image processing on the front display unit 50 and the rear display unit 53 that can be visually recognized by the driver. The emphasis image processing emphasizes the presence of the traffic participant set as the display object. Therefore, according to the driving assistance system 1, when there is a warning object that has a greater impact on the safe driving of the vehicle V around the vehicle V and the warning object is set as the display object, the emphasis image processing is performed to emphasize the presence of the warning object (display object) on the front display unit 50 or the rear display unit 53, so that the driver can easily recognize the presence of the warning object. In addition, in the driving assistance system 1, when it is detected that the driver has recognized a specific road user set as a display object, the display object setting unit 62 cancels the setting of the display object for the specific road user (for example, referring to Figure 5 Step ST25 and Figure 6In step ST46), the state of the release flag for the specific traffic participant is set to "ON". Accordingly, the display control unit 63 ends the emphasis image processing for the specific traffic participant whose recognition by the driver is detected. Thus, according to the driving assistance system 1, it is possible to prevent the driver from feeling annoyed by the emphasis image processing being performed even after the driver has been recognized. In addition, in the driving assistance system 1, as described above, after the setting of the display object for a specific traffic participant is released, if the degree of influence on the specific traffic participant increases, the display object setting unit 62 resets the state of the release flag for the specific traffic participant to "OFF" (for example, refer to Figure 9 Step ST35), and then set the specific traffic participant as the display object again (for example, referring to Figure 4 Step ST14, Figure 7 Steps ST54 and ST58 are performed. Accordingly, the display control unit 63 resumes highlighting the image of the specific road user whose impact has increased. Thus, according to the driving assistance system 1, even after the driver has identified a specific road user, highlighting the image of the same specific road user is continued based on subsequent changes in the situation. This allows the driver to be reminded of the presence of a specific road user who may affect the safe driving of the vehicle V without causing any nuisance, thereby assisting the driver in safe driving. Furthermore, this contributes to the development of a sustainable transportation system.

[0136] (2) In the driving assistance system 1, after the display object setting for a specific road user is canceled based on the detection of the driver's recognition as described above, if the degree of influence on the specific road user increases from the time when the display object setting is canceled or the time when the driver's recognition is detected, the display object setting unit 62 resets the state of the canceling flag for the specific road user from "ON" to "OFF" (see Figure 9 Step ST35), and then set the specific traffic participant as the display object again (for example, referring to Figure 4 Step ST14, Figure 7 Steps ST54 and ST58 are then performed, and processing of the emphasized image of the specific road user is restarted. Thus, the driving assistance system 1 uses the time at which the display object is released or the time at which the driver's recognition is detected as a basis for determining whether the impact has increased. This allows for a reasonable determination of whether the driver should be prompted to recognize the specific road user again, thereby assisting the driver in driving safely.

[0137] (3) In the driving assistance system 1, the alert object recognition unit 61 or the display object setting unit 62 calculates the degree of influence based on the movement state information related to the movement state of surrounding traffic participants. Thus, the driving assistance system 1 can accurately calculate the degree of influence on the safe driving of the vehicle V by taking into account the movement state of surrounding traffic participants. Therefore, surrounding traffic participants that the driver should reliably recognize can be set as display objects.

[0138] (4) In the driving assistance system 1 , the driver recognition detection device 4 detects whether the driver recognizes surrounding traffic participants based on the image data of the driver's face, thereby detecting whether the driver has visually recognized surrounding traffic participants.

[0139] (5) In the driving assistance system 1, the alert object recognition unit 61 or the display object setting unit 62 detects or predicts the behavior of surrounding traffic participants based on the movement status information of the surrounding traffic participants, and calculates the degree of influence of the surrounding traffic participants based on the results of these detections or predictions. Thus, the driving assistance system 1 can accurately calculate the degree of influence by reflecting the results of the detection or prediction of the behavior of surrounding traffic participants, so that surrounding traffic participants that the driver should reliably identify can be set as display objects.

[0140] (6) In the driving assistance system 1, the alert target recognition unit 61 or the display target setting unit 62 predicts the behavior of surrounding traffic participants based on traffic environment information related to the traffic environment surrounding the vehicle V. This allows the behavior of surrounding traffic participants to be predicted with high accuracy, and the degree of their influence to be calculated with high accuracy. Thus, the driving assistance system 1 allows surrounding traffic participants that the driver should reliably recognize to be set as display targets.

[0141] (7) In the driving assistance system 1, the alert object recognition unit 61 or the display object setting unit 62 calculates the degree of influence based on the movement state information and the driver state information. This allows the degree of influence to be calculated with high accuracy, reflecting the current driving ability of the driver of the vehicle V. Thus, the driving assistance system 1 can set surrounding traffic participants that the driver should reliably recognize as display objects.

[0142] (8) In the driving assistance system 1, after a specific road user identified as a warning target is set as a display target, the display target setting unit 62 cancels the setting of the display target for the specific road user after a predetermined emphasized display period has elapsed or when the driver's recognition of the display target is detected (see Figure 5 Step ST25, Figure 6Step ST46 is then performed, and the emphasized image processing ends. Thus, the driving assistance system 1 can prevent the driver from becoming annoyed by prolonged emphasized image processing. Furthermore, by setting the emphasized display period based on the degree of influence of a specific road user, the emphasized display period can be set to a reasonable length based on the degree of influence of the specific road user.

[0143] (9) In the driving assistance system 1, when multiple traffic participants are identified as alert targets, the display target setting unit 62 sets the traffic participants extracted in descending order of influence as display targets within a range not exceeding a predetermined display target setting upper limit number Nmax (or a remaining setting upper limit number Nres) (see Figure 7 Thus, according to the driving assistance system 1, even when multiple road users are identified as warning targets, the emphasized images of road users exceeding the set upper limit number Nmax of display targets are not simultaneously processed, thereby preventing the driver from feeling annoyed or confused.

[0144] (10) In the driving assistance system 1, the alert object recognition unit 61 recognizes as alert objects any traffic participant whose influence level is greater than a threshold level of influence among the traffic participants identified as surrounding traffic participants, and sets the threshold level of influence based on the driver's state information. Thus, the frequency of performing the emphasized image processing can be changed according to the driver's current state, thereby assisting the driver in driving safely.

[0145] (11) In the driving assistance system 1, the display control unit 63 uses the portion of the windshield 51 of the front display unit 50 where the image of the display object is projected as the front display area during the emphasis image processing, and superimposes the highlight image 631 on the front display area to emphasize the presence of the display object. This allows the driver to easily recognize the presence of the display object in front of the vehicle V.

[0146] (12) In the driving assistance system 1, the display control unit 63 uses the portion of the rear display unit 53 where the image of the display object is projected as the rear display area during the emphasis image processing, and superimposes the highlight image 631 on the rear display area to emphasize the presence of the display object. This allows the driver to easily recognize the presence of the display object behind the vehicle V.

[0147] While one embodiment of the present invention has been described above, the present invention is not limited thereto and the detailed structure may be appropriately modified within the scope of the gist of the present invention.

[0148] For example, in the above-described embodiment, the onboard world recognition device 2 identifies the surrounding traffic participants and the traffic environment of the vehicle V based on information acquired by onboard sensors such as the camera units 21a-21b and the lidar units 22a-22e that travel with the vehicle V. However, the present invention is not limited to this. The onboard world recognition device 2 may also identify the surrounding traffic participants and the traffic environment of the vehicle V based on information acquired through inter-vehicle communication between the vehicle V and the surrounding traffic participants, or through wireless communication between the vehicle V and a traffic management server.

[0149] Reference numerals

[0150] V: Vehicle (this vehicle)

[0151] 1: Driving assistance system

[0152] 2: Vehicle-mounted external recognition device (external recognition device)

[0153] 3: Driver status acquisition device

[0154] 4: Driver identification detection device

[0155] 50: Front display unit (display unit)

[0156] 53: Rear display unit (display unit)

[0157] 6: Image processing control device

[0158] 61: Alert Object Identification Department

[0159] 62: Display object setting unit

[0160] 63: Display control unit.

Claims

1. A driving assistance system, characterized in that: have: An external recognition device for recognizing traffic participants around the vehicle as surrounding traffic participants; a warning object recognition unit for recognizing, as a warning object, a traffic participant that has a greater impact on safe driving of the vehicle among the traffic participants identified as the surrounding traffic participants; a display object setting unit for setting the traffic participant identified as the aforementioned warning object as a display object; A display control unit performs an emphasis image processing on a display unit provided at a position where the driver of the host vehicle can visually recognize the display unit, wherein the emphasis image processing emphasizes the presence of a traffic participant set as the display object; and A driver identification detection device detects whether the driver has identified the surrounding traffic participants; and When it is detected that the driver has identified the specific traffic participant set as the display object, the display object setting unit cancels the setting of the display object for the specific traffic participant, and thereafter, when the degree of influence on the specific traffic participant increases, the specific traffic participant is set as the display object again.

2. The driving assistance system according to claim 1, wherein: After the setting of the display object for the specific traffic participant is released, if the degree of influence on the specific traffic participant increases from the time point when the setting of the display object is released or the time point when the identification of the driver is detected, the display object setting unit sets the specific traffic participant as the display object again.

3. The driving assistance system according to claim 1 or 2, wherein: The external recognition device obtains the movement state information related to the movement state of the surrounding traffic participants. The alert target recognition unit or the display target setting unit calculates the influence degree based on the movement state information.

4. The driving assistance system according to claim 1 or 2, wherein: The driver recognition detection device includes a camera for capturing the driver's face, and detects whether the driver recognizes the surrounding traffic participants based on image data acquired by the camera.

5. The driving assistance system according to claim 3, wherein: The alert object recognition unit or the display object setting unit detects or predicts the behavior of the surrounding traffic participants based on the movement state information, and calculates the influence degree based on the detection or prediction results.

6. The driving assistance system according to claim 5, wherein: The external recognition device obtains traffic environment information related to the traffic environment around the vehicle. The warning target recognition unit or the display target setting unit predicts the behavior of the surrounding traffic participants based on the traffic environment information.

7. The driving assistance system according to claim 3, wherein: The driving assistance system further includes a driver state acquisition device, which acquires driver state information related to the driving ability of the driver. The warning target recognition unit or the display target setting unit calculates the influence degree based on the movement state information and the driver state information.

8. The driving assistance system according to claim 1 or 2, wherein: After the specific road user is set as the display object, the display object setting unit cancels the setting of the display object for the specific road user after a predetermined emphasized display period has elapsed or when recognition of the display object by the driver is detected. The emphasized display period is set based on the degree of influence on the specific traffic participant.

9. The driving assistance system according to claim 1 or 2, wherein: When a plurality of traffic participants are identified as the alert targets, the display target setting unit sets the traffic participants extracted in order from the one with the greater influence as the display targets within a range not exceeding a predetermined upper limit.

10. The driving assistance system according to claim 1 or 2, wherein: The driving assistance system further includes a driver state acquisition device, which acquires driver state information related to the driving ability of the driver. The alert object recognition unit recognizes, as the alert object, a traffic participant whose influence degree is greater than a predetermined influence degree threshold among the traffic participants recognized as the surrounding traffic participants, The aforementioned influence degree threshold is set based on the aforementioned driver state information.

11. The driving assistance system according to claim 1 or 2, wherein: The display unit includes a front display unit, which displays an image in front of the vehicle. In the emphasized image processing, the display control unit uses the portion of the front display unit where the image of the display object is projected as a front display area, and displays a highlight image in the front display area in an overlapping manner, thereby emphasizing the presence of the display object.

12. The driving assistance system according to claim 11, wherein: The display unit includes a rear display unit that displays an image behind the vehicle. In the emphasized image processing, the display control unit uses the portion of the rear display unit where the image of the display object is projected as a rear display area, and displays a highlight image in the rear display area in an overlapping manner, thereby emphasizing the presence of the display object.

Images