Driver assistance device, driver assistance system and driver assistance method

Abstract

A driving assistance device according to the present disclosure includes: a prediction error calculation unit that calculates a prediction error that is a difference between an actual image in which an actual situation is detected and a predicted image predicted from an image in a traveling direction of a vehicle, the image being detected by a vehicle exterior camera that captures the periphery of the vehicle; an attention state estimation unit that estimates an attention state of a driver of the vehicle based on the prediction error; and an output control unit that outputs driving assistance information for inducing an attention state and / or a behavior change and / or an awareness change regarding a driving manipulation that are more suitable for driving at that time, based on the attention state.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a driving assistance device, a driving assistance system and a driving assistance method. TECHNICAL BACKGROUND

[0002] A driver drives a vehicle in accordance with traffic regulations, paying attention to pedestrians, obstacles, and the like based on traffic lights, road signs, lanes, and the like. Since the road conditions on which the vehicle is traveling change from moment to moment, it is possible to contribute to safe driving if driving assistance information can be presented in accordance with the change in the road conditions.

[0003] For example, Patent Literature 1 discloses a method for estimating a state of a driver's decreased attention function based on the driver's surroundings and the driver's gaze movement, and performing driving assistance. Furthermore, Patent Literature 2 discloses a method for calculating a visual recognition level indicating the driver's visual recognizability based on gaze direction information, driving environment information, driving ability information, and the like, and controlling a display position or the like of display information to be presented to the driver. CITATION LISTPatent literature Patent Literature 1: JP 2021-130389 A Patent literature 2: WO 2020 / 208804 A SUMMARY OF THE INVENTION

[0004] However, even if there is no impairment of attention function due to fatigue or the like, or if the traffic environment is easily recognized visually, a person may cause a traffic accident or a near-miss due to a cognitive factor. Therefore, even with the techniques of Patent Literatures 1 and 2, it is difficult to sufficiently suppress the traffic accident or near-miss due to the driver's cognitive factor.

[0005] The object of the present disclosure is to provide a driving assistance device, a driving assistance system and a driving assistance method capable of detecting a state in which it is difficult to pay attention to a target to be observed despite a perfect driving condition and / or a situation in which it is easy to recognize a traffic environment and appropriately presenting information to the driver.

[0006] A driving assistance device according to the present disclosure includes: a prediction error calculation unit that calculates a prediction error that is a difference between a predicted image and an actual image, the predicted image being predicted from an image in a traveling direction of a vehicle acquired by a vehicle exterior camera that captures a periphery of the vehicle, an actual situation being captured in the actual image; an attention state estimation unit that estimates an attention state of a driver based on the prediction error; and an output control unit that outputs driving assistance information for inducing an attention state and / or a behavior change and / or an awareness change regarding a driving manipulation that are more suitable for driving at that time, based on the attention state.

[0007] A driving assistance device, a driving assistance system, and a driving assistance method according to the present disclosure are capable of detecting a state in which attention is to be paid to a target to be paid attention to despite a perfect driving condition and / or a situation in which a traffic environment is easy to recognize in a driver's routine drive, and appropriately presenting information for prompting a more appropriate state of attention and / or a change in behavior and / or a change in awareness regarding a driving manipulation based on the state of attention that is more suitable for driving at that time. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram showing an example of the configuration of a vehicle according to a first embodiment; Fig. 2 is a block diagram showing an example of a functional configuration of a driving assistance device according to the first embodiment together with peripheral devices; Fig. 3 is a schematic view showing the flow of a process for calculating a prediction error by the driving assistance device according to the first embodiment; Fig. 4 is a diagram showing a correlation between a prediction error relied upon by the driving assistance device according to the first embodiment and the target of a gaze of a subject; Fig. 5 is a schematic view showing the flow of information presentation of the driving assistance device according to the first embodiment; Fig. 6 is a schematic view showing another example of information presentation by the driving assistance device according to the first embodiment; Fig. 7 is a flowchart showing an example of a process of driving assistance processing performed by the driving assistance device according to the first embodiment; Fig. 8 is a schematic view showing an example of a case where a driving assistance device according to a first modification of the first embodiment changes the number of pieces or the amount of presentation information; Fig. 9 is a schematic view showing an example of a case where the driving assistance device according to the first modification of the first embodiment changes a type of presentation information; Fig. 10 is a schematic view showing an example of the case where the driving assistance device according to the first modification of the first embodiment changes a type of presentation information; Fig. 11 is a schematic view showing an example of a case where the driving assistance device according to the first modification of the first embodiment changes a position of presentation information; Fig. 12 is a schematic view showing an example of driving assistance information generated by a driving assistance device according to a second modification of the first embodiment in a case where areas with large prediction errors are scattered in a time series; Fig. 13 is a schematic view showing another example of information presentation by the driving assistance device according to the second modification of the first embodiment in a case where a driver's attention state deviates; Fig. 14 is a schematic view showing another example of information presentation by the driving assistance device according to the second modification of the first embodiment in the case where the driver's attention state deviates; Fig. 15 is a schematic view showing another example of information presentation by the driving assistance device according to the second modification of the first embodiment in the case where the driver's attention state deviates; Fig. 16 is a schematic view showing an example of driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where areas with large prediction errors are located in a time series; Fig. 17 is a schematic view showing an example of driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where areas with large prediction errors are ubiquitous in a time series; Fig. 18 is a schematic view showing an example of driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where the number of areas with large prediction errors in a time series increases or decreases; Fig. 19 is a schematic view showing an example of driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where an area having a large prediction error hardly appears in a time series; Fig. 20 is a schematic view showing another example of presentation in a case where the driving assistance device according to the second modification of the first embodiment suppresses the driver's careless driving; Fig. 21 is a schematic view showing another example of presentation in a case where the driving assistance device according to the second modification of the first embodiment suppresses careless driving of the driver; Fig. 22 is a schematic view showing an example in which a driving assistance device according to a third modification of the first embodiment bisects an image to generate driving assistance information; Fig. 23 is a schematic view showing another example in which the driving assistance device according to the third modification of the first embodiment halves the image to generate driving assistance information; Fig. 24 is a block diagram showing an example of a functional configuration of a driving assistance device according to a fourth modification of the first embodiment, together with peripheral devices; Fig. 25 is a schematic view showing an example of information presentation by the driving assistance device according to the fourth modification of the first embodiment; Fig. 26 is a block diagram showing an example of a functional configuration of a driving assistance device according to a fifth modification of the first embodiment, together with peripheral devices; Fig. 27 is a schematic view showing an example of information presentation by the driving assistance device according to the fifth modification of the first embodiment; Fig. 28 is a schematic view showing another example of information presentation by the driving assistance device according to the fifth modification of the first embodiment; Fig. 29 is a schematic view showing an example of a configuration of a meter display on which the driving assistance device according to the fifth modification of the first embodiment displays latent information; Fig. 30 is a schematic view showing an example of presentation of latent information on the meter display by the driving assistance device according to the fifth modification of the first embodiment; Fig. 31 is a schematic view showing an example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is divided and displayed on a HUD and the meter display; Fig. 32 is a schematic view showing another example in which the presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is divided and displayed on the HUD and the meter; Fig. 33 is a schematic view showing an example of a configuration of a column display on which the driving assistance device according to the fifth modification of the first embodiment presents latent information; Fig. 34 is a schematic view showing an example of presentation of latent information on the column display by the driving assistance device according to the fifth modification of the first embodiment; Fig. 35 is a schematic view showing an example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is divided and displayed on the HUD and the column display; Fig. 36 is a schematic view showing another example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is shared and presented via the HUD and the column display; Fig. 37 is a schematic view showing an example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is shared and presented via the HUD and a speaker; Fig. 38 is a schematic view showing another example in which the presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is divided and presented via the HUD and the speaker; Fig. 39 is a schematic view showing an example of presentation of latent information on an outer peripheral portion of the HUD by the driving assistance device according to the fifth modification of the first embodiment; Fig. 40 is a schematic view showing an example of a configuration of an LED display on which the driving assistance device according to the fifth modification of the first embodiment presents latent information; Fig. 41 is a schematic view showing an example of presentation of latent information on the LED display by the driving assistance device according to the fifth modification of the first embodiment; Fig. 42 is a schematic view showing an example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is presented on the LED display by flashing LEDs; Fig. 43 is a schematic view showing an example in which presentation information generated by the driving assistance device according to the fifth modification of the first embodiment is presented on the LED display by lighting the LEDs in a plurality of colors; Fig. 44 is a schematic view showing an example of a configuration of a mirror display on which the driving assistance device according to the fifth modification of the first embodiment presents latent information; Fig. 45 is a schematic view showing an example of presentation of latent information on the mirror display by the driving assistance device according to the fifth modification of the first embodiment; Fig. 46 is a block diagram showing an example of a functional configuration of a driving assistance device according to a sixth modification of the first embodiment, together with peripheral devices; Fig. 47 is a schematic view showing an example of information presentation by the driving assistance device according to the sixth modification of the first embodiment; Fig. 48 is a schematic view showing another example of information presentation by the driving assistance device according to the sixth modification of the first embodiment; Fig. 49 is a schematic view showing another example of information presentation by the driving assistance device according to the sixth modification of the first embodiment; Fig. 50 is a block diagram showing an example of a functional configuration of a driving assistance device according to a seventh modification of the first embodiment, together with peripheral devices; Fig. 51 is a block diagram showing an example of a functional configuration of a driving assistance device according to a second embodiment together with peripheral devices; Fig. 52 is a flowchart showing an example of a process of driving assistance processing performed by the driving assistance device according to the second embodiment; Fig. 53 is a block diagram showing an example of a functional configuration of a driving assistance device according to a third embodiment together with peripheral devices; and Fig. 54 is a flowchart showing an example of a process of driving assistance processing performed by the driving assistance device according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS

[0008] Hereinafter, embodiments of a driving assistance device, a driving assistance system, and a driving assistance method according to the present disclosure will be described with reference to the drawings. [First embodiment]

[0009] A first embodiment will be described with reference to the drawings. (Example of vehicle configuration)

[0010] Fig. 1 is a block diagram showing an example of the configuration of a vehicle 100 according to the first embodiment. Fig. 1 shows various vehicle-side devices mounted in the vehicle 100.

[0011] As in Fig. 1, the vehicle 100 according to the first embodiment includes a driving assistance device 10, an electronic control unit (ECU) 20, a human machine interface (HMI) controller 30, a detection device 40, a vehicle control device 50, and an information presentation device 60.

[0012] These on-board devices included in the vehicle 100 are interconnected by an on-board network such as a Controller Area Network (CAN) so that they are able to send and receive information between each other.

[0013] The driving assistance device 10 is configured as a computer including, for example, a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage device 14, and an input / output (I / O) port 15. The driving assistance device 10 may be one of the electronic control units installed in the vehicle 100.

[0014] The CPU 11 controls the entire driving assistance device 10. The ROM 12 serves as the storage area within the driving assistance device 10. The information stored in the ROM 12 is retained even when the driving assistance device 10 is turned off. The RAM 13 functions as the primary memory and serves as the work area of ​​the CPU 11.

[0015] The CPU 11 loads, for example, a control program stored in the ROM 12 or the like into the RAM 13 and executes the control program, thereby realizing various functions of the driving assistance device 10, which are described in more detail below.

[0016] The storage device 14 is a hard disk drive (HDD), a solid state drive (SSD), or the like, and functions as an auxiliary storage device of the CPU 11. The I / O port 15 is configured to be capable of transmitting and receiving various types of information to and from, for example, the HMI control device 30, a vehicle exterior camera 41 to be described later, and the like.

[0017] In such a configuration, the driving assistance device 10 presents information to a driver of the vehicle 100 via the HMI control device 30 and the information presentation device 60 to draw attention to at least a specific area in a traveling direction of the vehicle 100, and assists the driver in driving the vehicle 100.

[0018] The ECU 20 is configured as a computer, such as an in-vehicle electronic unit including, for example, a CPU, a ROM, a RAM, and the like (not shown). The ECU 20 receives a detection result from the detection device 40, which detects a state of each part of the vehicle 100. Furthermore, based on the received detection result, the ECU 20 transmits various commands to the vehicle control device 50 to control the operation of the vehicle 100.

[0019] The HMI control device 30 as an information presentation control device is configured as a computer including, for example, a CPU, a ROM, a RAM, and the like (not shown). The HMI control device 30 may include a graphics processing unit (GPU) instead of or in addition to the CPU. The HMI control device 30 controls the information presentation device 60, which presents various types of information to the driver of the vehicle 100, to present various types of information output from the driving assistance device 10.

[0020] The detection device 40 includes the vehicle exterior camera 41, a driver monitor camera 42, a vehicle speed sensor 43, an acceleration sensor 44, a brake sensor 45, a steering angle sensor 46, and the like, detects a state of each part of the vehicle 100, and transmits the state to the ECU 20.

[0021] The vehicle exterior camera 41 and the driver monitoring camera 42 are digital cameras, each containing an imaging element such as a charge coupled device (CCD) or a CMOS image sensor (CIS).

[0022] The vehicle exterior camera 41 captures an image of the periphery of the vehicle 100. A plurality of vehicle exterior cameras 41, each capturing images of the front, rear, side, and the like of the vehicle 100, may be mounted on the vehicle 100. The vehicle exterior camera 41 transmits image data obtained by capturing at least the traveling direction of the vehicle 100 to the driving assistance device 10.

[0023] The driver monitoring camera 42 is mounted in the interior of a passenger compartment of the vehicle 100 and captures an image of a condition within the passenger compartment, e.g., the driver's face.

[0024] The vehicle speed sensor 43 detects the speed of the vehicle 100 based on the rotational speed of the vehicle's wheels. The acceleration sensor 44 detects the driver's actuation of an accelerator pedal. The brake sensor 45 detects the driver's actuation of the brake pedal. The steering angle sensor 46 detects the driver's actuation of a steering wheel, i.e., a steering angle.

[0025] The vehicle control device 50 includes a brake actuator 51 and an engine controller 52, and performs an operation for avoiding danger to the vehicle 100 by decelerating the vehicle 100 or the like according to a command from the ECU 20.

[0026] The brake actuator 51 brakes the wheels of the vehicle 100 based on a detection result of the brake sensor 45 during normal driving. The engine controller 52 performs engine power control based on a detection result of the acceleration sensor 44 during normal driving and performs acceleration / deceleration control of the vehicle 100.

[0027] For example, the vehicle control device 50 controls the brake actuator 51 according to a command from the ECU 20 to decelerate the vehicle 100 and thereby avoid a collision between the vehicle 100 and an obstacle or the like. Alternatively or additionally, the vehicle control device 50 causes the engine controller 52 to reduce the power of the engine for, for example, several seconds to reduce the acceleration of the vehicle 100 and thereby avoid the collision between the vehicle 100 and the obstacle or the like.

[0028] The information presentation device 60 includes a head-up display (HUD) 61, an in-vehicle monitor 62, a speaker 63, and the like in the passenger compartment of the vehicle 100, and presents various types of information to the driver according to a command from the HMI control device 30.

[0029] The HUD 61 projects a speed, gear position, driving instructions, warnings and the like onto a windshield in front of the driver's seat.

[0030] The in-vehicle monitor 62 is, for example, an in-dash monitor or an on-dash monitor configured as a liquid crystal display (LCD), an organic electroluminescence (EL) display, or the like. The in-vehicle monitor 62 displays an image of the periphery of the vehicle 100, driving guidance, a warning, and the like.

[0031] The loudspeaker 63 is installed, for example, in a dashboard or the like and presents the surroundings of the vehicle 100, driving instructions, a warning or the like to the driver via audio.

[0032] With such a configuration, the information presentation device 60 presents various types of information to the driver via video, audio, and the like. The information presentation device 60 may also include a display other than the one mentioned above, such as a light-emitting diode (LED) display. Furthermore, the information presentation device 60 may be configured to present information to the driver through vibration or the like, for example, through a vibrator mounted on the steering wheel, accelerator pedal, brake pedal, seat, headrest, seat belt, or the like. (Example of the driver assistance device configuration)

[0033] Hereinafter, a detailed configuration of the driving assistance device 10 of the first embodiment will be described with reference to Fig. 2 described.

[0034] Fig. 2 is a block diagram showing an example of a functional configuration of the driving assistance device 10 according to the first embodiment together with peripheral devices. As shown in Fig. 2, the driving assistance device 10 includes a prediction error calculation unit 110, a driver attention state estimation unit 120, and an output control unit 130 as functional configurations.

[0035] The prediction error calculation unit 110 generates a predicted image obtained by predicting the future based on an image of the periphery of the vehicle 100 captured by the vehicle exterior camera 41, e.g., the image in the traveling direction of the vehicle 100. In addition, an actual image at a time of the predicted image is captured by the vehicle exterior camera 41, and an error between the predicted image and the actual image is calculated as a prediction error.

[0036] Such a function of the prediction error calculation unit 110 is implemented, for example, by the CPU 11 executing the control program.

[0037] The driver attention state estimation unit 120 estimates the driver's attention state based on the prediction error calculated by the prediction error calculation unit 110. At this time, the driver attention state estimation unit 120 divides the image captured by the vehicle exterior camera 41 into a plurality of regions and estimates how easily each region attracts the driver's attention. As a result, data indicating the driver's attention state is obtained for each region.

[0038] Such a function of the driver attention state estimation unit 120 is implemented, for example, by the CPU 11 executing the control program.

[0039] The output control unit 130 determines an area to which the driver's attention should be directed from the plurality of areas in the image based on the driver's attention state estimated by the driver attention state estimation unit 120. Furthermore, the output control unit 130 outputs to the HMI control unit 30 driving assistance information containing information about the specific area to which attention should be directed. The driving assistance information includes, for example, presentation information to be presented to the driver, a presentation mode, and the like related to the area to which attention should be directed.

[0040] Such a function of the output control unit 130 is realized, for example, by the CPU 11 executing the control program and the I / O port 15 operating under the control of the CPU 11.

[0041] The HMI control device 30 selects a device that causes the driver to present information from the HUD 61, the in-vehicle monitor 62, the speaker 63, and the like included in the above-described information presentation device 60 based on the driving assistance information output from the driving assistance device 10, and transmits a command including an area in which information is to be presented, the presentation mode, and the like from the plurality of areas in the image to the selected information presentation device 60.

[0042] As a result, the information presentation device 60, for example, presents information in a predefined mode in the area to which attention is to be drawn from among the plurality of areas in the image. That is, in a case where the HUD 61 or the in-vehicle monitor 62 of the information presentation device 60 is an object through which information is presented, the HUD 61 or the in-vehicle monitor 62 displays the image of the vehicle exterior camera 41 and superimposes and displays the presentation information in the predefined area of ​​the image. Furthermore, in a case where the speaker 63 is an object through which information is presented, the speaker 63 of the information presentation device 60 outputs an announcement warning to pay attention to the predefined area of ​​the vehicle 100, such as the traveling direction of the vehicle 100, or a sound, such as a warning sound, to the interior of the vehicle. (Example of generating driver assistance information)

[0043] Next, an example of the generation of driving assistance information by the driving assistance device 10 according to the first embodiment will be described with reference to FIG. Fig. 3 to 5.

[0044] Fig. 3 is a schematic view showing the flow of a process for calculating a prediction error by the driving assistance device 10 according to the first embodiment. As shown in Fig. 3, the prediction error calculation unit 110 of the driving assistance device 10 generates a predicted image based on a prediction model “PredNet”.

[0045] Here, "PredNet" is a prediction model that mimics predictive coding processing in the cerebral cortex and is created, for example, in the context of deep learning. "PredNet" is described in detail in a paper such as "Lotter, W., Kreiman, G., and Cox, D., "Deep predictive coding networks for video prediction and unsupervised learning," https: / / ar-xiv.org / abs / 1605.08104."

[0046] Specifically, when images for a plurality of frames, such as 20 frames, captured by the vehicle exterior camera 41 are provided, the prediction error calculation unit 110 generates a predicted image corresponding to a future frame for the images for the plurality of frames based on the prediction model “PredNet”.

[0047] That is, if, for example, images from time t -20 until time t -1are provided, the prediction error calculation unit 110 generates a predicted image of a future frame at time t0 based on the prediction model "PredNet". Similarly, the prediction error calculation unit 110 generates a predicted image of a future frame at time t1 from the images at time t -19 until time t -0 . Similarly, the prediction error calculation unit 110 generates a predicted image of a future frame at time t2 from the images at time t -18 until time t -1 .

[0048] As described above, the prediction error calculation unit 110 generates predicted images of future frames for all frames using frames whose time is shifted frame by frame. Note that the number of frames used to generate one predicted image can be any number of frames, such as 30 frames.

[0049] The prediction error calculation unit 110 uses an actual image captured by the vehicle exterior camera 41 at the time of a generated prediction image as a correct image, compares the generated prediction image with the correct image in pixel units, and generates a prediction error image based on a difference between the respective pixel values ​​of the two images.

[0050] Fig. 3 shows an example in which a prediction error image is generated at time t0 based on a difference between the pixel values ​​of the predicted image at time t0 and an actual image at time t0, which is the correct image.

[0051] Furthermore, the prediction error calculation unit 110 calculates pixel values ​​of the prediction error image and sets the values ​​as prediction error values. Furthermore, the prediction error calculation unit 110 divides the entire image area of ​​the generated prediction image into a plurality of regions and calculates a sum, average, or the like of the prediction errors for each region.

[0052] Here, a value related to the prediction error, e.g., the sum or average of the prediction errors of each individual area, is an index indicating which area is more likely to attract the driver's attention than the other areas.

[0053] Fig. Fig. 4 is a graph showing a correlation between a prediction error, which the driving assistance device 10 according to the first embodiment relies on, and a target of a gaze of a subject. The horizontal axis of the Fig. The graph shown in Figure 4 represents a ratio at which a prediction error of a randomly selected pixel is equal to or greater than a threshold. The vertical axis of the graph represents a ratio at which a prediction error of the target of the participant's gaze is equal to or greater than the threshold. The graph in Fig. 4 can be determined as follows.

[0054] First, a prediction error value of a pixel that is a target of the driver's gaze point in a frontal view video viewed by the driver while driving is acquired. Furthermore, a prediction error value of a pixel randomly extracted from the same video is acquired. Furthermore, a threshold for the prediction errors is changed from the minimum value to the maximum value to calculate ratios at which the acquired prediction errors are equal to or greater than the threshold.

[0055] The ratio at which the prediction error of the randomly selected pixel is equal to or greater than the threshold is plotted on the horizontal axis of the graph, and the ratio at which the prediction error of the target of the participant's gaze point is equal to or greater than the threshold is plotted on the vertical axis of the graph, thereby obtaining the Fig. 4. As a result, the correlation between the prediction error and the destination of the driver’s gaze point is calculated using the area under the curve (AUC) value of a function as shown in Fig. 4 drawn curve.

[0056] Here, it is assumed that attention tends to be drawn to an area where the prediction error is large when the AUC value is greater than 0.5. Similarly, it is assumed that there is no correlation between them when the AUC value is 0.5.

[0057] The driver attention state estimation unit 120 of the driving assistance device 10 is configured to estimate the driver's attention state based on the above evaluation. That is, the driver attention state estimation unit 120 estimates an area likely to attract the driver's attention based on a prediction error value, a variance value, or the like.Specifically, the driver attention state estimation unit 120 estimates, as the driver's attention state, a state in a case where the area likely to attract the driver's attention or an area less likely to attract the driver's attention, the area estimated from the prediction error, deviates to the left, right, or the like with respect to a central part of the image, is excessively concentrated in one part, or is excessively scattered over the entire image, or the like. As described above, the driver attention state estimation unit 120 estimates the driver's attention state not only based on the magnitude of the prediction error value, the variance value, or the like, but also based on the spatial arrangement of the prediction error or the like.

[0058] As an actual process, the driver attention state estimation unit 120 divides the entire image area of ​​the image of the vehicle exterior camera 41 into a plurality of regions similar to the plurality of regions of the predicted image, and estimates the driver's attention state in the plurality of regions. That is, for example, the larger the prediction error of a target region, the more likely the region is to attract the driver's attention. Furthermore, the driver attention state estimation unit 120 also estimates the driver's attention states with the spatial arrangement of the prediction error added as a determination condition of the driver's attention state as described above.

[0059] The output control unit 130 of the driving assistance device 10 determines an area to which the driver's attention should be directed, or the like, according to the driver's attention state in each of the areas. That is, when it is estimated that the driver's attention state is in an unsuitable state due to a deviation or the like in the driver's attention state based on the values ​​of the prediction errors or the variance value, it is assumed that there is an area to which the driver's attention is low. Therefore, the output control unit 130 extracts an area that is assumed to attract the driver's low attention as an area to which the driver's attention should be directed, or as an area to which attention should be directed to restore the driver's attention to an appropriate state.

[0060] As described above, there is a possibility that a latent danger that has not yet manifested is contained in the area to which the driver is not paying sufficient attention, and it is necessary to arouse the driver's attention or prompt a correction of attention. For example, if an event that has already occurred, such as a pedestrian jumping onto the roadway, is defined as a danger manifested at this time, a situation in which the driver unconsciously focuses their attention on a fence at the left end of the roadway, a vehicle parked on the street, a bicycle parked on the sidewalk, or the like, may prevent the driver from quickly responding to a vehicle approaching from the right or the like and may present a latent danger. The driving assistance device 10 can also display information to the driver indicating such a latent danger.Hereinafter, the information given to the driver by the driving assistance device 10 is also referred to as latent information or the like.

[0061] Fig. 5 shows examples of displaying latent information on the HUD 61 and the in-vehicle monitor 62 based on the above.

[0062] Fig. 5 is a schematic view showing the flow of an information presentation process of the driving assistance device 10 according to the first embodiment. Fig. 5(Aa) to 5(Ab) show the display example of the latent information on the HUD 61, and Fig. 5(Ba) to 5(Bb) show the display example of the latent information on the vehicle monitor 62.

[0063] It should be noted that Fig. 5 is an example of a case where information is presented at least on the HUD 61 and the in-vehicle monitor 62. However, the information may be presented in different modes on the HUD 61 and the in-vehicle monitor 62, or different information may be displayed. That is, when information is presented on both the HUD 61 and the in-vehicle monitor 62, the display contents of the HUD 61 and the in-vehicle monitor 62 may be different.

[0064] As in Fig. As shown in Figs. 5(Aa) and 5(Ba), the driver attention state estimation unit 120 divides the image of the vehicle exterior camera 41 into a plurality of regions, e.g., 5 x 5 regions in height and width, ie, a total of 25 regions, and estimates the driver's attention state based on the sum of the values ​​of the prediction errors in each of the regions.

[0065] In Fig. 5(Aa) and 5(Ba) are the shaded areas that have a high prediction error and are likely to attract the driver's attention. That is, in the example of Fig. For example, 5(Aa) and 5(Ba) are areas where a parent accompanying the child on the near left side of the image, two people on the far left side of the image, and a person walking on a sidewalk on the near right side of the image are areas where the driver's attention state is estimated to be deviant.

[0066] However, since the driver's attention state is estimated by the internal operation of the driver attention state estimation unit 120, Fig. 5(Aa) and 5(Ba) do not show actual display examples for the HUD 61 and the in-vehicle monitor 62.

[0067] The driver attention state estimation unit 120 determines values ​​of prediction errors, a variance value, and the like based on one or more threshold values ​​to estimate the driver's attention state, and accordingly estimates a state in which an area likely to attract the driver's attention or an area less likely to attract the driver's attention deviates to the left, right, or the like with respect to the central part of the image, is excessively concentrated in one part, is excessively scattered in the entire area, or the like.For example, the prediction error is sorted into low, medium, high, or the like according to the probability of attracting the driver's attention based on the three thresholds, and a deviation, concentration degree, dispersion degree, or the like of the driver's attention is matched with low, medium, high, or the like.

[0068] The output control unit 130 determines an area to which the driver's attention should be drawn based on the driver's attention state estimated by the driver attention state estimation unit 120. That is, areas on the right side of the image opposite to the areas including the parent accompanying the child on the near left side and the two people on the far left side that easily attract the driver's attention, areas on the left side of the image opposite to the areas including the person walking on the sidewalk on the near right side of the image, and the like can be areas to which the driver's attention should be drawn.

[0069] The output control unit 130 outputs to the HMI controller 30 the presentation information to be displayed in the area on the right side of the image and the area on the left side of the image, which are opposite to the areas likely to attract the driver's attention, as well as driving assistance information including the presentation mode and the like. At this time, in the presentation mode, the output control unit 130 may include information about a magnitude of an intensity of attentional appeal of the presentation information to be displayed in each area based on the deviation, degree of concentration, degree of dispersion, or the like of the driver's attention, which are estimated according to the values ​​of the prediction errors, the variance value, or the like.

[0070] As in Fig. 5(Ab) and 5(Bb), the HMI control device 30 causes the HUD 61 or the in-vehicle monitor 62 to display information presented to the driver based on the driving assistance information output from the driving assistance device 10.

[0071] In the example of Fig. 5(Ab) and 5(Bb), the HUD 61 or the in-vehicle monitor 62 displays, for example, latent information 601 to 603 and an arrow 604 as presentation information. The latent information 601 to 603 includes, for example, circles indicating a pedestrian, and the like. The arrow 604, for example, indicates the front of the vehicle 100 and indicates the traveling direction of the vehicle 100.

[0072] In addition, the latent information 601 to 603 successively increases in size, ie the diameter of the circle in the example of Fig. 5(Ab) and 5(Bb). This indicates that deviations or the like of the driver's attention, estimated from the prediction errors in areas opposite to the display areas of the pieces of latent information 601 to 603, increase in order.

[0073] That is, it is assumed that the driver's attention is easily attracted to the areas including the parent accompanying the child on the near left side of the image, the areas opposite to the areas where the latent information 601 indicating the person on the near right side of the image is displayed, and that the driver's attention is hardly attracted to the areas including the person on the near right side of the image, the areas opposite to the areas where the latent information 603 indicating the parent accompanying the child is displayed. Moreover, the driver's attention is moderately strong in the leftmost areas of the image compared to the areas where the latent information 602 indicating the two leftmost people in the image is displayed.

[0074] In practice, the driver may be attracted to the parent accompanying the child on the left and pay little attention to the person on the right. In this regard, it is possible to attract the driver's attention more by displaying latent information 501 indicating that the person on the right is relatively tall.

[0075] As described above, the deviations of the driver's attention or the like are estimated according to the prediction errors of the respective areas, and the sizes of the pieces of the latent information 601 to 603 displayed in the areas opposite to such deviations are changed, so that it is possible to adjust an effect of distributing the driver's attention.

[0076] Note that the output control unit 130 may change a brightness, a color, or the like of each piece of the latent information 601 to 603 in accordance with the deviation, the concentration degree, the dispersion degree, or the like of the driver's attention instead of or in addition to the size of each piece of the latent information 601 to 603, thereby adjusting the effect of distributing the driver's attention.

[0077] In addition, the arrow 604 indicating the direction of travel of the vehicle 100 can draw the attention of the driver, who tends to orient himself to the left and right of the direction of travel of the vehicle 100, to the direction of travel of the vehicle 100.

[0078] It should be noted that even if one or more thresholds for the deviation, concentration level, dispersion level, or the like of the driver's attention are provided as described above, there may be a case where the deviation, concentration level, dispersion level, or the like of the driver's attention with a magnitude exceeding the thresholds is not included in the image. An example of the display in such a case is shown in Fig. 6 shown.

[0079] Fig. 6 is a schematic view showing another example of information presentation by the driving assistance device 10 according to the first embodiment. Fig. 6(a) is a display example of the latent information on the HUD 61, and Fig. 6(b) is a display example of the latent information on the vehicle monitor 62.

[0080] In the example of Fig. 6, it is assumed that only the deviation of the driver's attention corresponding to a middle value between low, medium and high is included in the image, e.g. in a case where the deviation, concentration degree, dispersion degree or the like of the driver's attention has been determined based on the three thresholds.

[0081] As in Fig. In such a case, as shown in Figure 6, the driving assistance device 10 causes the HUD 61 or the in-vehicle monitor 62, via the HMI control device 30, to display the latent information 602 indicating the two people in areas opposite to the areas where the driver's attention error magnitude is medium, that is, the areas containing the two people on the far left of the image. At this time, the driving assistance device 10 can output to the HMI control device 30 driving assistance information including the latent information 602 with a size corresponding to the driver's attention error magnitude.

[0082] Note that, also in this case, the output control unit 130 may change the brightness, color, or the like of each piece of the latent information 601 to 603 in accordance with the direction of the driver's attention instead of or in addition to the size of each piece of the latent information 601 to 603, thereby adjusting the effect of distributing the driver's attention.

[0083] For example, as described above, even if not all pieces of the latent information 601 to 603 are displayed, it is advantageous to display each of the latent information 601 to 603 with an intensity corresponding to the driver's attention state.

[0084] Furthermore, the driving assistance device 10 can hide the latent information, for example, when the vehicle 100 is traveling at a low speed or when the vehicle 100 is stopped. This can prevent the driver from being bothered by the display of unnecessary information.

[0085] It should be noted that a case where it is estimated that the driver's attention deviates based on the values ​​of prediction errors, variance value, or the like, in the cases described above, Fig. 5 and Fig. 6. However, as described above, the driver's attention state also includes the concentration level, the dispersion level, and the like of the driver's attention in addition to the case where the attention deviates, and the driving assistance device 10 can comprehensively recognize a case where such a driver's attention state is estimated to be an inappropriate state.

[0086] Therefore, for example, in a case where the driver's attention is estimated to be excessively concentrated on one point or the like based on the values ​​of the prediction errors, the variance value, and the like, the driving assistance device 10 can perform information presentation that causes the driver to focus their attention on the entire periphery of the vehicle 100 or the entire image. Furthermore, in a case where the driver's attention is estimated to be excessively scattered, for example, based on the values ​​of the prediction errors, the variance value, and the like, the driving assistance device 10 can perform information presentation that causes the driver's attention to be returned to the traveling direction of the vehicle 100 or the central part of the image.

[0087] When the area likely to attract the driver's attention or the area less likely to attract the driver's attention, the area being estimated based on the prediction error, deviates from the central part of the image, is excessively concentrated in one part, or is excessively dispersed over the entire image, the driving assistance device 10 of the first embodiment presents information to draw the driver's attention to an area opposite to such an area.

[0088] Furthermore, in a case where information is presented through the speaker 63 or the like, the information can be presented to the driver with an intensity corresponding to the degree of deviation, degree of concentration, degree of dispersion, or the like of the driver's attention by adjusting the volume of the sound such as an announcement or a warning sound. Furthermore, in a case where information is presented through the vibrator or the like provided on the steering wheel, accelerator pedal, brake pedal, seat, headrest, seat belt, or the like, it is possible to present the information to the driver with an intensity corresponding to the degree of deviation, degree of concentration, degree of dispersion, or the like of the driver's attention by adjusting the intensity of the vibration. (Example of processing of the driver assistance device)

[0089] Next, an example of the processing of the driving assistance in the driving assistance device 10 of the first embodiment will be explained with reference to Fig. 7 described. Fig. 7 is a flowchart showing an example of a procedure of driving assistance processing performed by the driving assistance device 10 according to the first embodiment.

[0090] As in Fig. As shown in Figure 7, the prediction error calculation unit 110 of the driving assistance device 10 generates predicted images corresponding to a plurality of images based on the respective images and divides each of the predicted images into a plurality of regions (step S110). Furthermore, the prediction error calculation unit 110 calculates, for each of the plurality of regions, a value related to the prediction error, such as a sum or an average of the prediction errors, which can be used to estimate the driver's alertness state (step S120).

[0091] The driver attention state estimation unit 120 similarly divides an image captured by the vehicle's exterior camera into a plurality of areas and determines whether or not a degree to which the driver's attention state is not appropriate in a predefined area is higher than a threshold TH1 based on the value related to the prediction error (step S130).

[0092] If the prediction error is greater than the threshold TH1 (step S130: Yes), the driver attention state estimation unit 120 estimates that the degree of inappropriateness of the attention state is high, for example, because the driver's attention to the area is deviated, overly concentrated, or overly scattered (step S132). The output control unit 130 extracts an area opposite the area and selects a mode ST1 with a high attention intensity, which is called the mode of presentation information to be displayed in the extracted area (step S133).

[0093] If the prediction error is equal to or less than the threshold TH1 (step S130: No), the driver attention state estimation unit 120 determines whether the degree to which the driver's attention state is inappropriate in the area is higher than a threshold TH2 or not based on the value related to the prediction error (step S140). A value less than the threshold TH1 is set as the threshold TH2.

[0094] If the prediction error is greater than the threshold TH2 (step S140: Yes), the driver attention state estimation unit 120 estimates that the degree of inappropriateness of the driver's attention state for the area is medium (step S142). The output control unit 130 extracts an area opposite the area and selects a mode ST2 with a medium attention intensity, which is called the mode of presentation information to be displayed in the extracted area (step S143).

[0095] If the prediction error is equal to or less than the threshold TH2 (step S140: No), the driver attention state estimation unit 120 determines whether the degree to which the driver's attention state is inappropriate in the area is higher than a threshold TH3 based on the value related to the prediction error (step S150). A value less than the threshold TH2 is set as the threshold TH3.

[0096] If the prediction error is greater than the threshold TH3 (step S150: Yes), the driver attention state estimation unit 120 estimates that the degree of inappropriateness of the driver's attention state for the area is low (step S152). The output control unit 130 extracts an area opposite the area and selects a mode ST3 with a low attention intensity, which is called the mode of presentation information to be displayed in the extracted area (step S153).

[0097] When the prediction error is equal to or smaller than the threshold TH3 (step S150: No), the driver attention state estimation unit 120 determines that the driver's attention state between the area and an area opposite to the area is not in an inappropriate state so that driving of the vehicle 100 may be hindered, and proceeds to processing of step S160.

[0098] After these operations, the driving assistance device 10 determines whether or not the processing for all the divided areas is completed (step S160). If there is an unprocessed area (step S160: No), the driving assistance device 10 repeats the processing from step S120. If the processing for all areas is completed (step S160: Yes), the output control unit 130 outputs the generated driving assistance information to the HMI control device 30 (step S170).

[0099] Thus, the driving assistance processing by the driving assistance device 10 of the first embodiment is completed. (Overview)

[0100] According to the driving assistance device 10 of the first embodiment, the driver's attention state is estimated for each of the plurality of areas in the image based on the prediction error that is the difference between the predicted image predicted from the image in the traveling direction of the vehicle 100 captured by the vehicle exterior camera 41 and the actual image in which an actual situation is captured, and the driving assistance information including the information regarding the area to which the driver's attention should be drawn among the plurality of areas in the image is output based on the estimated attention state.

[0101] Consequently, it is possible to grasp an area where a cognitive error is likely to occur even if the driver's attention function is normal, an area where a cognitive error is likely to occur even if the information is presented in a way that is easy to recognize, and the like as the area to which the driver's attention should be directed or the area to which attention should be distributed in order to return the driver's attention to a suitable state.

[0102] According to the driving assistance device 10 of the first embodiment, an area where the prediction error is greater than a predefined threshold is extracted from the plurality of areas in the image, and at least one state in which the extracted area deviates from the center of the plurality of areas, is excessively concentrated in one part, and / or is excessively scattered throughout the area is estimated. Furthermore, when it is estimated that the driver's attention state corresponds to any of the above-mentioned states, an area opposite the extracted area is extracted as the area to which the driver's attention should be directed. This makes it possible to extract the area opposite the area where the degree of inappropriateness of the driver's attention state is high as the area where the driver is likely to make a cognitive error.

[0103] According to the driving assistance device 10 of the first embodiment, information such as latent information 601 to 603 and arrow 604 associated with the driver attention-requesting areas and presented to the driver is output to the HMI control device 30, and the latent information 601 to 603, arrow 604, and the like are presented to the information presentation device 60 in association with the attention-requesting areas. This makes it possible to draw the driver's attention to an area to which the driver tends to pay little attention and in which a cognitive error may occur.

[0104] According to the driving assistance device 10 of the first embodiment, when the degree of inappropriateness of the attention state in the area to which the driver's attention is to be directed is higher, information such as latent information 601 to 603 that has a higher intensity for attracting the driver's attention or that has a higher effect on directing the driver's attention to an appropriate state is output. Therefore, it is possible to cause the information presentation device 60 to present the latent information 601 to 603 or the like by adjusting the intensity of the driver's attention request according to the degree of inappropriateness of the driver's attention state. (First modification)

[0105] Next, a driving assistance device according to a first modification of the first embodiment will be described with reference to FIG. Fig. 8 to 11. The driving assistance device of the first modification differs from the above-described first embodiment in that the number of pieces, a quantity, a position, a type, and the like of the information to be presented by the information presentation device 60 are changed.

[0106] Fig. 8 is a schematic view showing an example in which the driving assistance device according to the first modification of the first embodiment changes the number of pieces or the amount of presentation information.

[0107] As in Fig. 8(a) and Fig. 8(b), the driving assistance device according to the first modification causes the HUD 61 or the in-vehicle monitor 62 to display only the latent information 603 based on the highest degree of inappropriateness among the latent information 601 to 603 based on a magnitude of the degree of inappropriateness of the driver's attention state classified by, for example, the plurality of threshold values ​​TH1 to TH3.

[0108] This makes it possible to preferentially display information such as latent information 603 in an area where the driver's attention is extremely low and the risk is higher. This prevents a multitude of information from being presented that distracts the driver's attention and distributes the driver's attention to a more important part.

[0109] Fig. 9 and Fig. 10 are schematic views showing examples of a case where the driving assistance device according to the first modification of the first embodiment changes a type of presentation information.

[0110] As in Fig. 9(a) and Fig. As shown in Fig. 9(b), the driving assistance device according to the first modification causes the HUD 61 or the in-vehicle monitor 62 to display a message 614 that attracts attention, e.g., “Slow down,” instead of the arrow 604 indicating the traveling direction of the vehicle.

[0111] As in Fig. 10(a) and Fig. As shown in Figure 10(b), the driving assistance device according to the first modification may cause the speaker 63 to output an announcement 624 to attract attention, e.g., "Slow down" instead of the arrow 604 indicating the vehicle's direction of travel. The presentation information output via the speaker 63 may be a warning tone or the like.

[0112] Since the configurations of the Fig. 9 and Fig. 10 presentation information of a different type than the latent information 601 to 603 is presented, the driver's attention is distributed more reliably among the individual pieces of information. Furthermore, since the information is presented to the driver with verbal information such as message 614 or announcement 624 depending on the content of the attention request, the driver can grasp the information more accurately.

[0113] Fig. 11 is a schematic view showing an example of a case where the driving assistance device according to the first modification of the first embodiment changes a position of presentation information.

[0114] As in Fig. As shown in Figures 11(Aa) and 11(Ba), it is assumed that a plurality of areas where the degree of inappropriateness of the driver's attention state is high are extracted by the driving assistance device of the first modification. The driving assistance device of the first modification determines the driver's recognizability based on the extracted areas and preferentially presents information in an area that is more difficult for the driver to recognize and where a cognitive error is more likely to occur.

[0115] For example, the driver's recognizability may be determined in advance for each of a plurality of divided regions of an image. Alternatively, the driver's recognizability may be appropriately determined by the driving assistance device of the first modification according to a road situation in the direction of travel of the vehicle, or the like, that is detected from the image at that time.

[0116] In a Fig. 11(Ab) and 11(Bb), the driving assistance device according to the first modification determines that areas including a person on the near right side are areas that are more difficult for the driver to recognize than areas including a parent accompanying a child among the plurality of extracted areas, and displays latent information 612 indicating the person on the near right side.

[0117] In a Fig. 11(Ac) and 11(Bc), the driving assistance device according to the first modification determines that areas including the parent accompanying the child on the near left side and two people on the far left side are more difficult for the driver to recognize than the areas including the person on the near right side among the plurality of extracted areas, and displays the latent information 612 indicating the parent accompanying the child and the people.

[0118] As a result, the presentation information such as latent information 612 can be preferentially presented in an area where the driver is more likely to make a cognitive error. Therefore, the driver's cognitive error can be further suppressed.

[0119] With the driving assistance device of the first modification, in addition to the effects described above, similar effects to those of the driving assistance device 10 of the first embodiment are achieved. (Second modification)

[0120] Next, a driving assistance device according to a second modification of the first embodiment will be described with reference to FIG. Fig. 12 to 21. The driving assistance device of the second modification differs from the first embodiment described above in that the driving assistance information is generated based on time series data.

[0121] Fig. 12 is a schematic view showing an example of the driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where areas where a degree of attracting the driver's attention is high are scattered in the time series.

[0122] Fig. 12(Aa) to 12(Ac) show time series data referred to when the driving assistance device of the second modification generates the driving assistance information. Fig. 12(Aa) to 12(Ac) are shaded areas where the driver's attention level is estimated to be high. However, this does not mean that the Fig. 12(Aa) to 12(Ac) are actually displayed on the HUD 61 or the like.

[0123] In a Fig. In the example shown in Figure 12(Aa), areas where the degree of driver attention is estimated to be high are extracted on the left side of the image by the driving assistance device of the second modification. In an example shown in Fig. In the example presented in Figure 12(Ab), areas where the driver's attention is attracted are extracted on the right side of the image by the driving assistance device of the second modification. Fig. In the example shown in Fig. 12(Ac), regions where a degree of attracting the driver's attention is high are extracted by the driving assistance device of the second modification on the near right side of the image.

[0124] As described above, in the examples of Fig. 12(Aa) to 12(Ac) the areas where a degree is high to attract the driver's attention are scattered in time series.

[0125] Fig. 12(Ba) to 12(Bc) show a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. That is, the Fig. 12(Ba) to 12(Bc) show processes that occur at the same times as those in the Fig. The images shown in Figures 12(Aa) to 12(Ac) can be used.

[0126] As in Fig. 12(Ba), the driving assistance device according to the second modification causes the latent information 622 to be displayed in areas on the right side opposite to the areas extracted on the left side of the image in which a degree of attracting the driver's attention is high, in accordance with a display timing of the image in Fig. 12(Aa).

[0127] As in Fig. 12(Bb), the driving assistance device according to the second modification causes the latent information 622 to be displayed in areas on the left side opposite to the areas extracted on the right side of the image in which a degree of attracting the driver's attention is high, in accordance with a display timing of the image in Fig. 12(Ab).

[0128] As in Fig. 12(Bc), the driving assistance device according to the second modification causes the latent information 622 to be displayed in areas on the near left side opposite to the areas extracted on the near right side of the image, in which a degree of attracting the driver's attention is high, in accordance with a display timing of the image in Fig. 12(Ac).

[0129] In a case where the areas where the driver's attention is high are scattered in the time series, the driving assistance device according to the second modification appropriately presents information in areas opposite to these areas as described above.

[0130] It should be noted that in the examples of Fig. 12(Ba) to 12(Bc), for example, the latent information 622 is displayed in a case where the driver's attention state deviates in the areas of the image. However, information other than the above may also be displayed when the driver's attention state deviates.

[0131] Fig. 13 to 15 are schematic views showing another example of information presentation by the driving assistance device according to the second modification of the first embodiment in the case where the driver's attention state deviates.

[0132] In a Fig. In the example shown in Fig. 13(a), an area where a degree of attracting the driver's attention is high is extracted on the right side of an image by the driving assistance device of the second modification.

[0133] As in Fig. 13(b), the driving assistance device according to the second modification may display, instead of the latent information 622, a triangular icon 622a in a region near the left of the region extracted on the near right side of the image, in which a degree of attracting the driver's attention is high, the icon being displayed on the region in accordance with a display timing of the image in Fig. 13(a). However, the presentation information pointing to the area may also have a different shape or mode, such as an arrow.

[0134] In a Fig. In the example shown in Fig. 14(Aa), an area where a degree of attracting the driver's attention is high is extracted on the near left side of an image by the driving assistance device of the second modification.

[0135] As in Fig. As shown in Figure 14(Ab), if the area where the driver's attention is high is on the left side of the image, it is assumed that the driver's effective field of view (VF) also deviates to the left side of the vehicle when the driver looks at that area. In this case, for example, the vicinity of the center of a road in front of the vehicle is considered to be outside the driver's effective field of view.

[0136] As in Fig. 14(Ba), the driving assistance device of the second modification may, instead of the latent information 622, in accordance with a display timing of the image of Fig. 14(Aa) display straight lines 622b extending from the front to the back on the left and right sides of the center of a screen.

[0137] As in Fig. 14(Bb), causes the display of lines 622b and the like in Fig. 14(Ba) the driver to direct his gaze towards lines 622b so that the driver's effective field of vision VF also moves close to the center of the road in front of the vehicle and the driver's attention can be returned to a suitable state.

[0138] It should be noted that the information to be displayed may have a different shape or mode than the lines 622b, as long as the driver's effective field of view VF can be returned to the vicinity of the center of the roadway.

[0139] In a Fig. In the example shown in Fig. 15(a), an area where a degree of attracting the driver's attention is high is extracted on the near left side of an image by the driving assistance device of the second modification.

[0140] As in Fig. 15(b), the driving assistance device according to the second modification may cause the speaker 63 to output an announcement 622d to attract attention, e.g., “Pay attention to the right side,” instead of the latent information 622, in accordance with a display timing of the image in Fig. 15(a). The presentation information output via the loudspeaker 63 may be a warning tone or the like.

[0141] Fig. 16 is a schematic view showing an example of the driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where areas where a degree of attracting the driver's attention is high are located in time series.

[0142] The Fig. 16(a) to 16(d) show time series data referred to when the driving assistance device of the second modification generates the driving assistance information. Fig. 16(a) to 16(d) are shaded areas where the driver’s level of attention is estimated to be high. However, this does not mean that the Fig. 16(a) to 16(d) are actually displayed on the HUD 61 or the like.

[0143] In a Fig. In the example shown in Fig. 16(a), areas where the degree of attracting the driver's attention is high are extracted on the left side of the image by the driving assistance device of the second modification. In an example shown in Fig. In the example shown in Fig. 16(b), regions where a degree of attracting the driver's attention is high are extracted near the center on the left side of the image by the driving assistance device according to the second modification. Fig. In the example shown in Fig. 16(c), regions where a degree of attracting the driver's attention is high are extracted near the center on the left side of the image by the driving assistance device according to the second modification. Fig. In the example shown in Fig. 16(d), areas where a degree of attracting the driver's attention is high are extracted on the left side of the image by the driving assistance device of the second modification.

[0144] As described above, in the examples of Fig. 16(a) to 16(d), the areas where a degree of attracting the driver's attention is high are shifted to the left in the time series.

[0145] Fig. Fig. 16(e) shows a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. That is, Fig. Figure 16(e) shows a process that takes place at a time after the acquisition of the Fig. 16(a) to 16(d).

[0146] As in Fig. As shown in Fig. 16(e), the driving assistance device according to the second modification presents the latent information 632 in areas on the right side opposite to the areas where there is a high degree of attention, and which are based on the images of Fig. 16(a) to 16(d) are extracted so that it deviates to the left side of the image.

[0147] In a case where the areas where a degree of attracting the driver's attention is high are located in time series as described above, the driving assistance device according to the second modification may present information such as the latent information 632 in areas opposite to the areas where the degree of attracting the driver's attention is located in time series, instead of sequentially changing the information presentation positions at the same time points as those in Fig. 16(a) to 16(d).

[0148] Fig. 17 is a schematic view showing an example of the driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where areas where a degree of attracting the driver's attention is high are ubiquitous in time series.

[0149] The Fig. 17(a) to 17(d) show time series data referred to when the driving assistance device of the second modification generates the driving assistance information. Fig. 17(a) to 17(d) are shaded areas where the driver’s level of attention is estimated to be high. However, this does not mean that the Fig. 17(a) to 17(d) are actually displayed on the HUD 61 or the like.

[0150] In a Fig. In the example shown in Fig. 17(a), an area where a degree of attracting the driver's attention is high is extracted by the driving assistance device of the second modification on the near right side of the image. In an example shown in Fig. In the example shown in Fig. 17(b), an area where a degree of attracting the driver's attention is high is extracted near the center on the left side of the image by the driving assistance device according to the second modification. In an example shown in Fig. In the example shown in Fig. 17(c), an area in which a degree of attracting the driver's attention is high is extracted near the left side of the image by the driving assistance device of the second modification. In an example shown in Fig. In the example shown in Fig. 17(d), an area where a degree of attracting the driver's attention is high is extracted by the driving assistance device of the second modification on the right side of the image.

[0151] As described above, in the examples of Fig. 17(a) to 17(d) the areas where the driver's attention is high are evenly distributed throughout the image in the time series.

[0152] Fig. Fig. 17(e) shows a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. That is, Fig. Figure 17(e) shows a process that takes place at a time after the acquisition of the Fig. 17(a) to 17(d).

[0153] As in Fig. 17(e), the driving assistance device of the second modification determines that the display of presentation information such as the latent information 622 by the HUD 61 is based on the images of Fig. 17(a) to 17(d) is unnecessary.

[0154] In a case where the areas where a degree of attracting the driver's attention is high are ubiquitous in time series as described above, the driving assistance device according to the second modification may intentionally skip the presentation of information such as the latent information 622, assuming that the driver pays attention evenly to the entire vehicle front.

[0155] Fig. 18 is a schematic view showing an example of the driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where the number of areas in which a degree of attracting the driver's attention is high increases or decreases in time series.

[0156] Fig. 18(a) to 18(d) show time series data referred to when the driving assistance device of the second modification generates the driving assistance information. Fig. 18(a) to 18(d) are shaded areas where the driver’s level of attention is estimated to be high. However, this does not mean that the Fig. 18(a) to 18(d) are actually displayed on the HUD 61 or the like.

[0157] In a Fig. In the example shown in Fig. 18(a), an area where a degree of attracting the driver's attention is high is extracted by the driving assistance device according to the second modification near the center on the left side of the image. In an example shown in Fig. In the example shown in Fig. 18(b), two areas where a degree of attracting the driver's attention is high are extracted on the left side closer to the image center by the driving assistance device according to the second modification. In an example shown in Fig. In the example shown in Figure 18(c), three areas where a degree of attracting the driver's attention is high are extracted on the left side of the image by the driving assistance device of the second modification. In an example shown in Fig. In the example shown in Fig. 18(d), five areas where a degree of attracting the driver's attention is high are extracted near the center on the left side of the image by the driving assistance device according to the second modification.

[0158] As described above, in the examples of Fig. 18(a) to 18(d), the number of areas where a degree of attracting the driver's attention is high gradually increases in the time series.

[0159] Fig. Fig. 18(e) shows a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. That is, Fig. Figure 18(e) shows a process that takes place at a time after the acquisition of the Fig. 18(a) to 18(d).

[0160] As in Fig. 18(e), the driving assistance device according to the second modification does not cause the HUD 61 to display the latent information 622 or the like in a specific area of ​​an image, but causes the HUD 61 to display a message 642 indicating that a cognitive load of the driver is increasing, e.g., “Slow down,” based on the images of Fig. 18(a) to 18(d). Information indicating that the driver's cognitive load is increasing can be output, for example, by an announcement via loudspeaker 63.

[0161] In a case where the number of areas where a degree of attracting the driver's attention is high increases in the time series as described above, the driving assistance device of the second modification can recognize that the driver's cognitive load is increasing and present the information for reducing the driver's cognitive load instead of the presentation information associated with the specific area of ​​the image.

[0162] Fig. 19 is a schematic view showing an example of the driving assistance information generated by the driving assistance device according to the second modification of the first embodiment in a case where a region in which a degree of attracting the driver's attention is high hardly appears in time series.

[0163] Fig. 19(A) to 19(D) show time series data referred to when the driving assistance device of the second modification generates the driving assistance information. Fig. 19(A) to 19(D) is a shaded area that easily attracts the driver's attention and is considered to have a high degree of inadequacy of the driver's attention state. However, this does not mean that the Fig. 19(A) to 19(D) are actually displayed on the HUD 61 or the like.

[0164] In a Fig. In the example shown in Figure 19(A), an area where the degree of inappropriateness of the driver's attention state exceeds a predefined threshold was not extracted. In an example shown in Fig. 19(B), no area was extracted where the degree of inappropriateness of the driver's attention state exceeds the predefined threshold. In a Fig. In the example shown in Figure 19(C), an area with a high degree of inappropriateness of the driver's attention state was extracted on the near right side of the image by the driving assistance device of the second modification. Fig. In the example shown in Fig. 19(D), a state in which an area in which the degree of inappropriateness of the driver's attention state exceeds the predefined threshold has not been extracted is repeated.

[0165] As described above, in the examples of Fig. 19(A) to 19(D), the area which easily attracts the driver's attention and has a high degree of inappropriateness of the driver's attention state also hardly appears in the time series.

[0166] Fig. 19(Ea) shows a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. That is, Fig. 19(Ea) shows a process that takes place at a time after the acquisition of the Fig. 19(A) to 19(D).

[0167] As in Fig. 19(Ea) based on the images of Fig. As shown in FIGS. 19(A) to 19(D), the driving assistance device according to the second modification does not cause the HUD 61 to display the latent information 622 or the like in a specific area of ​​an image, but causes the HUD 61 to display a message 642a indicating that the driver's attention to the entire front of the vehicle is low, e.g., "Pay attention to the front." Information indicating that attention to the entire front of the vehicle is low can be presented, for example, by an announcement from the speaker 63.

[0168] Fig. 19(Eb) is another example showing a state in which the driving assistance device of the second modification generates the driving assistance information based on the time series data. Fig. 19(Eb) also shows an exemplary process that takes place at a time after the recording of the Fig. 19(A) to 19(D).

[0169] As in Fig. 19(Ea), the driving assistance device of the second modification may, for example, display straight lines 642b extending from the near side to the far side to the left and right of the center of the screen, instead of displaying the message 642a such as “Pay attention to the front” based on the images of Fig. 19(A) to 19(D). This makes it possible to direct the driver's attention to the front of the vehicle, similar to the example described above in Fig. 14(Ba) and 14(Bb).

[0170] In a case where the range where a degree of attracting the driver's attention is high hardly appears even in time series as described above, the driving assistance device of the second modification can present information for suppressing a driver's careless driving on the assumption that the driver's attention to the entire front of the vehicle is low.

[0171] It should be noted that the presentation of information to suppress the driver’s careless driving is not limited to the example of Fig. 19 and may be presented in a mode other than that mentioned above.

[0172] Fig. 20 is a schematic view showing another example of presentation in a case where the driving assistance device according to the second modification of the first embodiment suppresses the driver's careless driving.

[0173] Fig. 20(a) shows only the state of Fig. 19(C) under the Fig. Time series data shown in Figures 19(A) to 19(D).

[0174] As in Fig. As shown in Fig. 20(b), the driving assistance device according to the second modification detects based on the time series data including Fig. 20(a) that the driver is likely to drive carelessly and causes the HUD 61 to display a message 642c recommending that the driver "drive defensively," such as "Drive slowly."

[0175] Defensive driving means driving with a high degree of safety while anticipating that a dangerous situation will occur. Examples of defensive driving include predicting that "a pedestrian might jump out" when approaching a crosswalk or similar area and preparing for a hazard, and predicting that "an oncoming vehicle might increase its speed" when turning right at an intersection and preparing for a hazard.

[0176] As described above, the information recommending “defensive driving” to the driver may be provided, for example, by an announcement via the loudspeaker 63 or similar.

[0177] Fig. 21 is a schematic view showing another example of presentation in the case where the driving assistance device according to the second modification of the first embodiment suppresses the driver's careless driving.

[0178] Fig. 21(a) also shows only the state of Fig. 19(C) under the Fig. Time series data shown in Figures 19(A) to 19(D).

[0179] As in Fig. 21(b), the driving assistance device according to the second modification recognizes based on the time series data including Fig. 20(a) that the driver is likely to drive carelessly and then, for example, if the driver is driving slowly, displays a message 642d commending “defensive driving,” such as “Appropriate slowing is being performed,” on the HUD 61.

[0180] Such “defensive driving” of the driver can be detected, for example, by the driver assistance device of the second modification receiving a detection result of the detection device 40 (see Fig. 1), such as the vehicle speed sensor 43, the acceleration sensor 44, the brake sensor 45 and the steering angle sensor 46, directly or via the ECU 20 described above (see Fig. 1) recorded.

[0181] The driving assistance device of the second modification can detect that the driver is decelerating the vehicle and driving slowly, for example, from the detection results of the vehicle speed sensor 43, the acceleration sensor 44, and the brake sensor 45. Furthermore, the driving assistance device of the second modification can recognize that the driver has steered the vehicle appropriately, for example, from a detection result of the steering angle sensor 46 or the like.

[0182] As described above, information praising the driver's "defensive driving" may be given, for example, by an announcement over the loudspeaker 63 or the like.

[0183] The presentation of information that supports the driver's "defensive driving" enables the driver to recognize that a current driving manipulation of the vehicle is correct and to be continuously motivated to perform the driving manipulation with a high sense of safety.

[0184] It should be noted that the above-described Fig. 12 to 21 show the examples in which the driving assistance device of the second modification causes the HUD 61 to display information only. However, the driving assistance device of the second modification may cause the vehicle monitor 62 to display the information described above in the Fig. 12 to 21 shows the different types of information.

[0185] According to the driving assistance device of the second modification, it is possible to cause at least a more suitable state of attention for driving at that time, as well as a change of behavior and a change of consciousness with respect to a driving manipulation based on the driver's state of attention, such as returning the driver's effective field of view VF to a suitable position as in the above-described examples of Fig. 14 and Fig. 15, a reduction of the cognitive load of the driver as in the example of Fig. 18, a suppression of the driver’s careless driving as in the example of Fig. 19 or promoting “defensive driving” as in the example of Fig. 20 or Fig. 21.

[0186] That is, it is possible to unconsciously change the visual behavior to correct the deviation when there is the deviation in the attention state estimated by the attention state estimation unit, to scatter the concentration when there is the excessive concentration on one part, and to cause concentration when there is the excessive scattering in the whole area, and it is possible to change the driving behavior such as slow driving or fixation to create a time for which the necessary attention can be distributed.

[0187] According to the driving assistance device of the second modification, in addition to the effects described above, similar effects to those of the driving assistance device 10 of the first embodiment are achieved. (Third modification)

[0188] Next, a driving assistance device according to a third modification of the first embodiment will be described with reference to FIG. Fig. 22 and Fig. 23. The driving assistance device of the third modification differs from the above-described first embodiment in that an image is divided by a different number of divisions.

[0189] Fig. 22 is a schematic view showing an example in which the driving assistance device according to the third modification of the first embodiment halves the image to generate driving assistance information.

[0190] As in Fig. 22(a) and Fig. As shown in Figure 22(b), the driving assistance device according to the third modification divides the image obtained by detecting the traveling direction of the vehicle into left and right halves and calculates whether the degree of inappropriateness of the driver's attention state exceeds a predefined threshold for each of the left and right regions. When a difference in the degree of inappropriateness of the attention state exceeds the predefined threshold in a region, the driving assistance device of the third modification displays latent information 652a in the entire region opposite the region.

[0191] In the Fig. 22(a) and Fig. In the examples shown in Figure 22(b), the degree of inappropriateness of the attention state exceeds the predefined threshold in the left area among the left and right divided areas, and the driving assistance device of the third modification displays the latent information 652a in the entire right area opposite the left area. At this time, the driving assistance device of the third modification can adjust the degree of intensity of the driver's attention request by changing, for example, a brightness, a color, or the like of the latent information 652a according to the degree of inappropriateness of the driver's attention state.

[0192] Fig. 23 is a schematic view showing another example in which the driving assistance device according to the third modification of the first embodiment halves the image to generate the driving assistance information.

[0193] Also in the Fig. 23(a) and Fig. In the examples shown in Figure 23(b), the driving assistance device according to the third modification divides the image obtained by detecting the traveling direction of the vehicle into left and right halves and calculates whether the degree of inappropriateness of the driver's attention state exceeds a predefined threshold for each of the left and right regions. When the degree of inappropriateness of the attention state in a region exceeds the predefined threshold, the driving assistance device of the third modification displays latent information 652b in a part of a region instead of the entire region opposite the region.

[0194] In the Fig. 23(a) and Fig. In the examples shown in Figure 23(b), if the degree of inappropriateness of the driver's attention state exceeds the predefined threshold in the left area among the left and right divided areas, the driving assistance device of the third modification displays the latent information 652b in a presentation area that is a rectangular area in the lower part of the windshield along the instrument panel, for example, at a lower end of the right area opposite the left area. At this time, the driving assistance device of the third modification can adjust the degree of intensity of the driver's attention request by changing, for example, a length, a thickness, a brightness, a color, or the like of the latent information 652b according to the degree of inappropriateness of the driver's attention state.

[0195] Note that the driving assistance device of the third modification may display the latent information 652a and 652b in an area opposite to an area where the degree of inappropriateness of the driver's attention state is higher when the degree of inappropriateness of the driver's attention state exceeds the predefined threshold in both the left and right areas.

[0196] As described above, the information to be presented to the driver can be simplified, e.g., by reducing the number of image divisions, and it is possible to present the information more appropriately to the driver by suppressing the driver's distraction.

[0197] With the driving assistance device of the third modification, further effects similar to those of the driving assistance device 10 of the first embodiment described above are achieved.

[0198] Note that the example of halving the image was described in the third modification described above. Furthermore, the example of dividing the image into a total of 25 areas of 5 × 5 in height × width was described in the first embodiment described above, and the like. However, the examples of image division are not limited to this, and the image can be divided into various numbers and arrangements. For example, the image can be divided for each pixel. In this case, the image is divided in height and width into 160 × 120, 1920 × 1080, or the like. (Fourth modification)

[0199] Next, a driving assistance device 10a according to a fourth modification of the first embodiment will be described with reference to FIG. Fig. 24 and Fig. 25. The driving assistance device 10a of the fourth modification differs from the above-described first embodiment in that the driving assistance information is output to the ECU 20.

[0200] Fig. 24 is a block diagram showing an example of a functional configuration of the driving assistance device 10a according to the fourth modification of the first embodiment together with peripheral devices.

[0201] As in Fig. As shown in Figure 24, the driving assistance device 10a of the fourth modification is mounted in a vehicle 101 of the fourth modification instead of the driving assistance device 10 of the above-described first embodiment. The driving assistance device 10a includes an output control unit 130a that outputs driving assistance information to the ECU 20 instead of the output control unit 130 of the above-described first embodiment.

[0202] The output control unit 130a generates the driving assistance information associated with an area to which the driver's attention should be directed. The driving assistance information includes, for example, operation information instructing the operation of the vehicle 101, a degree of operation determined based on the degree of inadequacy of the driver's attention state, and the like in connection with the area to which attention should be directed.

[0203] The instruction for operating the vehicle 101 contained in the driver assistance information includes, for example, at least one instruction for decelerating the vehicle 101 or one instruction for reducing the acceleration of the vehicle 101. The degree of the operation contained in the driver assistance information may, for example, be a magnitude of a braking effect of the vehicle 101.

[0204] That is, the output control unit 130a can output the driving assistance information including the operation information with a higher braking effect of the vehicle 101 because the area to which the driver's attention is to be directed is based on a higher degree of inadequacy of the driver's attention state.

[0205] The ECU 20, which is the electronic unit in the vehicle, initiates at least one (see Fig. 1) the brake actuators 51 and the engine controller 52 included in the above-described vehicle control device 50 to perform an operation for decelerating the vehicle 101 based on the driving assistance information output from the driving assistance device 10a to the ECU 20.

[0206] That is, the ECU 20 brakes the vehicle 101, for example, through the brake actuator 51, which is a braking device. Furthermore, the ECU 20 reduces the acceleration of the vehicle 101 through the engine controller 52, which is an engine control device, instead of or in addition to the process of decelerating the vehicle 101 through the brake actuator 51.

[0207] As a result, in a case where the area to which the driver's attention should be drawn is extracted by the driving assistance device 10a, it is possible to cause the vehicle 101 to perform an operation to avoid a danger that may exist in the area. For example, in a case where the driver is not paying attention to the area and there is a pedestrian or the like that is at risk of contact with the vehicle 101 in the area, it is possible to avoid contact with the pedestrian by decelerating the vehicle 101.

[0208] In addition, the output control unit 130a is assigned to the area to which the driver's attention is to be directed and also generates the driving assistance information to be output to the HMI control device 30 (see Fig. 1). The driving assistance information to be output to the HMI control device 30 includes, for example, presentation information for informing the driver of an operation performed by the vehicle 101 in connection with the area to which attention is to be drawn.

[0209] Based on the driving assistance information output from the driving assistance device 10a, the HMI control device 30 causes the information presentation device 60 (see Fig. 1) to present the presentation information to inform the driver that the vehicle 101 has been caused to perform the evasive action as described above.

[0210] Fig. 25 is a schematic view showing an example of presentation of information by the driving assistance device 10a according to the fourth modification of the first embodiment.

[0211] As in Fig. 25(a) and Fig. As shown in Fig. 25(b), the driving assistance device 10a of the fourth modification outputs the above-described driving assistance information to the HMI control device 30 to cause the vehicle 101 to perform the above-described evasive action, and causes the HUD 61 or the in-vehicle monitor 62 to display a message 634 indicating that the braking control of the vehicle 101 has been performed, e.g., "Slow driving has been performed." The information that the braking control of the vehicle 101 has been performed may be provided, for example, by an announcement via the speaker 63.

[0212] Thus, when the driving assistance device 10a has caused the vehicle 101 to perform the evasive action, it can be prevented that the driver misunderstands that a fault in the vehicle or the like occurs due to an unintentional braking control of the vehicle.

[0213] It should be noted that a case where the braking control of the vehicle 101 is performed is shown in the examples of Fig. 24 and Fig. 25, but other operational controls, such as setting a lane by steering angle control of the steering wheel and turn signal manipulation, can be performed on the vehicle 101.

[0214] According to the driving assistance device 10a of the fourth modification, the operation information associated with the area to which the driver's attention should be directed and instructing the operation of the vehicle 101 is output to the ECU 20 that controls the vehicle 101. The ECU 20 causes the brake actuator 51, the engine controller 52, and the like that decelerate the vehicle to perform the operation based on the operation information output from the driving assistance device 10a. This makes it possible to more reliably avoid a danger that the driver is likely to miss.

[0215] According to the driving assistance device 10a of the fourth modification, the operating information is output with a higher braking effect of the vehicle 101 because the area to which the driver's attention should be directed is based on a larger prediction error. This further increases the accuracy of hazard avoidance.

[0216] According to the driving assistance device 10a of the fourth modification, other effects similar to those of the driving assistance device 10 of the first embodiment described above are achieved. (Fifth Modification)

[0217] Next, a driving assistance device according to a fifth modification of the first embodiment will be described with reference to FIG. Fig. 26 to 45. The driving assistance device of the fifth modification differs from that of the first embodiment described above in that manifest information is also presented in addition to the latent information.

[0218] Fig. 26 is a block diagram showing an example of a functional configuration of a driving assistance device 10b according to the fifth modification of the first embodiment together with peripheral devices.

[0219] As in Fig. 26, instead of the driving assistance device 10 according to the first embodiment described above, the driving assistance device 10b according to the fifth modification is installed in a vehicle 102 according to the fifth modification.

[0220] The driving assistance device 10b includes a manifest information calculation unit 111 that generates the manifest information in addition to the configuration of the driving assistance device 10 of the first embodiment described above, and further includes an output control unit 130b instead of the output control unit 130 that acquires the driver's attention state estimated by the driver attention state estimation unit 120 and the manifest information generated by the manifest information calculation unit 111 and outputs driving assistance information of the first embodiment described above.

[0221] As described above, the hazards encountered during the vehicle's travel include the hazard manifested at that time and the latent hazard that is not manifested at that time. Regarding the various types of latent information described above for notifying the driver of the latent hazard, the manifest information calculation unit 111 generates the manifest information for notifying the driver of the hazard manifested at that time.

[0222] More specifically, the manifest information calculation unit 111 extracts, from an image in the traveling direction of a vehicle 102 taken by the vehicle exterior camera 41, the danger manifested at that time, such as a pedestrian who suddenly jumped onto the roadway, and generates the manifest information including such danger information.

[0223] The manifestation information includes, for example, presentation information for notifying the driver of the obvious danger, a presentation mode, information about the intensity of the attention request of the presentation information, and the like in connection with the area to which attention is to be drawn.

[0224] The driver attention state estimation unit 120 outputs the estimated driver attention state and the manifest information generated by the manifest information calculation unit 111 to the output control unit 130b.

[0225] The output control unit 130b outputs to the HMI control device 30 the driving assistance information including information about the area to which the driver's attention should be directed and extracted based on the driver's attention state, the manifest information transmitted from the driver attention state estimation unit 120, and the latent information.

[0226] Fig. 27 is a schematic view showing an example of presentation of information by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0227] As in Fig. 27(a) and Fig. As shown in Fig. 27(b), the driving assistance device 10b of the fifth modification causes the HUD 61 or the in-vehicle monitor 62 to display manifest information 662a and latent information 605a. In the example of Fig. 27(a) and Fig. 27(b), for example, an area with a person on the right is assumed to be the area to which the driver's attention should be directed.

[0228] The driving assistance device 10b of the fifth modification displays the manifest information 662a, which is elliptical and indicates the person to the right of the person, at the person's feet. Furthermore, the driving assistance device 10b of the fifth modification displays the latent information 605a, which is rectangular and indicates the person to the near right, at a lower position away from the person's feet.

[0229] However, the forms, presentation positions, presentation modes and the like of the manifest information 662a and the latent information 605a are not limited to the above-described example of Fig. 27 limited.

[0230] Fig. 28 is a schematic view showing another example of information presentation by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0231] In the Fig. 28(a) and Fig. In the examples shown in Figure 28(b), the driving assistance device 10b of the fifth modification displays the manifest information 662b, which is rectangular and indicates the person on the near right side, so that it surrounds the person. Furthermore, the driving assistance device 10b of the fifth modification displays the latent information 605b, which is elliptical and indicates the person on the near right side, at a lower position, away from the person's feet.

[0232] Note that the driving assistance device 10b of the fifth modification can, for example, change the size, brightness, color, or the like of each piece of the manifest information 662a and 662b to adjust the intensity of the driver's attention request. Furthermore, the driving assistance device 10b of the fifth modification can adjust the degree of the intensity of the driver's attention request by changing a size, brightness, color, or the like of each piece of the latent information 605a and 605b according to the degree of inappropriateness of the driver's attention state.

[0233] As described above, the forms, presentation positions, presentation modes, and the like of the manifest information 662 and the latent information 605 can be presented in different modes.

[0234] With the above configuration, the driver's alertness can be increased by presenting not only the latent danger but also the already manifest danger. This makes it possible to present the information to the driver even more appropriately.

[0235] It should be noted that when both the manifest information and the latent information are presented to the driver as in the configuration described above, the manifest information and the latent information may be presented on different information presentation devices 60, respectively. At this time, it is advantageous to present the manifest information indicating a higher risk to the HUD 61, the in-vehicle monitor 62, or the like, which have high visibility and are easily perceived by the driver among the various configurations included in the information presentation device 60.

[0236] On the other hand, the latent information indicating a relatively low risk may be presented in a configuration in which the information presentation is possible without directing the driver's attention to the front or a target to be observed among the various configurations of the information presentation device 60.

[0237] As an example of such a configuration, examples are described below where the latent information is displayed on a meter display 64 (see Fig. 29), a column display 65 (see Fig. 33), the loudspeaker 63, an outer edge of the HUD 61, an LED display 66 (see Fig. 40), a mirror display 67 (see Fig. 44) and the like.

[0238] Fig. 29 is a schematic view showing an example of a configuration of the meter display 64 on which the driving assistance device 10b according to the fifth modification of the first embodiment presents the latent information.

[0239] As in Fig. As shown in Figure 29, the meter display 64 includes various counters 642, such as a speedometer, an engine tachometer, a fuel gauge, a water temperature gauge, and an odometer, as well as a center information display (CID) 641 disposed between and around these counters 642 and provided on an instrument panel or the like. The counter 642 may be a counter with a physical configuration, or the counter 624 itself may be a video displayed on a display.

[0240] Fig. 30 is a schematic view showing an example of presentation of latent information on the meter display 64 by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0241] As in Fig. For example, as shown in Fig. 30(A), in a case where there is an area where the degree of inappropriateness of the driver's attention state is high over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays latent information 615 surrounding an outer peripheral area of ​​the meter display 64 on the entire circumference of the meter display 64. However, the latent information for requesting high attention with respect to the entire front of the vehicle may have a different form or mode than the latent information 615.

[0242] As in Fig. 30(Ba), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays a pair of latent information 619a at both the left and right ends of the meter display 64.

[0243] For example, latent information 619a may have a shape resulting from the combination of two triangles pointing near the center of meter display 64, which is compared to the front of the vehicle. However, the latent information intended to draw faint attention to the entire front of the vehicle may also have other shapes or modes, such as an arrow.

[0244] As in Fig. 30(Bb), the driving assistance device 10b of the fifth modification may display, as another type of the latent information 619a, for example, in a case where there is a region where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the latent information 619b, which is triangular and points to the center of the meter display 64, at a lower end of the center of the meter display 64.

[0245] As in Fig. For example, as shown in Fig. 30(Ca), in a case where there is an area where the degree of inadequacy of the driver's attention state is high, the area deviating to the right side of the vehicle, the driving assistance device 10b of the fifth modification displays latent information 616a indicating an area on the left side opposite to the area at the left end of the meter display 64.

[0246] For example, latent information 616a may be presentation information presented in a rectangular area along the left end of meter display 64, which is comparable to the area on the left side of the vehicle. However, latent information requesting high attention with respect to the driver with a large deviation in attention state may have a different shape or mode than latent information 616a.

[0247] As in Fig. 30(Cb), as another mode of the latent information 616a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle, the driving assistance device 10b of the fifth modification may display the latent information 616b indicating an area on the left side opposite to the area at a left lower end of the meter display 64.

[0248] For example, the latent information 616b may be presentation information arranged in a rectangular area along the lower left end of the meter display 64, which is comparable to the area on the left side of the vehicle.

[0249] As in Fig. 30(Da), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is relatively low is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification displays, in addition to the latent information 616a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 617a indicating an area on the right side opposite to the area on the left side of the vehicle at the right end of the meter display 64.

[0250] For example, latent information 617a may be presentation information presented in a rectangular area along the left end of meter display 64, which is comparable to the area on the right side of the vehicle. Furthermore, latent information 617a is presented in a mode in which a brightness, color, or the like differs from that of latent information 616a, so that the intensity of the attention request can be suppressed to be lower than that of latent information 616a.

[0251] It should be noted that the latent information for requesting weak attention with respect to the driver with a small deviation in the attention state may have a different form or mode than the latent information 617a.

[0252] As in Fig. 30(Db), the driving assistance device 10b of the fifth modification may display, as another type of the latent information 617a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is relatively low is located to deviate to the left side of the vehicle, in addition to the latent information 616b, latent information 617b indicating a right area opposite to the left area at the left lower end of the meter display 64.

[0253] For example, latent information 617b may be presentation information located in a rectangular area along the lower right end of meter display 64, which is comparable to the area on the left side of the vehicle. Furthermore, latent information 617b has a mode in which the intensity of the attention request is lower than that of latent information 616b.

[0254] As in Fig. 30(Ea), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is medium is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification presents, in addition to the latent information 616a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 618a indicating an area on the left side opposite to the area on the right side of the vehicle at the right end of the meter display 64.

[0255] For example, latent information 618a may be presentation information presented in the rectangular area located along the left end of the meter display 64, which is comparable to the area on the right side of the vehicle. Furthermore, latent information 618a is presented in a mode in which brightness, color, or the like differs from those of latent information 616a and 617a, so that the intensity of the attention request can be suppressed to be lower than that of latent information 616a while being set to a higher value than that of latent information 617a.

[0256] It should be noted that the latent information for requesting moderate attention with respect to the driver having a moderate deviation in the attention state may have a different form or mode than the latent information 618a.

[0257] As in Fig. 30(Eb), the driving assistance device 10b of the fifth modification may display, as another mode of the latent information 618a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is medium is located to deviate to the left side of the vehicle, in addition to the latent information 616b indicating an area on the right side versus the area on the left side of the vehicle, latent information 618b indicating an area on the right side versus the area on the left side at the left lower end of the meter display 64.

[0258] For example, latent information 618b may be presentation information presented in a rectangular area arranged along a lower right end of meter display 64, which is comparable to the area on the left side of the vehicle. Furthermore, latent information 617b has a mode in which the intensity of the attention request is higher than that of latent information 617b and the intensity of the attention request is lower than that of latent information 616b.

[0259] With the above configuration, various types of the latent information 615, 616a to 619a, 616b to 619b and the like can be displayed on the meter display 64 based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area where the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like.

[0260] Next, an example will be described in which the driving assistance device 10b according to the fifth modification presents information on the HUD 61 and the meter display 64 in combination.

[0261] Fig. 31 is a schematic view showing an example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is divided and displayed on the HUD 61 and the meter display 64.

[0262] In the Fig. 31(Aa), 31(Ba) and 31(Ca), the driving assistance device 10b of the fifth modification is in a state where it detects the latent danger such as a deviation in the driver's attention state, but not the manifest danger such as a pedestrian who is likely to collide with the base vehicle.

[0263] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described rectangular latent information 605a, the above-described elliptical latent information 605b, or the like, for example, to indicate the person to the right. On the other hand, the driving assistance device 10b according to the fifth modification does not cause the meter display 64 to display any information.

[0264] When only the latent danger is detected, the driving assistance device 10b of the fifth modification may preferably display the latent information 605a, 605b or the like on the HUD 61 with high visibility as described above.

[0265] In the Fig. 31(Ab), 31(Bb), and 31(Cb), the driving assistance device 10b of the fifth modification is in a state in which the manifest danger is also detected in addition to the latent danger.

[0266] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described elliptical manifest information 662a, the above-described rectangular manifest information 662b, or the like, for example, to point to the person on the near right. At this time, the driving assistance device 10b of the fifth modification changes a size, brightness, color, or the like of each piece of the manifest information 662a and 662b according to a size of the detected obvious danger.

[0267] The driving assistance device 10b according to the fifth modification causes the meter display 64 to display the latent information 616a described above or the like.

[0268] Although the latent information 616a in the example of Fig. 31(Ab) is displayed at this time, the driving assistance device 10b of the fifth modification may display each of the latent information 615, 616a to 619a, and 616b to 619b on the meter display 64 according to a magnitude of the detected latent danger, that is, a magnitude of the degree of inadequacy of the driver's attention state.

[0269] When both the latent danger and the manifest danger are detected, the driving assistance device 10b of the fifth modification may display the manifest information 662a, 662b or the like on the HUD 61 with high visibility and display the latent information 616a or the like on the meter display 64, which may less distract the driver's attention, as described above.

[0270] Fig. 32 is a schematic view showing another example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is divided and displayed on the HUD 61 and the meter display 64.

[0271] In the Fig. 32(a), Fig. 32(b) and Fig. In the examples shown in Fig. 32(c), the driving assistance device 10b of the fifth modification is in a state in which it detects the latent danger but does not detect the manifest danger, that is, in a state similar to that shown in Fig. Fig. 31(Aa), 31(Ba) and 31(Ca).

[0272] In this case, the driving assistance device 10b according to the fifth modification may also cause the meter display 64 to display the latent information 616a described above or the like. In this case, the driving assistance device 10b according to the fifth modification does not cause the HUD 61 to display information.

[0273] Even if only the latent danger is detected, the driving assistance device 10b of the fifth modification can always cause the meter display 64 to display the latent information 616a or the like as described above.

[0274] Fig. 33 is a schematic view showing an example of a configuration of the column display 65 on which the driving assistance device 10b according to the fifth modification of the first embodiment presents latent information.

[0275] As in Fig. As shown in Figure 33, the pillar display 65 is provided, for example, in each of the pillars PL on both sides of the windshield. Since images of parts obscured by the driver as seen from the pillars PL are displayed on the pillar displays 65, it is possible to provide the driver with a view as if looking out of the vehicle through the pillars PL, for example. The driving assistance device 10b of the fifth modification, for example, causes such pillar displays 65 to display latent information.

[0276] Fig. 34 is a schematic view showing an example of presentation of latent information on the column display 65 by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0277] As in Fig. 34(Aa), for example, in an area where the degree of inadequacy of the driver's attention state is high over the entire front of the vehicle, the driving assistance device 10b of the fifth modification causes the column displays 65 on both sides of the windshield to display latent information 625.

[0278] The latent information 625 may be presentation information arranged in rectangular areas along the pillars PL on both sides of the windshield. However, the latent information intended to draw attention to the entire front of the vehicle may also have a different form or mode than the latent information 625 described above.

[0279] As in Fig. 34(Ab), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the driving assistance device 10b of the fifth modification causes the column displays 65 on both sides of the windshield to display a pair of latent information 629.

[0280] For example, latent information 629 may have a shape obtained by combining a pair of triangles pointing toward the vicinity of the center of the windshield located at the front of the vehicle. However, latent information intended to draw faint attention to the entire front of the vehicle may also have other shapes or modes, such as an arrow.

[0281] As in Fig. 34(B), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is high, the area deviating to the right side of the vehicle, the driving assistance device 10b of the fifth modification causes the column display 65 on the left side of the windshield to display latent information 626 indicating an area on the left side opposite to the area.

[0282] For example, latent information 626 may be presentation information arranged in a rectangular area along the pillar PL on the left side of the windshield at the front of the vehicle. However, latent information requesting high attention regarding the driver with a large deviation in attention state may have a different form or mode than latent information 626.

[0283] As in Fig. 34(Ca), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is relatively low is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification causes the column displays 65 on both sides of the windshield to display latent information 627 in addition to the latent information 626 indicating an area on the right side versus the area on the left side of the vehicle, indicating the area on the left side versus the area on the right side of the vehicle.

[0284] For example, latent information 627 may be presentation information arranged in a rectangular area along the pillar PL to the right of the windshield at the front of the vehicle. Furthermore, latent information 627 is presented in a mode in which brightness, color, or the like differs from latent information 626, so that the intensity of the attention request can be suppressed to be lower than that of latent information 626.

[0285] It should be noted that the latent information for prompting weak attention regarding the driver with a small deviation in the attention state may have a different form or mode than the latent information 627.

[0286] As in Fig. 34(Cb), for example, in a case where the degree of inappropriateness of the driver's attention state is high, is arranged to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is medium is arranged to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification causes the column displays 65 on both sides of the windshield to display latent information 628 in addition to the latent information 626 indicating an area on the right side versus the area on the left side of the vehicle, indicating the area on the left side versus the area on the right side of the vehicle.

[0287] For example, the latent information 628 may be presentation information arranged in a rectangular area along the pillar PL on the right side of the windshield at the front of the vehicle. Furthermore, the latent information 628 is presented in a mode in which brightness, color, or the like differs from those of the latent information 626 and 627, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 626 while being adjusted to be higher than that of the latent information 627.

[0288] It should be noted that the latent information for requesting moderate attention with respect to the driver having a moderate deviation in the attention state may have a different form or mode than the latent information 628.

[0289] With the above configuration, various types of the latent information 625 and 626 to 629 and the like can be displayed on the column displays 65 based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area where the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like.

[0290] Next, an example will be described in which the driving assistance device 10b according to the fifth modification presents information on the HUD 61 and the column display 65 in combination.

[0291] Fig. 35 is a schematic view showing an example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is divided and displayed on the HUD 61 and the column display 65.

[0292] In the Fig. In the examples shown in Figs. 35(Aa) and 35(Ba), the driving assistance device 10b of the fifth modification is in a state where it detects the latent danger such as a deviation in the driver's attention state, but does not detect the manifest danger such as a pedestrian who is likely to collide with the base vehicle.

[0293] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described rectangular latent information 605a, the above-described elliptical latent information 605b, or the like, for example, to indicate the person to the right. On the other hand, the driving assistance device 10b of the fifth modification does not cause the column display 65 to display any information.

[0294] When only the latent danger is detected, the driving assistance device 10b of the fifth modification may preferentially display the latent information 605a, 605b or the like on the HUD 61 having high visibility as described above.

[0295] In the Fig. In the examples shown in Figs. 35(Ab) and 35(Bb), the driving assistance device 10b of the fifth modification is in a state in which the manifest danger is also detected in addition to the latent danger.

[0296] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described elliptical manifest information 662a, the above-described rectangular manifest information 662b, or the like, for example, to indicate the person on the near right. At this time, the driving assistance device 10b of the fifth modification changes a size, brightness, color, or the like of each piece of the manifest information 662a and 662b according to a size of the detected manifest danger.

[0297] In addition, the driving assistance device 10b of the fifth modification displays the above-described latent information 626 or the like on the column display 65 on the right side of the windshield.

[0298] Although the latent information 626 in the examples of Fig. 35(Ab) and 35(Bb) is displayed at this time, the driving assistance device 10b of the fifth modification can display each of the above-described latent information 625 and 626 to 629 on the column display 65 according to a magnitude of the detected latent danger, that is, a magnitude of the degree of inadequacy of the driver's attention state.

[0299] When both the latent danger and the manifest danger are detected, the driving assistance device 10b of the fifth modification may display the manifest information 662a, 662b or the like on the HUD 61 with high visibility and display the latent information 626 or the like on the column display 65, which may less distract the driver's attention, as described above.

[0300] Fig. 36 is a schematic view showing another example in which the presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is divided and displayed on the HUD 61 and the column display 65.

[0301] In the Fig. 36(a) and Fig. In the examples shown in Fig. 36(b), the driving assistance device 10b of the fifth modification is in a state in which it detects the latent danger but does not detect the manifest danger, that is, in a state similar to that shown in Figs. Fig. 35(Aa) and 35(Ba) shown above.

[0302] In this case, the driving assistance device 10b of the fifth modification may also cause the column display 65 to display the latent information 626 described above, or the like. In this case, the driving assistance device 10b according to the fifth modification does not cause the HUD 61 to display information.

[0303] Even if only the latent danger is detected, the driving assistance device 10b of the fifth modification can always cause the column display 65 to display the latent information 626 or the like as described above.

[0304] Next, an example will be described in which the driving assistance device 10b according to the fifth modification presents information through the HUD 61 and the speaker 63 in combination.

[0305] Fig. 37 is a schematic view showing an example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is shared and presented via the HUD 61 and the speaker 63.

[0306] In the Fig. In the examples shown in Figs. 37(Aa) and 37(Ba), the driving assistance device 10b of the fifth modification is in a state where it detects the latent danger such as a deviation in the driver's attention state, but not the manifest danger such as a pedestrian who is likely to collide with the base vehicle.

[0307] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described rectangular latent information 605a, the above-described elliptical latent information 605b, or the like, for example, to indicate the person to the right. On the other hand, the driving assistance device 10b according to the fifth modification does not cause the speaker 63 to output information.

[0308] When only the latent danger is detected, the driving assistance device 10b of the fifth modification may preferentially display the latent information 605a, 605b or the like on the HUD 61 having high visibility as described above.

[0309] In the Fig. In the examples shown in Figs. 37(Ab) and 37(Bb), the driving assistance device 10b of the fifth modification is in a state in which the manifest danger is also detected in addition to the latent danger.

[0310] In this case, the driving assistance device 10b of the fifth modification causes the HUD 61 to display the above-described elliptical manifest information 662a, the above-described rectangular manifest information 662b, or the like, for example, to indicate the person on the near right. At this time, the driving assistance device 10b of the fifth modification changes a size, brightness, color, or the like of each piece of the manifest information 662a and 662b according to a size of the detected manifest danger.

[0311] In addition, the driving assistance device 10b of the fifth modification causes the speaker 63 to output a sound 636 containing latent information such as an announcement or a warning sound for warning to pay attention to an area on the right side.

[0312] When both the latent danger and the manifest danger are detected, the driving assistance device 10b of the fifth modification may display the manifest information 662a, 662b or the like on the HUD 61 having high visibility as described above, and may output the latent information such as the sound 636 to the speaker 63 which less obstructs the driver's view.

[0313] Fig. 38 is a schematic view showing another example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is shared and presented via the HUD 61 and the speaker 63.

[0314] In the Fig. 38(a) and Fig. 38(b), the driving assistance device 10b of the fifth modification is in a state in which the latent danger is detected but the manifest danger is not detected, that is, in a state similar to that in the above-described examples in Fig. 37(Aa) and 37(Ba) is similar to the state shown.

[0315] In this case, the driving assistance device 10b according to the fifth modification also causes the speaker 63 to output the sound 636 including the latent information. In this case, the driving assistance device 10b according to the fifth modification does not cause the HUD 61 to display information.

[0316] Even if only the latent danger is detected, the driving assistance device 10b of the fifth modification can always cause the speaker 63 to output the latent information such as the sound 636 as described above.

[0317] Fig. 39 is a schematic view showing an example of presentation of latent information on the outer peripheral portion of the HUD 61 by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0318] For example, as described below, the display of the latent information using the outer peripheral area of ​​the HUD 61 may be performed in a similar manner to the display of the latent information using the Fig. 30 shown meter display 64, which was described above.

[0319] As in Fig. For example, as shown in FIG. 39(A), in a case where there is an area where the degree of inappropriateness of the driver's attention state is high over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays latent information 645 on an outer peripheral area of ​​the HUD 61 surrounding the entire circumference of the HUD 61. However, the latent information for requesting high attention with respect to the entire front of the vehicle may have a form or mode different from that of the latent information 645.

[0320] As in Fig. 39(Ba), for example, in an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays a pair of latent information 649a at both the left and right ends of the HUD 61.

[0321] For example, the latent information 649a may have a shape obtained by combining a pair of triangles pointing near the center of the HUD 61 located at the front of the vehicle. However, the latent information intended to draw faint attention to the entire front of the vehicle may also have other shapes or modes, such as an arrow.

[0322] As in Fig. 39(Bb), the driving assistance device 10b of the fifth modification may display, as another type of the latent information 649a, for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, latent information 649b that is triangular and points to the center of the HUD 61 at a lower end of the center of the HUD 61.

[0323] As in Fig. 39(Ca), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is high, the area deviating to the right side of the vehicle, the driving assistance device 10b of the fifth modification displays, at the left end of the HUD 61, latent information 646a indicating an area on the left side opposite to the area.

[0324] For example, latent information 646a may be presentation information presented in a rectangular area along the left end of the HUD 61 located at the front of the vehicle. However, latent information requesting high attention regarding the driver with a large deviation in attention state may have a different form or mode than latent information 646a.

[0325] As in Fig. 39(Cb), as another mode of the latent information 646a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle, the driving assistance device 10b of the fifth modification may display latent information 646b indicating an area on the left side opposite to the area at a left lower end of the HUD 61.

[0326] The latent information 646b may, for example, be presentation information presented in a rectangular area along the lower left end of the HUD 61 arranged at the front of the vehicle.

[0327] As in Fig. 39(Da), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is relatively low is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification displays, at the right end of the HUD 61, in addition to the latent information 646a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 647a indicating an area on the right side opposite to the area on the left side of the vehicle.

[0328] For example, the latent information 647a may be presentation information presented in a rectangular area along the right side of the A-pillar of the HUD 61 arranged at the front of the vehicle. Furthermore, the latent information 647a is presented in a mode in which a brightness, a color, or the like differs from that of the latent information 646a, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 646a.

[0329] It should be noted that the latent information for requesting weak attention with respect to the driver having a small deviation in the attention state may have a different form or mode than the latent information 647a.

[0330] As in Fig. 39(Db), as another embodiment of the latent information 647a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is relatively low is located to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth embodiment may display, in addition to the latent information 646b, latent information 647b indicating an area on the right side opposite to the area on the left side at the lower left end of the HUD 61.

[0331] For example, latent information 647b may be presentation information presented in a rectangular area along a lower right end of HUD 61 located at the front of the vehicle. Furthermore, latent information 647b has a mode in which the intensity of the attention request is lower than that of latent information 646b.

[0332] As in Fig. 39(Ea), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is medium is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification displays, at the right end of the HUD 61, in addition to the latent information 646a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 648a indicating an area on the right side opposite to the area on the left side of the vehicle.

[0333] For example, the latent information 648a may be presentation information arranged in a rectangular area along the right end of the HUD 61 arranged at the front of the vehicle. Furthermore, the latent information 648a is presented in a mode in which a brightness, a color, or the like differs from those of the latent information 646a and 647a, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 646a while being set higher than that of the latent information 647a.

[0334] It should be noted that the latent information for requesting moderate attention with respect to the driver having a moderate deviation in the attention state may have a different form or mode than the latent information 648a.

[0335] As in Fig. 39(Eb), in another mode of the latent information 648a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is medium is located to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification may display, in addition to the latent information 646b, latent information 658b indicating an area on the right side opposite to the area on the left side at the left lower end of the HUD 61.

[0336] For example, latent information 648b may be presentation information presented in a rectangular area along the lower right end of the HUD 61 located at the front of the vehicle. Furthermore, latent information 648b has a mode in which the intensity of the attention request is higher than that of latent information 647b and the intensity of the attention request is lower than that of latent information 646b.

[0337] With the above configuration, various types of the latent information 645, 646a to 649a, 646b to 649b and the like can be presented on the HUD 61 based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area in which the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like.

[0338] It should be noted that even if various types of the latent information 645, 646a to 649a, 646b to 649b and the like are presented using the outer peripheral area of ​​the HUD 61, various images displayed in the Fig. 31, Fig. 32 and the like described above are displayed in the central area of ​​the HUD 61 except for the outer peripheral area.

[0339] Fig. 40 is a schematic view showing an example of a configuration of the LED display 66 on which the driving assistance device 10b according to the fifth modification of the first embodiment presents latent information.

[0340] As in Fig. As shown in Figure 40, the LED display 66, which is a light-emitting device, includes, for example, a plurality of LEDs arranged along a lower end of the HUD 61 disposed on the windshield. The driving assistance device 10b of the fifth modification turns these LEDs on / off in a predefined mode to present latent information on the LED display 66.

[0341] Fig. 41 is a schematic view showing an example of presentation of latent information on the LED display 66 by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0342] As in Fig. 41 (Aa), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is high over the entire front of the vehicle, the driving assistance device 10b of the fifth modification presents latent information 655 in which the entire LED display 66 extending along the lower end of the HUD 61 is turned on.

[0343] That is, the latent information 655 may be presentation information in a state where all of the plurality of LEDs arranged along the lower end of the HUD 61 are turned on. However, the latent information for requesting high attention with respect to the entire vehicle front may have a different mode from the latent information 655 described above.

[0344] As in Fig. 41 (Ab), for example, in a case where there is an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the driving assistance device 10b of the fifth modification presents latent information 659 in which a central part of the LED display 66 extending along the lower end of the HUD 61 is turned on.

[0345] That is, the latent information 659 may be presentation information in a state where some LEDs arranged in the central portion among the plurality of LEDs arranged along the lower end of the HUD 61 are turned on. However, the latent information for requesting weak attention with respect to the entire vehicle front may have a different mode from the latent information 655 described above.

[0346] As in Fig. For example, as shown in Fig. 41(B), in a case where there is an area where the degree of inadequacy of the driver's attention state is high, the area deviating to the right side of the vehicle, the driving assistance device 10b of the fifth modification causes the LED display 66 to present latent information 656 indicating an area on the left side opposite to the area.

[0347] The latent information 656 may be presentation information in a state where a left portion of the LED display 66 extending along the lower end of the HUD 61 is turned on. That is, some of the left-arranged LEDs among the plurality of LEDs arranged along the lower end of the HUD 61 are turned on. However, the latent information for requesting high attention regarding the driver who has a large deviation in the attention state may have a different form or mode than the latent information 656.

[0348] As in Fig. 41 (Ca), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is relatively low is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification causes the LED display 66 to display latent information 657 indicating an area on the right side opposite to the area on the left side of the vehicle, in addition to the latent information 656 indicating an area on the left side opposite to the area on the right side of the vehicle.

[0349] The latent information 657 may be presentation information in a state where a right part of the LED display 66 extending along the lower end of the HUD 61 is turned on. That is, some LEDs arranged on the right side of the plurality of LEDs arranged along the lower end of the HUD 61 are turned on. Moreover, the latent information 657 is presented in a mode where the number, brightness, color, or the like of the LEDs to be turned on is different from that of the latent information 656, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 656.

[0350] It should be noted that the latent information for the weak attention request with respect to the driver who has a small deviation in the attention state may have a different form or mode than the latent information 657.

[0351] As in Fig. 41(Cb), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is medium is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification causes the LED display 66 to display latent information 658 indicating an area on the right side opposite to the area on the left side of the vehicle, in addition to the latent information 656 indicating an area on the left side opposite to the area on the right side of the vehicle.

[0352] The latent information 658 may be presentation information in a state where a right portion of the LED display 66 extending along the lower end of the HUD 61 is turned on. Furthermore, the latent information 658 is presented in a mode where the number, brightness, color, or the like of the LEDs to be turned on differs from those of the latent information 656 and 657, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 656 while being set higher than that of the latent information 657.

[0353] It should be noted that the latent information for requesting moderate attention with respect to the driver who has a moderate deviation in the attention state may have a different form or mode than the latent information 658.

[0354] With the above configuration, the LED displays 66 can be caused to present various types of the latent information 655 and 656 to 659 and the like, based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area in which the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like.

[0355] Incidentally, in the above-described Fig. 41, the intensity of the attention request of each piece of latent information 655 and 656 to 659 is presented in different modes, for example, by presenting the number, brightness, color, or the like of the LEDs to be switched in different modes. However, a method for presenting the intensity of the attention request in different modes by the LED display 66 is not limited to that described above.

[0356] Fig. 42 is a schematic view showing an example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is presented on the LED display 66 by flashing the LEDs.

[0357] As in Fig. As shown in Fig. 42(a), the driving assistance device 10b of the fifth modification may cause the LED display 66 to present latent information by flashing the LEDs that serve as objects through which the latent information is presented. In an example of Fig. 42(a), after the end of an off-period in which the brightness of the LEDs is set to 0%, a transition is made to an on-period in which the brightness of the LEDs is set to 100%. During the transition to the off-period following the end of the on-period, the brightness of the LEDs is set to 0%.

[0358] At this time, it is possible to increase the intensity of the attentional cue of the latent information by increasing the brightness of the LEDs in the on-phase and to decrease the intensity of the attentional cue of the latent information by reducing the brightness of the LEDs in the on-phase to a low value.

[0359] Alternatively, it is possible to increase the intensity of the attentional cue of the latent information by shortening the blinking cycle of the LEDs and to decrease the intensity of the attentional cue of the latent information by lengthening the blinking cycle of the LEDs.

[0360] Alternatively, it is possible to increase the intensity of the attentional cue of the latent information by setting the on-time of the LEDs longer than the off-time, and to decrease the intensity of the attentional cue of the latent information by setting the on-time of the LEDs shorter than the off-time.

[0361] In addition, the LEDs can be produced using a different process than in Fig. 42(a) can be switched on and off.

[0362] In an example from Fig. 42(b), after the end of the off-period, in which the brightness of the LEDs is set to 0%, a transition to the on-period is made by gradually increasing the brightness of the LEDs to 100%. During the transition to the off-period after the end of the on-period, the brightness of the LEDs is gradually reduced to 0%.

[0363] In this case, in addition to the method for adjusting the intensity of the attention request in Fig. 42(a), for example, the intensity of the attentional cue of the latent information can be increased when the increase / decrease in the brightness of the LEDs is steeper, and the intensity of the attentional cue of the latent information can be decreased when the increase / decrease in the brightness of the LEDs is smoother.

[0364] Fig. 43 is a schematic view showing an example in which presentation information generated by the driving assistance device 10b according to the fifth modification of the first embodiment is presented on the LED display 66 by lighting the LEDs in a plurality of colors.

[0365] As in Fig. As shown in Fig. 43(a), the driving assistance device 10b of the fifth modification can cause the LED display 66 to present latent information by periodically lighting the LEDs, which serve as objects through which the latent information is presented, in different colors. In an example of Fig. 43(a) the LEDs light up alternately in color X and color Y.

[0366] At this time, it is possible to increase the intensity of attentional cue of the latent information by increasing a difference in hue, saturation, brightness, or the like between colors X and Y, and to decrease the intensity of attentional cue of the latent information by decreasing the difference in hue, saturation, brightness, or the like between colors X and Y.

[0367] That is, to increase a hue difference between colors X and Y, for example, complementary colors can be used as the colors X and Y. In addition, to suppress the hue difference between colors X and Y so that it is small, for example, similar colors can be used as the colors X and Y.

[0368] For example, to increase a saturation difference between colors X and Y, a color with high saturation and a color with low saturation can be used in combination as colors X and Y. For example, to increase a brightness difference between colors X and Y, a color with high brightness and a color with low brightness can be used in combination as colors X and Y.

[0369] Furthermore, in the above case, the individual LEDs can alternately light up in color X or color Y on an array of LEDs, which serve as objects through which the latent information is presented. Furthermore, it is also possible to present the latent information in such a way that each of the colors X and Y flows on such an array of LEDs.

[0370] That is, at time t1, the odd LEDs on the array light up in color X and the even LEDs on the array light up in color Y. At the subsequent time t2, the odd LEDs light up in color Y and the even LEDs light up in color X. This is repeated alternately, which can achieve the visual effect described above.

[0371] In addition, it is also possible to adjust the intensity of the attentional request of the latent information to be presented on the LED display 66 by a method other than that described in Fig. 43(a).

[0372] In an example from Fig. 43(b), the driving assistance device 10b of the fifth modification sets a light-on period of color X longer than a light-on period of color Y.

[0373] At this time, it is possible to increase the intensity of the attentional cue of the latent information by using a color with high hue, saturation, or brightness as color X and a color with low hue, saturation, or brightness as color Y. Furthermore, it is possible to decrease the intensity of the attentional cue of the latent information by using a color with low hue, saturation, or brightness as color X and a color with high hue, saturation, or brightness as color Y.

[0374] Even with the above configuration, it is possible to adjust the intensity of the attention request of the latent information to be presented on the LED display 66.

[0375] Note that in the above example, the LED indicator 66 is provided at the bottom of the HUD 61, but the position of the LED indicator 66 is not limited thereto. For example, the LED indicator 66 may be provided at both ends of the HUD 61 instead of or in addition to the bottom of the HUD 61, or it may be provided to surround the outer periphery of the HUD 61.

[0376] Even in such a case, it is possible to present various kinds of latent information based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area where the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like in modes consistent with various modes set in Fig. 41 described above.

[0377] Fig. 44 is a schematic view showing an example of a configuration of the mirror display 67 on which the driving assistance device 10b according to the fifth modification of the first embodiment presents latent information.

[0378] As in Fig. As shown in FIG. 44, the mirror display 67 is installed, for example, at a position of a rearview mirror for checking the rear of the vehicle, instead of the rearview mirror. The mirror display 67 is configured, for example, as a rearview mirror and performs a function equivalent to the rearview mirror by projecting an image of the rear of the vehicle. The driving assistance device 10b of the fifth modification, for example, causes such a mirror display 67 to display latent information.

[0379] Fig. 45 is a schematic view showing an example of presentation of latent information on the mirror display 67 by the driving assistance device 10b according to the fifth modification of the first embodiment.

[0380] As described below, the presentation of the latent information using an outer edge region of the mirror display 67 can be carried out in a similar manner to the display of the latent information using the Fig. 30 shown meter display 64, which was described above, for example.

[0381] As in Fig. For example, as shown in Fig. 45(A), in a case where there is an area where the degree of inappropriateness of the driver's attention state is high over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays latent information 665 surrounding an outer peripheral area of ​​the mirror display 67 on the entire circumference of the mirror display 67. However, the latent information for requesting high attention with respect to the entire front of the vehicle may have a different form or mode than the latent information 665.

[0382] As in Fig. 45(Ba), for example, in an area where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, the driving assistance device 10b of the fifth modification displays a pair of latent information 669a at both the left and right ends of the mirror display 67.

[0383] For example, the latent information 669a may have a shape obtained by combining a pair of triangles pointing toward the vicinity of the center of the mirror display 67, which is comparable to the front of the vehicle. However, the latent information intended to draw faint attention to the entire front of the vehicle may also have other shapes or modes, such as an arrow.

[0384] As in Fig. 45(Bb), the driving assistance device 10b of the fifth modification may display, as another mode of the latent information 669a, for example, in a case where there is a region where the degree of inadequacy of the driver's attention state is relatively low over the entire front of the vehicle, latent information 669b that is triangular and points to the center of the mirror display 67 at a lower end of the center of the mirror display 67.

[0385] As in Fig. For example, as shown in Fig. 45(Ca), in a case where there is an area where the degree of inadequacy of the driver's attention state is high, the area deviating to the right side of the vehicle, the driving assistance device 10b of the fifth modification displays latent information 666a indicating an area on the left side opposite to the area at the left end of the mirror display 67.

[0386] For example, latent information 666a may be presentation information presented in a rectangular area along the left end of the mirror display 67, which is comparable to the area on the left side of the vehicle. However, latent information requesting high attention with respect to the driver with a large deviation in attention state may have a different shape or mode than latent information 666a.

[0387] As in Fig. 45(Cb), the driving assistance device 10b of the fifth modification may display, as another mode of the latent information 666a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle, latent information 666b indicating an area on the left side opposite to the area at a left lower end of the mirror display 67.

[0388] The latent information 666b may, for example, be presentation information arranged in a rectangular area along the lower left end of the mirror display 67, which is comparable to the area on the left side of the vehicle.

[0389] As in Fig. 45(Da), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is relatively low is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification displays, at the right end of the mirror display 67, in addition to the latent information 666a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 667a indicating an area on the right side opposite to the area on the left side of the vehicle.

[0390] For example, the latent information 667a may be presentation information arranged in a rectangular area along the right end of the mirror display 67, which is compared with the area on the right side of the vehicle. Furthermore, the latent information 667a is presented in a mode in which a brightness, color, or the like differs from that of the latent information 666a, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 666a.

[0391] It should be noted that the latent information for prompting weak attention with respect to the driver who has a small deviation in the attention state may have a different form or mode than the latent information 667a.

[0392] As in Fig. 45(Db), the driving assistance device 10b of the fifth modification may display, as another mode of the latent information 667a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is relatively low is located to deviate to the left side of the vehicle, in addition to the latent information 666b, latent information 667b indicating a right area versus the left area at the left lower end of the mirror display 67.

[0393] For example, latent information 667b may be presentation information arranged in a rectangular area along a lower right end of the mirror display 67, which is comparable to the area on the right side of the vehicle. Furthermore, latent information 667b has a mode in which the intensity of the attention request is lower than that of latent information 666b.

[0394] As in Fig. 45(Ea), for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located so as to deviate to the right side of the vehicle, and an area where the degree of inappropriateness of the driver's attention state is medium is located so as to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification displays, at the right end of the mirror display 67, in addition to the latent information 666a indicating an area on the left side opposite to the area on the right side of the vehicle, latent information 668a indicating an area on the right side opposite to the area on the left side of the vehicle.

[0395] For example, the latent information 668a may be presentation information presented in a rectangular area arranged along the right end of the mirror display 67, which is comparable to the area on the right side of the vehicle. Furthermore, the latent information 668a is presented in a mode in which brightness, color, or the like differs from those of the latent information 666a and 667a, so that the intensity of the attention request can be suppressed to be lower than that of the latent information 666a while being set higher than that of the latent information 667a.

[0396] It should be noted that the latent information for requesting moderate attention with respect to the driver having a moderate deviation in the attention state may have a different form or mode than the latent information 668a.

[0397] As in Fig. 45(Eb), in another mode of the latent information 668a, for example, in a case where an area where the degree of inappropriateness of the driver's attention state is high is located to deviate to the right side of the vehicle and an area where the degree of inappropriateness of the driver's attention state is medium is located to deviate to the left side of the vehicle, the driving assistance device 10b of the fifth modification may display, in addition to the latent information 666b, latent information 668b indicating an area on the right side opposite to the area on the left side, at the left lower end of the mirror display 67.

[0398] For example, latent information 668b may be presentation information located in a rectangular area along the lower right end of the mirror display 67, which is comparable to the area on the right side of the vehicle. Furthermore, latent information 668b has a mode in which the intensity of the attention request is higher than that of latent information 667b and the intensity of the attention request is lower than that of latent information 666b.

[0399] With the above configuration, various types of the latent information 665, 666a to 669a, 666b to 669b and the like can be displayed on the mirror display 67 based on the magnitude of the degree of inappropriateness of the driver's attention state, the position in the image of the area where the degree of inappropriateness of the attention state is equal to or greater than the predefined threshold, and the like.

[0400] According to the driving assistance device 10b of the fifth modification, two or more information presentation devices of the HUD 61, the in-vehicle monitor 62, the speaker 63, the meter display 64, the column display 65, the LED display 66, and the mirror display 67 are caused to present presentation information divided into a plurality of parts, respectively.

[0401] As a result, it is possible to present manifest information indicating a higher risk, e.g., on a component that has high visibility and easily attracts the driver's attention among the HUD 61, the in-vehicle monitor 62, the speaker 63, the meter display 64, the column display 65, the LED display 66, and the mirror display 67 of the various components included in the information presentation device 60, and to present latent information indicating a relatively low risk, e.g., on a configuration in which the information presentation is possible without distracting the driver's attention.

[0402] According to the driving assistance device 10b of the fifth modification, other effects similar to those of the driving assistance device 10 of the first embodiment described above are achieved.

[0403] It should be noted that in the driving assistance device 10b of the fifth modification described above, for example, the HUD 61 or the in-vehicle monitor 62 is caused to mainly present the manifest information, while one of the other configurations is caused to present the latent information, so that these configurations are combined to present information to the driver.

[0404] However, as in the driving assistance device 10 of the first embodiment described above, for example, in a configuration in which only the latent information is presented, any of the other configurations, such as the speaker 63, the meter display 64, the column display 65, the outer periphery of the HUD 61, the LED display 66, or the mirror display 67, may be used alone to present the latent information without being combined with the HUD 61, the in-vehicle monitor 62, or the like. (Sixth Modification)

[0405] Next, a driving assistance device according to a sixth modification of the first embodiment will be described with reference to FIG. Fig. 46 to 48. The driving assistance device of the sixth modification differs from the fifth modification described above in that a time-to-collision (TTC) is also taken into account when latent information and manifest information are presented.

[0406] Fig. 46 is a block diagram showing an example of a functional configuration of a driving assistance device 10c according to the sixth modification of the first embodiment together with peripheral devices.

[0407] As in Fig. 26, instead of the driving assistance device 10b according to the fifth modification described above, the driving assistance device 10c according to the sixth modification is installed in a vehicle 103 according to the sixth modification.

[0408] The driving assistance device 10c includes a TTC calculation unit 112 that calculates the TTC in addition to the configuration of the driving assistance device 10b of the fifth modification described above, and further includes an output control unit 130c that acquires the driver's attention state estimated by the driver attention state estimation unit 120, the manifest information generated by the manifest information calculation unit 111, and the TTC information calculated by the TTC calculation unit 112, and outputs driving assistance information instead of the output control unit 130b of the fifth modification described above.

[0409] The TTC calculated by the TTC calculation unit 112 is a time until the vehicle actually collides with an obstacle that is likely to collide with the vehicle 103, such as a pedestrian in front of the vehicle 103. If the TTC value decreases, it means that there is insufficient time until the collision between the obstacle and the vehicle 103. If the TTC value increases, it means that there is sufficient time left before the collision between the obstacle and the vehicle 103.

[0410] The TTC calculation unit 112 calculates the TTC for an obstacle such as a pedestrian included in an area as a target of information presentation from a distance between the vehicle 103 and the obstacle, a speed of the vehicle 103, and the like.

[0411] The driver attention state estimation unit 120 outputs the estimated attention state of the driver, the manifest information generated by the manifest information calculation unit 111, and the TTC information calculated by the TTC calculation unit 112 for each obstacle to the output control unit 130c.

[0412] The output control unit 130c outputs to the HMI control device 30 the driving assistance information including information about a region to which the driver's attention is to be drawn and extracted based on the driver's attention state and the manifest information transmitted from the driver attention state estimation unit 120.

[0413] Furthermore, the output control unit 130c determines whether or not to include latent information in the driving assistance information including the manifest information based on the TTC information calculated by the TTC calculation unit 112. If the TTC is equal to or greater than a predefined threshold, the output control unit 130c outputs the driving assistance information including the latent information to the HMI control device 30. If the TTC is less than the predefined threshold, the output control unit 130c outputs the driving assistance information to the HMI control device 30 without including the latent information.

[0414] Fig. 47 is a schematic view showing an example of presentation of information by the driving assistance device 10c according to the sixth modification of the first embodiment.

[0415] In the example of Fig. 47(a) and Fig. 47(b), for example, an area with a person on the near right and an area with a parent accompanying a child on the near left are assumed to be the areas to which the driver's attention should be directed. It is also assumed that the TTC for the person on the near right is equal to or greater than the predefined threshold, and the TTC for the parent accompanying the child on the near left is less than the predefined threshold.

[0416] As in Fig. 47(a) and Fig. 47(b), the driving assistance device 10c of the sixth modification causes the HUD 61 or the in-vehicle monitor 62 to display pieces of the above-described elliptical manifest information 662a indicating the person on the right side or the parent accompanying the child on the left side, respectively.

[0417] Furthermore, the driving assistance device 10c of the sixth modification causes the HUD 61 or the in-vehicle monitor 62 to display the above-described rectangular latent information 605a indicating the person on the near right side whose TTC is equal to or greater than the predefined threshold. On the other hand, the driving assistance device 10c according to the sixth modification does not display latent information for the parent accompanying the child whose TTC is less than the predefined threshold.

[0418] As described above, the latent information 605a is displayed in a state where the TTC is sufficiently long and there is sufficient time left before the collision, so that it is possible to prompt the driver to perform appropriate manipulation according to the latent information. On the other hand, the latent information is not displayed in a state where the TTC is short and the time until the collision is insufficient, so that it is possible to prevent the driver from being distracted by the latent information and preventing him from performing appropriate manipulation.

[0419] However, the forms, presentation positions, presentation modes and the like of the manifest information 662a and the latent information 605a are not limited to the above-described example of Fig. 47 limited.

[0420] Fig. 48 is a schematic view showing another example of information presentation by the driving assistance device 10c according to the sixth modification of the first embodiment.

[0421] In the Fig. 48(a) and Fig. In the example shown in Fig. 48(b), the driving assistance device 10c of the sixth modification displays portions of the rectangular manifest information 662b indicating the person on the near right side and the parent accompanying the child on the near left side, so as to surround the person and the parent accompanying the child, respectively. Furthermore, the driving assistance device 10c of the sixth modification displays the latent information 605b, which is elliptical and indicates the person on the near right side, at a lower position away from the person's feet.

[0422] Note that the driving assistance device 10c of the sixth modification may, for example, change the size, brightness, color, or the like of each piece of the manifest information 662a and 662b according to a length of the TTC to adjust the degree of intensity of the driver's attention request. Furthermore, the driving assistance device 10c of the sixth modification may adjust the degree of intensity of the driver's attention request by changing the size, brightness, color, or the like of each piece of the latent information 605a and 605b according to the degree of inappropriateness of the driver's attention state.

[0423] As described above, the forms, presentation positions, presentation modes, and the like of the manifest information 662 and the latent information 605 can be presented in different modes.

[0424] It should be noted that the driving assistance device 10c of the sixth modification may also cause various information presentation devices 60 to present the manifest information and the latent information, respectively, as shown in the Fig. 29 to 45 and the like of the fifth modification described above.

[0425] In addition, the driving assistance device 10c of the sixth modification can present information based on the prediction error and the TTC, which is different from that of the above-described examples of the Fig. 47 and Fig. 48 differ.

[0426] Fig. 49 is a schematic view showing another example of information presentation by the driving assistance device 10c according to the sixth modification of the first embodiment. In the example of Fig. 49, the driving assistance device 10c of the sixth modification presents latent information with different intensity to attract the driver's attention based on indexes obtained by combining the magnitude of the prediction error and the length of the TTC.

[0427] Fig. 49(a) shows an example of a presentation mode table of the latent information given to the driving assistance device 10c of the sixth modification. As shown in Fig. As shown in Figure 49(a), the latent information presentation mode table defines a plurality of latent information modes with different attentional cue intensities. These latent information attentional cue intensities are determined by two parameters: the prediction error and the TTC.

[0428] That is, three thresholds are set for the prediction error: low, medium, and high, and three thresholds are set for the TTC: short, medium, and long. In the latent information presentation mode table, latent information modes are defined for nine patterns, which are obtained by combining cases where the prediction error is low, medium, and high and cases where the TTC is short, medium, and long.

[0429] Among these nine patterns, the intensity of the attentional cue of latent information is lowest when the prediction error is low and the TTC is long. Furthermore, the intensity of the attentional cue of latent information is highest when the prediction error is low and the TTC is short, when the prediction error is medium and the TTC is short, and when the prediction error is high and the TTC is short or medium.

[0430] Furthermore, in a case where the prediction error is medium and the TTC is long, the intensity of the attentional demand of the latent information is higher than in a case where the prediction error is small and the TTC is long. Furthermore, in a case where the prediction error is small and the TTC is medium, and in a case where both the prediction error and the TTC are medium, the intensity of the attentional demand of the latent information is higher than in a case where the prediction error is medium and the TTC is long. Furthermore, in a case where the prediction error is high and the TTC is long, the intensity of the attentional demand of the latent information is higher than in a case where the prediction error is low and the TTC is medium, or the like.

[0431] As described above, in the latent information presentation mode table, four modes of latent information are defined for the nine patterns, which are obtained by combining the magnitude of the prediction error and the length of the TTC. Fig. However, the definition of the modes of latent information shown in Figure 49(a) is only an example, and different modes of latent information can be defined.

[0432] For example, the levels of prediction error magnitude and TTC length may each be two levels or four or more levels instead of the above three levels. Furthermore, there may be, for example, three or fewer or five or more modes of latent information instead of the above four modes for the combination patterns of prediction error magnitude and TTC length.

[0433] The driving assistance device 10c of the sixth modification determines a presentation mode of the latent information, for example, according to the Fig. 49(a) shown table of presentation modes of latent information. Fig. 49(Aa) to 49(Cb) show examples of a case where the latent information is displayed on the HUD 61 or the in-vehicle monitor 62.

[0434] As in Fig. 49, the driving assistance device 10c of the sixth modification divides an image into, for example, left and right areas and calculates a prediction error and a TTC for each of the areas.

[0435] In the examples of Fig. 49(Aa) and 49(Ba), it is assumed that both the prediction error and the TTC are classified as medium for the left area with a parent accompanying a child on the near left side, and that the prediction error is classified as low and the TTC is classified as long for the right area containing no person.

[0436] In this case, the driving assistance device 10c of the sixth modification displays latent information 676b arranged in a rectangular area along a lower end of the left area. The latent information 676b has the intensity of the attention request for the case where both the prediction error and the TTC are medium.

[0437] Furthermore, the driving assistance device 10c of the sixth modification displays latent information 676a arranged in a rectangular area along a lower end of the right area. The latent information 676a has the intensity of the attention prompt for a case where the prediction error is small and the TTC is long.

[0438] In the examples of Fig. 49(Ab) and 49(Bb), it is assumed that the prediction error is found to be high and the TTC is long for the left region which contains a parent accompanying a child on the near left side and which contains two people on the far left side, and that the prediction error is found to be low and the TTC is long for the right region which does not contain any person.

[0439] In this case, the driving assistance device 10c of the sixth modification displays latent information 676c arranged in a rectangular area along a lower end of the left area. The latent information 676c has the intensity of the attention prompt for a case where the prediction error is high and the TTC is long.

[0440] In addition, the driving assistance device 10c of the sixth modification displays the latent information 676a extending along a lower end of the right area, similarly to the above-described examples of Fig. 49(Aa) and 49(Ba).

[0441] It should be noted that the driving assistance device 10c of the sixth modification can adjust the degree of the intensity of the driver's attention request by changing, for example, the length, thickness, brightness, color, or the like of each piece of the latent information 676a to 676c when the latent information 676a to 676c or the like is visually displayed on the HUD 61, the in-vehicle monitor 62, or the like, as shown in Fig. 49 described above.

[0442] Since the latent information is presented based on the two parameters of the prediction error and the TTC, the driving assistance information based on more information can be generated with high accuracy, and the presentation mode can be simplified by using, for example, only the latent information 676a to 676c.

[0443] In the driving assistance device 10c of the sixth modification, in addition to the effects described above, similar effects to those of the driving assistance device 10b of the fifth modification are achieved. (Seventh Modification)

[0444] Next, a driving assistance device 10d according to a seventh modification of the first embodiment will be described with reference to Fig. 50. The driving assistance device 10d of the seventh modification differs from that of the first embodiment described above in that the driving assistance information is output to an external server 90.

[0445] Fig. 50 is a block diagram showing an example of a functional configuration of the driving assistance device 10d according to the seventh modification of the first embodiment together with peripheral devices.

[0446] As in Fig. 50, instead of the driving assistance device 10 according to the first embodiment described above, the driving assistance device 10d according to the seventh modification is installed in a vehicle 104 according to the seventh modification. The driving assistance device 10d includes an output control unit 130d that outputs the driving assistance information to the external server 90, instead of the output control unit 130 of the first embodiment described above.

[0447] The output control unit 130d generates the driving assistance information associated with an area to which the driver's attention is to be drawn, and outputs the generated driving assistance information to the external server 90.

[0448] The external server 90 is connected to a plurality of vehicles, including the vehicle 104 of the seventh modification, to be able to send and receive information to and from each other, for example, via a wireless local area network (LAN) or the like. The external server 90 acquires and collects driving assistance information generated by the driving assistance devices of the plurality of vehicles, including the driving assistance device 10d of the seventh modification.

[0449] The driving assistance information collected in the external server 90 is stored, for example, in a database to be used in driving assistance to a driver of another vehicle. Consequently, driving assistance information generated by a predefined vehicle can be shared among the plurality of vehicles to provide driving assistance to drivers. Furthermore, since the driving assistance information collected from the plurality of vehicles is stored in the database, the accuracy of the driving assistance can be improved.

[0450] According to the driving assistance device 10d of the seventh modification, other effects similar to those of the driving assistance device 10 of the first embodiment described above are achieved. [Second embodiment]

[0451] A second embodiment will be described with reference to the drawings. The second embodiment differs from the above-described first embodiment in that a driving assistance device generates driving assistance information taking into account a driver's viewpoint.

[0452] Fig. 51 is a block diagram showing an example of a functional configuration of a driving assistance device 210 according to the second embodiment together with peripheral devices.

[0453] As in Fig. 51, instead of the driving assistance device 10 according to the first embodiment described above, the driving assistance device 210 according to the second embodiment is installed in a vehicle 200 according to the second embodiment.

[0454] The driving assistance device 210 includes a viewpoint estimation unit 140 that estimates a driver's viewpoint in addition to the configuration of the driving assistance device 10 of the first embodiment described above, and further includes a driver attention state estimation unit 220 that receives information from the prediction error calculation unit 110 and the viewpoint estimation unit 140 and estimates a driver's attention state instead of the driver attention state estimation unit 120 of the first embodiment described above.

[0455] The viewpoint estimation unit 140 estimates the driver's viewpoint from a face image of the driver captured, for example, by the driver monitoring camera 42. Specifically, when an image in a traveling direction of the vehicle 200 captured by the vehicle exterior camera 41 is divided into a plurality of regions, for example, as described above, the viewpoint estimation unit 140 estimates a region to which the driver's gaze is directed from the plurality of regions.

[0456] Similar to the driver attention state estimation unit 120 of the first embodiment described above, the driver attention state estimation unit 220 calculates a degree of probability that the driver's attention is attracted in the plurality of areas in the image. Furthermore, the driver attention state estimation unit 220 estimates the driver's attention state for each of the plurality of areas based on a magnitude of a degree of inappropriateness of the attention state in each of the plurality of areas and the direction of the driver's gaze point estimated by the gaze point estimation unit 140.

[0457] Here the driver tends, as in Fig. 4 of the first embodiment, to pay attention to an area with a large prediction error. According to the Fig. 4, the driver attention state estimation unit 220 estimates the probability of attracting the driver's attention based on a prediction error for an area to which the driver is looking and a prediction error for any area to which the driver is not looking.

[0458] That is, the driver attention state estimation unit 220 extracts the area to which the driver is looking and the arbitrary area, calculates an AUC value for the area to which the driver is looking from a ratio in which the prediction error of each of the areas is equal to or larger than a predefined threshold, and uses information including an area in which the AUC value is larger than a predefined threshold to estimate the driver's attention state.

[0459] Since, as described above, not only the prediction error but also the driver's gaze point is taken into account, it is possible to estimate with greater accuracy whether the driver's attention state is appropriate for a given area. Note that the AUC value calculated as described above is also referred to as the attention state index.

[0460] Fig. Fig. 52 is a flowchart showing an example of a method of driving assistance processing performed by the driving assistance device 210 according to the second embodiment. Note that in the Fig. 52, the step S101 is a process newly added to the processing shown in Fig. 7 shown processing of the first embodiment described above.

[0461] As in Fig. As shown in Figure 52, the gaze point estimation unit 140 included in the driving assistance device 210 according to the second embodiment estimates a driver's gaze target from a driver's facial image captured by the driver monitoring camera 42 (step S101). Furthermore, the driver attention state estimation unit 220 divides an image captured by the vehicle exterior camera 41 into a plurality of regions (step S110) and calculates a prediction error for each of the regions (step S120).

[0462] In addition, when an AUC value of a predefined area obtained from the calculated prediction error and the estimated viewpoint, that is, an attention state index, is larger than a threshold TH21 (step S130: Yes), the driver attention state estimation unit 220 estimates that a degree of inappropriateness of the driver's attention state for the area is high (step S132).

[0463] Based on such an estimation by the driver attention state estimation unit 220, the output control unit 130 selects the mode ST1 with the high intensity of the attention request for the presentation of information in an area opposite to the area.

[0464] When the attention state index of the predefined area is greater than a threshold TH22 which is smaller than the threshold TH21 (step S130: No, step S140: Yes), the driver attention state estimation unit 220 estimates that the degree of inappropriateness of the driver's attention state for the area is medium (step S142).

[0465] Based on such an estimation by the driver attention state estimation unit 220, the output control unit 130 selects the mode ST2 with the medium intensity of the attention request for the presentation of driving information in the area opposite to the area.

[0466] When the attention state index of the predefined area is greater than a threshold TH23 which is smaller than the threshold TH22 (step S140: No, step S150: Yes), the driver attention state estimation unit 220 estimates that the degree of inappropriateness of the driver's attention state for the area is low (step S152).

[0467] Based on such an estimation by the driver attention state estimation unit 220, the output control unit 130 selects the mode ST2 with the low intensity of the attention request for presenting driving information in the area opposite to the area.

[0468] When the attention state index of the predefined area is smaller than the threshold TH23 (step S150: No), the driver attention state estimation unit 220 determines that the driver's attention state is sufficiently suitable for the area, and the output control unit 130 does not generate driving assistance information for presentation in the area opposite to the area.

[0469] The driving assistance device 210 of the second embodiment repeats the above processing for all areas (step S160: No), and after completing the processing for all areas (step S160: Yes), outputs the generated driving assistance information to the HMI control device 30 (step S170).

[0470] This ends the driving assistance processing by the driving assistance device 210 of the second embodiment.

[0471] As described above, the driving assistance device 210 divides the image into a plurality of regions and performs various processes in the above description, which is similar to the driving assistance device 10 of the first embodiment. However, in the second embodiment, it is only necessary to extract the driver's gaze point and each other part, and it is not always necessary to divide the image into the plurality of regions. That is, the above processing can be performed by performing extraction for each of the pixels in the image, for example, a pixel to which the driver's gaze is directed and a pixel to which the driver's gaze is not directed.

[0472] Note that the information included in the driving assistance information output from the driving assistance device 210 of the second embodiment to the HMI control device 30 as described above can be presented by, for example, the information presentation device 60 in the various modes described in the first embodiment and the first to third modifications.

[0473] As a result, in a state where attention is likely to be attracted, the state being estimated by the attention state estimation unit, the visual behavior can be unconsciously changed to guide the driver into a suitable attention state, and it is possible to attract attention to make the driver alert.

[0474] At this time, the driving assistance device 210 of the second embodiment may be equipped with the manifest information calculation unit 111, the TTC calculation unit 112, or the like, similar to the fifth or sixth modification or the like of the first embodiment described above, so that latent information and manifest information included in the driving assistance information can be presented.

[0475] Alternatively, the driving assistance device 210 of the second embodiment may be configured to output the driving assistance information to the ECU 20, the external server 90, or the like, similarly to the above-described fourth or seventh modification or the like of the first embodiment.

[0476] According to the driving assistance device 210 of the second embodiment, the driver's attention state is estimated based on the driver's gaze point in addition to the prediction error. This makes it possible to estimate the driver's attention state with higher accuracy and present more appropriate information to the driver.

[0477] According to the driving assistance device 210 of the second embodiment, the probability of attracting attention is estimated based on the prediction error of the area to which the driver is looking and the prediction error of the arbitrary area among the plurality of areas in the image obtained by detecting the traveling direction of the vehicle 104.

[0478] For example, even in an area with high prediction error, the presentation of unnecessary information can be avoided if the driver's attention level does not deviate.

[0479] With the driving assistance device 210 of the second modification, other effects similar to those of the driving assistance device 10 of the first embodiment described above are achieved. [Third Embodiment]

[0480] A third embodiment will be described with reference to the drawings. The third embodiment differs from the above-described first embodiment in that a driving assistance device generates driving assistance information taking into account a driver's skill level.

[0481] Fig. 53 is a block diagram showing an example of a functional configuration of a driving assistance device 310 according to the third embodiment together with peripheral devices.

[0482] As in Fig. 53, instead of the driving assistance device 10 according to the first embodiment described above, the driving assistance device 310 according to the third embodiment is installed in a vehicle 300 according to the third embodiment.

[0483] The driving assistance device 310 includes a viewpoint estimation unit 140 and a driver skill level determination unit 150 that determines the skill level of the driver in addition to the above-described configuration of the driving assistance device 10 of the first embodiment, and further includes a driver attention state estimation unit 320 that acquires information from the prediction error calculation unit 110, the viewpoint estimation unit 140, and the driver skill level determination unit 150 to estimate an attention state of the driver, instead of the driver attention state estimation unit 120 of the above-described first embodiment.

[0484] The driver competence level determination unit 150 determines the competence level of the driver, for example, based on various detection results obtained by the ECU 20 from the detection device 40 (see Fig. 1). From the various detection results of the detection device 40, it is possible to detect a situation of driving manipulation of the vehicle 104 by the driver. Therefore, based on a detection result in which, for example, sudden steering, sudden starting, and sudden stopping frequently occur, it is possible to estimate that the driver's skill level is low.

[0485] The driver attention state estimation unit 320 estimates the driver's attention state for each of a plurality of areas based on the magnitude of a degree of inadequacy of the driver's attention state based on a prediction error in each of the plurality of areas and a driver's gaze point estimated by the gaze point estimation unit 140, similarly to the driver attention state estimation unit 220 of the second embodiment described above.

[0486] At this time, the driver attention state estimation unit 320 changes the threshold values ​​of the attention state indices in the plurality of driver attention state estimation areas according to the driver's proficiency level. That is, the driver attention state estimation unit 320 sets the threshold values ​​of the attention state indices low when the driver's proficiency level is low. Furthermore, the driver attention state estimation unit 320 sets the threshold values ​​of the attention state indices high when the driver's proficiency level is high.

[0487] Such adjustment by the driver attention state estimation unit 320 is based on the fact that a driver with a lower skill level is more likely to focus his gaze on an area with a large prediction error than a driver with a higher skill level.

[0488] Fig. Fig. 54 is a flowchart showing an example of a flow of the driving assistance processing performed by the driving assistance device 310 according to the third embodiment. It should be noted that in the Fig. 54, steps S102 and S103 are processes that are new to the processing shown in Fig. 52 shown processing of the second embodiment described above.

[0489] As in Fig.As shown in Figure 54, the viewpoint estimation unit 140 included in the driving assistance device 310 according to the third embodiment estimates a driver's viewpoint from a driver's facial image acquired by the driver monitoring camera 42 (step S101). Furthermore, the driver proficiency level determination unit 150 receives detection results of the detection device 40 from the ECU 20 and determines a driver's proficiency level based on these detection results (step S102).

[0490] The driver attention state estimation unit 320 changes the setting of the threshold values ​​of the attention state indexes calculated from the prediction error, the driver's gaze point, and the like, based on a determination result of the driver skill level determination unit 150 (step S103).

[0491] If it is determined that the driver's proficiency level is low, the driver alertness state estimation unit 320 sets a threshold TH31, a threshold TH32 greater than the threshold TH31, and a threshold TH33 greater than the threshold TH32. If it is determined that the driver's proficiency level is high, the driver alertness state estimation unit 320 sets a threshold TH34 greater than the threshold TH31, a threshold TH35 greater than the thresholds TH32 and TH34, and a threshold TH36 greater than the thresholds TH33 and TH35.

[0492] Furthermore, the driver attention state estimation unit 220 divides an image captured by the vehicle exterior camera 41 into a plurality of areas (step S110) and calculates a prediction error for each of the areas (step S120).

[0493] Furthermore, the driver attention state estimation unit 220 sorts the attention state indexes calculated from the prediction error and the driver's gaze point according to the threshold values ​​TH31 to TH33 or the threshold values ​​TH34 to TH36 set based on the driver's proficiency level (steps S130, S140, and S150), and estimates the driver's attention state based on these attention state indexes (steps S132, S142, and S152).

[0494] The output control unit 130 selects the latent information presentation modes ST1 to ST3 based on an estimation result of the driver attention state estimation unit 220 (steps S133, S143, and S153).

[0495] The driving assistance device 310 of the third embodiment repeats the above processing for all areas (step S160: No), and after completing the processing for all areas (step S160: Yes), outputs the generated driving assistance information to the HMI control device 30 (step S170).

[0496] This ends the driving assistance processing by the driving assistance device 310 of the third embodiment.

[0497] Note that the information included in the driving assistance information output from the driving assistance device 310 of the third embodiment to the HMI control device 30 as described above can be presented by, for example, the information presentation device 60 in the various modes described in the first embodiment and the first to third modifications.

[0498] At this time, the driving assistance device 310 of the third embodiment may be equipped with the manifest information calculation unit 111, the TTC calculation unit 112, or the like, similar to the fifth or sixth modification or the like of the first embodiment described above, so that latent information and manifest information included in the driving assistance information can be presented.

[0499] Alternatively, the driving assistance device 310 of the third embodiment may be configured to output the driving assistance information to the ECU 20, the external server 90, or the like, similarly to the fourth or seventh modification or the like of the first embodiment described above.

[0500] According to the driving assistance device 310 of the third embodiment, the driver's attention level is estimated based on the driver's driving skill level in addition to the prediction error. This makes it possible to estimate the attention level with higher accuracy according to the individual driver.

[0501] According to the driving assistance device 310 of the third embodiment, an area in which the attention state index is greater than one of the threshold values ​​TH31 to TH33 is extracted from the plurality of areas when the driver's proficiency level is low, and an area in which the attention state index exceeds one of the threshold values ​​TH34 to TH36 is extracted from the plurality of areas when the driver's proficiency level is high.

[0502] This makes it possible to detect the degree of inadequacy of the driver's attention level depending on the driver's skill level. Therefore, it is possible to provide appropriate information presentation according to the individual driver.

[0503] Note that in the third embodiment described above, the driving assistance device 310 includes the viewpoint estimation unit 140. However, the driving assistance device 310 does not necessarily need to include the viewpoint estimation unit 140, and in this case, the driving assistance device 310 according to the third embodiment may be configured to perform processing for switching the setting of the threshold values ​​according to the driver's skill level, in addition to the processing of the driving assistance device 10 according to the first embodiment.

[0504] In the first to third embodiments and the first to seventh modifications described above, the driving assistance device 10, 210, 310, or the like determines the magnitude of an index calculated from the prediction error and the like based on, for example, three threshold values. However, the number of threshold values ​​set for such an index may be two or fewer, or four or more.

[0505] Furthermore, in the first to third embodiments and the first to seventh modifications described above, the driving assistance device 10, 210, 310, or the like is configured as a device such as a control device. However, the functions described in the first to third embodiments and the first to seventh modifications described above may be implemented by a driving assistance system configured by combining a plurality of devices. In this case, a device implementing some functions may be provided outside the vehicle.

[0506] Although some embodiments of the present invention have been described, these embodiments have been presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other modes, and various omissions, substitutions, and changes can be made within a scope that does not deviate from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention and are also included in the invention described in the claims and the corresponding scope thereof. EXPLANATIONS OF LETTERS OR NUMBERS 10, 10a, 10b, 10c, 10d, 210, 310 DRIVING ASSISTANCE DEVICE 20 ECU 30 HMI CONTROL DEVICE 40 DETECTION DEVICE 41 VEHICLE EXTERIOR CAMERA 42 DRIVER MONITORING CAMERA 50 VEHICLE CONTROL DEVICE 51 BRAKE ACTUATOR 52 ENGINE CONTROL 60 INFORMATION PRESENTATION DEVICE 61 HUD 62 VEHICLE MONITOR 63 SPEAKERS 64 METER DISPLAY 65 COLUMN DISPLAY 66 LED DISPLAY 67 MIRROR DISPLAY 90 EXTERNAL SERVER 100, 101 to 104, 200, 300 VEHICLE 110 PREDICTION ERROR CALCULATION UNIT 111 MANIFEST INFORMATION CALCULATION UNIT 112 TTC CALCULATION UNIT 120, 220, 320 DRIVER ATTENTION ASSESSMENT UNIT 130, 130a, 130b, 130c, 130d OUTPUT CONTROL UNIT QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2021-130389 A

[0003] WO 2020 / 208804 A

[0003] Cited non-patent literature

[0000] Lotter, W., Kreiman, G., and Cox, D., “Deep predictive coding networks for video prediction and unsupervised learning,” https: / / ar-xiv.org / abs / 1605.08104

[0045]

Claims

[1] Driving assistance device comprising: a prediction error calculation unit that calculates a prediction error that is a difference between a predicted image and an actual image, the predicted image being predicted from an image in a traveling direction of a vehicle captured by a vehicle exterior camera that captures a periphery of the vehicle, an actual situation being captured in the actual image; an attention state estimation unit that estimates an attention state of a driver based on the prediction error; and an output control unit that outputs driving assistance information to prompt an attention state and / or a behavior change and / or a consciousness change with respect to a driving manipulation that are more suitable for driving at that time based on the attention state. [2] Driving assistance device according to claim 1, wherein the attention state estimation unit extracting a region in which the prediction error is greater than a predefined threshold from a plurality of regions in the image, and estimating one or more states in which the extracted region deviates from a center of the plurality of regions, is excessively concentrated in one part, and / or is excessively scattered over the entire region. [3] Driving assistance device according to claim 1, wherein the attention state estimation unit extracts a region in which a temporal variance value of the prediction error is greater than a predefined threshold from a plurality of regions of a temporally continuous image group, and estimates one or more states in which the extracted region deviates from a center of the plurality of regions, is excessively concentrated in one part, and / or is excessively scattered over the entire region. [4] Driving assistance device according to one of claims 1 to 3, wherein the output control unit Outputs driving assistance information for unconsciously changing visual behavior to correct deviation when the attention state estimated by the attention state estimation unit deviates, to disperse excessive concentration when there is excessive concentration on a part, and to concentrate excessive dispersion when there is excessive dispersion in the entire area. [5] Driving assistance device according to one of claims 1 to 3, wherein the output control unit Outputs driving assistance information for attracting attention to correct deviation when the attention state estimated by the attention state estimation unit deviates, to disperse excessive concentration when there is excessive concentration in a part, and to concentrate excessive dispersion when there is excessive dispersion in the entire area. [6] Driving assistance device according to one of claims 1 to 5, further comprising a gaze point estimation unit that estimates, from a face image of the driver captured by a driver monitoring camera that captures an image of an interior of the vehicle, a region to which a driver's gaze is directed in the prediction error, wherein the attention state estimation unit estimates the driver's attention level based on the direction of gaze in addition to the prediction error. [7] Driving assistance device according to claim 6, wherein the attention state estimation unit estimates the probability of attracting attention based on a prediction error for an area to which the driver's gaze is directed and a prediction error for any area to which the gaze is not directed. [8] Driving assistance device according to claim 7, wherein the output control unit Outputs driving assistance information to unconsciously change the visual behavior to induce an appropriate attention state when the probability of attracting attention is high in the attention state estimated by the attention state estimation unit. [9] Driving assistance device according to claim 7, wherein the output control unit Driver assistance information for attracting attention to alert the driver when the probability of attracting attention is high in the attention state estimated by the attention state estimation unit. [10] Driving assistance device according to one of claims 1 to 9, further comprising a driving skill level determination unit that determines a driving skill level by the driver based on manipulation information of the vehicle by the driver obtained from a detection device that detects a state of each part of the vehicle, wherein the attention state estimation unit controls the output control unit based on the competency level in addition to the prediction error. [11] Driving assistance device according to claim 10, wherein the output control unit in a more conspicuous mode in a case where the proficiency level is lower, outputs driving assistance information for unconsciously changing the visual behavior to correct a deviation when the attention state estimated by the attention state estimation unit deviates, to disperse excessive concentration when there is excessive concentration on a part, and to concentrate excessive dispersion when there is excessive dispersion in the whole area. [12] Driving assistance device according to claim 10, wherein the output control unit in a more conspicuous mode in a case where the proficiency level is lower, outputs driving assistance information for attracting attention to correct a deviation when the attention state estimated by the attention state estimation unit deviates, to disperse excessive concentration when there is excessive concentration on a part, and to concentrate excessive dispersion when there is excessive dispersion in the entire range. [13] Driving assistance device according to one of claims 1 to 12, wherein the output control unit outputs the driving assistance information to an information presentation control device that controls an information presentation device that presents information to the driver, the information presentation device a head-up display and an in-vehicle monitor, a meter display, a column display, a mirror display, a light-emitting device and a loudspeaker mounted in the vehicle, and the information presentation control device causes at least one of the head-up display, the in-vehicle monitor, the meter display, the column display, the mirror display, the light emitting device, and the speaker to present the information. [14] Driving assistance device according to one of claims 1 to 12, wherein the output control unit outputs the driving assistance information to an information presentation control device that controls an information presentation device that presents information to the driver, the information presentation device a head-up display and an in-vehicle monitor, a meter display, a column display, a mirror display, a light-emitting device and a loudspeaker mounted in the vehicle, and the information presentation control device causes two or more information presentation devices among the head-up display, the in-vehicle monitor, the meter gauge, the column gauge, the mirror gauge, the light emitting device, and the speaker to each present the information divided into a plurality of parts. [15] Driving assistance device according to one of claims 1 to 14, wherein the output control unit to an on-board electronic control unit that controls the vehicle, outputs operating information associated with an area to which the driver's attention is to be drawn, and gives an instruction for operating the vehicle, and the vehicle's own electronic control unit causes a vehicle control device to decelerate the vehicle to perform the operation based on the operation information. [16] Driving assistance device according to claim 15, wherein the vehicle control device a braking device that brakes the vehicle and an engine control device that performs control of an engine output of the vehicle, and the vehicle's electronic control unit brakes the vehicle by performing at least one of the following procedures Braking of the vehicle by the braking device, and Reducing the acceleration of the vehicle by the engine control device. [17] Driving assistance device according to claim 15 or 16, wherein the output control unit which outputs operating information with a higher braking effect of the vehicle when the prediction error on which the area to which the driver's attention should be directed is based is larger. [18] Driving assistance device according to one of claims 1 to 17, wherein the output control unit Outputs attention state information and vehicle manipulation information collected from a vehicle network to an external server that collects the attention state information. [19] Driving assistance device according to claim 18, wherein the external server evaluates whether the driver performs appropriate vehicle manipulation for the attention state based on information about the attention state collected from a plurality of vehicles including the vehicle and the vehicle manipulation information, and outputs an evaluation result to the driving assistance device. [20] Driver assistance system comprising: a prediction error calculation unit that calculates a prediction error that is a difference between a predicted image and an actual image, the predicted image being predicted from an image in a traveling direction of a vehicle, the image being captured by a vehicle exterior camera capturing a periphery of the vehicle, an actual situation being captured in the actual image; an attention state estimation unit that estimates an attention state of a driver based on the prediction error; and an output control unit that outputs driving assistance information to prompt an attention state and / or a behavior change and / or a consciousness change with respect to a driving manipulation that are more suitable for driving at that time based on the attention state. [21] Driver assistance system comprising: Calculating a prediction error that is a difference between a predicted image and an actual image, the predicted image being predicted from an image in a traveling direction of a vehicle captured by a vehicle exterior camera capturing a periphery of the vehicle, an actual situation being captured in the actual image; Estimating a driver’s attention state based on the prediction error; and Outputting driving assistance information to prompt a state of attention and / or a change in behavior and / or a change in awareness with respect to a driving manipulation that is more appropriate for driving at that time based on the state of attention.

Images