Driving assistance device, driving assistance method, and driving assistance program

Abstract

To provide qualified driving support information according to the surrounding conditions of a moving object.SOLUTION: A detection unit detects a latent recognition area that is unconsciously recognized by a human being out of a total imaging area of a captured image based on the captured image of at least a direction of travel of the moving object captured by an external camera installed on the moving object. An information presentation control unit presents predetermined driving support information in an area on an information presentation unit that is estimated according to the detected latent recognition area. As an example, the detection unit detects, as a latent recognition area, a divided area that has a predetermined feature amount exceeding a predetermined threshold among each divided area in which the total captured area is divided into multiple divided areas. For example, a prediction error which is a difference between a current image predicted from a past image and an actual current image, or an edge part of an object in the captured image is used as a feature amount. By presenting driving support information based on latent recognition areas that the human being unconsciously recognizes, accurate driving support information can be provided to drivers.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a driving assistance device, a driving assistance method, and a driving assistance program. [Background technology]

[0002] Drivers of moving objects such as vehicles drive their vehicles in accordance with traffic laws and regulations, paying attention to pedestrians and obstacles, based on traffic lights, road signs, lanes, etc. Because road conditions on which vehicles travel change constantly, being able to present information to assist driving in response to changes in road conditions can contribute to safe driving, etc.

[0003] Patent Document 1 (JP 2019-096109 A) discloses a driving assistance device that aims to provide assistance information for the next driving scene appropriately in accordance with the traffic environment around the vehicle. In this driving assistance device, a controller estimates the timing of the driver's cognitive behavior for the next driving scene from information related to at least the driver's driving operation, and provides assistance information corresponding to the next driving scene before the estimated timing of the cognitive behavior. In this case, the controller estimates the required driving ability required of the driver in accordance with the traffic environment around the vehicle, and provides a reduced amount of assistance information as the required driving ability becomes higher (the traffic environment becomes more complex). This makes it possible to provide assistance information that is appropriate for the traffic environment around the vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-096109 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-086355 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the driving assistance device disclosed in Patent Document 1 provides assistance information at predetermined times, so the provision of assistance information is, so to speak, fixed, which makes it difficult in practice to present appropriate driving assistance information according to the surrounding conditions.

[0006] The present disclosure provides a driving assistance device, a driving assistance method, and a driving assistance program that can provide appropriate driving assistance information according to the surrounding conditions of a moving object. [Means for solving the problem]

[0007] The driving assistance device according to the present disclosure includes a detection unit that detects a potential recognition area that humans unconsciously recognize within the entire imaging area of ​​the captured image based on an image of at least the moving direction of the moving body captured by an external camera unit provided on the moving body, and an information presentation control unit that presents predetermined driving assistance information in an area on the information presentation unit that is estimated according to the detected potential recognition area. [Effects of the Invention]

[0008] The driving assistance device according to the present disclosure can provide appropriate driving assistance information according to the situation around the moving object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing the flow from when a driver senses the situation around the vehicle using their senses such as sight and hearing to when they operate the vehicle. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of the driving assistance device according to the first to eighth embodiments. [Figure 3] FIG. 3 is a functional block diagram of each function realized by the control unit of the driving assistance device according to the first embodiment executing the driving assistance program stored in the storage unit. [Figure 4] FIG. 4 is a flowchart for explaining the operation of presenting driving support information. [Figure 5] FIG. 5 is a schematic diagram showing the flow of the operation for generating a prediction error image. [Figure 6] FIG. 6 is a diagram showing a first threshold (High) for the upper limit of the sum of prediction errors and a second threshold (Low) for the lower limit of the sum of prediction errors shown in FIG. [Figure 7] FIG. 7 is a diagram schematically showing a flow up to displaying the driving support information on the information presentation unit. [Figure 8] FIG. 8 is a diagram for explaining the inconvenience that occurs when driving support information is not presented. [Figure 9] FIG. 9 is a diagram illustrating an example of presentation of driving support information in the driving support device according to the embodiment. [Figure 10] FIG. 10 is a flowchart for explaining the operation of determining the display position of the driving support information for the position of the divided area with a large prediction error. [Figure 11] FIG. 11 is a diagram showing a case where a large number of prediction errors occur on either the left or right side of the screen presenting driving support information. [Figure 12] FIG. 12 is a diagram showing an example of display of driving support information. [Figure 13] FIG. 13 is a diagram showing another display example of the driving support information. [Figure 14] FIG. 14 is a diagram showing an example in which driving support information is presented by voice. [Figure 15] FIG. 15 is a diagram showing a case where there is no bias in the prediction errors occurring in the divided areas of the information acquisition screen. [Figure 16] FIG. 16 is a diagram showing an example in which the number of divided regions in which prediction errors exceeding the threshold value occur gradually increases. [Figure 17] FIG. 17 is a diagram showing an example in which no divided regions with prediction errors exceeding the threshold value occur overall. [Figure 18] FIG. 18 is a diagram for explaining the presentation period of the driving support information. [Figure 19] FIG. 19 is a diagram illustrating an example of changing whether or not driving support information is provided. [Figure 20] FIG. 20 is a diagram illustrating an example of changing the number of pieces of driving support information. [Figure 21]FIG. 21 is a diagram showing an example of changing the position of the driving support information. [Figure 22] FIG. 22 is a diagram showing an example of changing the type of driving support information. [Figure 23] FIG. 23 is a functional block diagram illustrating a functional configuration of a driving assistance device according to the second embodiment. [Figure 24] FIG. 24 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device according to the second embodiment. [Figure 25] FIG. 25 is a diagram showing an example of presentation of driving support information recommending "possible driving." [Figure 26] FIG. 26 is a diagram showing an example of the driving support information presented when the driver is performing "possible driving." [Figure 27] FIG. 27 is a functional block diagram illustrating a functional configuration of a driving assistance device according to the third embodiment. [Figure 28] FIG. 28 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device according to the third embodiment. [Figure 29] FIG. 29 is a diagram showing an example of presentation of driving support information indicating that deceleration control has been performed. [Figure 30] FIG. 30 is a functional block diagram showing the functional configuration of a driving assistance device according to the fourth embodiment. [Figure 31] FIG. 31 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device according to the fourth embodiment. [Figure 32] FIG. 32 is a schematic diagram showing a flow until the driving assistance device according to the fifth embodiment presents driving assistance information based on the saliency map. [Figure 33] FIG. 33 is a schematic diagram showing a flow until the driving assistance device according to the sixth embodiment presents driving assistance information based on the prediction error and the saliency map. [Figure 34] FIG. 34 is a flowchart showing the flow of the operation of presenting driving support information based on the prediction error and saliency map in the driving support device according to the sixth embodiment. [Figure 35] FIG. 35 is a schematic diagram showing a flow until the driving assistance device according to the seventh embodiment presents driving assistance information based on the prediction error and the gaze retention time. [Figure 36] FIG. 36 is a schematic diagram showing a flow until the driving assistance device according to the eighth embodiment presents driving assistance information based on saliency and gaze retention time. BEST MODE FOR CARRYING OUT THE INVENTION

[0010] Below, a description will be given of a driving assistance device according to an embodiment serving as an example of the present disclosure. The driving assistance device according to the embodiment is useful for driving assistance for vehicles such as passenger cars and freight trucks, as well as for flying objects, ships, and the like. It is also useful for driving assistance for remotely operated moving objects where no one is on board the moving object itself. Below, the configuration and effects of the driving assistance device according to the embodiment will be described using driving assistance for a vehicle driver as an example. Please refer to the following description when providing driving assistance for other moving objects such as flying objects and ships.

[0011] [overview] First, Figure 1 is a schematic diagram showing the process by which a driver senses the situation around the vehicle and operates the vehicle. As shown in Figure 1, a driver senses the situation around the vehicle based on senses such as sight and hearing. Of this, 90% of the information necessary for driving a vehicle is obtained from sight. Once the information necessary for driving is obtained through sight, etc., the driver unconsciously predicts the situation around the vehicle in the immediate future based on memory of the past, for example, a few seconds, and the information necessary for driving obtained from sight, etc. Then, "unconscious attention" is activated due to the difference between this unconscious prediction and reality. In addition to this unconscious attention, "conscious attention" is also activated based on the information necessary for driving obtained from sight, etc.

[0012] The driver makes an "unconscious driving decision" based on such "unconscious attention" and a "conscious driving decision" based on "conscious attention," and performs an "unconscious driving operation" and a "conscious driving operation" based on the results of these decisions. The driver drives the vehicle through such "unconscious driving operations" and "conscious driving operations."

[0013] The driving assistance device of the embodiment detects areas around the vehicle that the driver senses and where the above-mentioned "unconscious attention" is active. Then, based on the detection results, it identifies areas that the driver is likely to overlook and presents information (text, images, audio, etc.) to attract the driver's conscious attention. This enables appropriate driving assistance according to the situation around the vehicle.

[0014] Furthermore, "danger" while a vehicle is traveling is classified into two types: "danger that is manifested (at that time)" and "danger that is not manifested (latent danger)." Furthermore, according to cognitive psychology, such as predictive coding theory, "danger" is classified into two types based on human cognitive characteristics: "danger that humans are likely to overlook" and "danger that humans are difficult to overlook."

[0015] Here, consider a case where, when an "unmanifested danger (potential danger)" is detected, all of the detected "unmanifested dangers (potential dangers)" are presented to the driver. In this case, the detected "unmanifested dangers (potential dangers)" are presented even if they are "dangers that are difficult for humans to overlook," so there is a concern that the driver may feel annoyed. Furthermore, if the driver feels annoyed, even if the driving assistance information is presented, there is a risk that the driver's attention will not be fully drawn (the "power" of drawing the driver's attention will be weakened).

[0016] To prevent such inconveniences, the driving assistance device of the embodiment selectively presents "dangers that humans are likely to overlook" from among "unmanifested dangers (potential dangers)." This allows the driver's attention to be appropriately drawn without being bothersome to the driver, thereby contributing to safe driving.

[0017] [First embodiment] (Hardware configuration) 2 is a block diagram showing the hardware configuration of the driving assistance device 1 according to the first embodiment. As shown in FIG. 2, the driving assistance device 1 includes a detection device 2, a control unit 3, a storage unit 4, an information presentation unit 5, and a vehicle control unit 6.

[0018] The detection device 2 is a device that mainly detects the situation around the vehicle, and includes, for example, a vehicle speed sensor 21 (an example of a travel speed detection unit) that detects the traveling speed of the vehicle, and a GPS (Global Positioning System) sensor 21 that detects the current geographical location of the vehicle. The detection device 2 also includes an exterior camera unit 23 (an example of an external camera unit) that mainly captures images of the scenery ahead in the traveling direction of the vehicle, and a driver camera unit 24 that captures images of the driver driving the vehicle. The captured image of the driver's face captured by the driver camera unit 24 is mainly used to detect the driver's line of sight. The detection device 2 also includes a microphone unit 25 that collects sounds outside the vehicle.

[0019] The memory unit 4 stores a driving assistance program for presenting driving assistance information that assists the driver in driving. The control unit 3 presents the driving assistance information to the information presentation unit 5 based on this driving assistance program. Specifically, if the information presentation unit 5 is a display unit, the control unit 3 controls the display of the driving assistance information on the display unit. On the other hand, if the information presentation unit 5 is a speaker unit, the control unit 3 controls the audio output of the driving assistance information via the speaker unit. Note that such display control and audio output control of the driving assistance information may be used together.

[0020] The vehicle control unit 6 is an example of an operation control unit, and for example, when the vehicle is traveling at a speed above a predetermined level in a location where danger is predicted, it controls the vehicle's braking to suppress the traveling speed, thereby ensuring safety.

[0021] (Functional configuration) 3 is a functional block diagram showing the functions realized by the control unit 3 executing the driving assistance program stored in the storage unit 4. As shown in this Fig. 3, the control unit 3 functions as a potential danger determination unit 31 and an information presentation control unit 32 by executing the driving assistance program.

[0022] The potential danger determination unit 31 is an example of a detection unit, and determines (detects) a potential recognition area that humans unconsciously recognize from the entire image capture area of ​​the captured image based on an image (forward image) of the vehicle traveling direction captured by the exterior camera unit 23. The information presentation control unit 32 presents predetermined driving assistance information in an area on the information presentation unit estimated from the potential recognition area detected by the potential danger determination unit 31. When the driving assistance information is presented as an image or video, the information presentation unit 5 can be, for example, a vehicle meter display device, a HUD (Head-Up Display) device, a monitor device of a car navigation system, or the like. When the driving assistance information is presented as audio (including audio messages and electronic sounds, etc.), a speaker unit can be used as the information presentation control unit 32.

[0023] In this example, the potential danger determination unit 31 and the information presentation control unit 32 are realized by software using a driving assistance program. However, all or part of these may be realized by hardware such as an integrated circuit (IC).

[0024] The driving assistance program may be provided by being recorded in the form of file information in an installable or executable format on a computer-readable recording medium such as a CD-ROM or a flexible disk (FD). The driving assistance program may be provided by being recorded on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) disk, or a semiconductor memory. The driving assistance program may be provided by being installed via a network such as the Internet. The driving assistance program may be provided by being pre-installed in a ROM or the like within the device.

[0025] (Driving assistance information presentation operation) Fig. 4 is a flowchart for explaining the operation of presenting driving support information. First, in step S1, the potential danger determination unit 31 shown in Fig. 3 divides a moving image of the scenery ahead of the vehicle captured by the exterior camera unit 23 into a total of 25 divided regions, for example, 5 regions x 5 regions, and extracts a "feature amount" for each divided region.

[0026] As an example, the amount of prediction error detected for each divided region can be used as the "feature amount." In addition, saliency, which indicates the proportion of object edges included in each divided region, can be used as the "feature amount." It is also possible to use both prediction error and saliency. Below, a case where only the amount of prediction error is used as the "feature amount" will be described, and saliency and other aspects will be described later.

[0027] Fig. 5 is a schematic diagram for explaining the operation of generating a prediction error image by the potential danger determination unit 31. As shown in Fig. 5, the potential danger determination unit 31 generates a prediction image based on "PredNet," a deep learning prediction model constructed within the framework of deep learning, by imitating the predictive coding process in the cerebral cortex. Specifically, when 20 frames of past images captured by the outside camera unit 23 are supplied as shown in Fig. 5, the potential danger determination unit 31 generates a prediction image corresponding to a future frame based on the deep learning prediction model.

[0028] PredNet is described in detail in Lotter, W., Kreiman, G., and Cox, D., "Deep predictive coding networks for video prediction and unsupervised learning," https: / / arxiv.org / abs / 1605.08104.

[0029] That is, at time t -20 ~time t -1If past images up to time t are supplied, the potential danger determination unit 31 generates a predicted image of a future frame at time t based on the deep learning prediction model. -19 ~time t -0 Based on the deep learning prediction model, the potential danger determination unit 31 generates a predicted image of a future frame at time t1 from past images up to time t -18 ~time t -1 A predicted image of a future frame at time t2 is generated based on the deep learning prediction model from past images up to time t1.

[0030] The potential danger determination unit 31 generates predicted images for all future frames using past images that are shifted in time by one frame in this manner.

[0031] Furthermore, the potential danger determination unit 31 compares the generated predicted image with an actual image (correct image) actually captured by the outside camera unit 23 at the time the generated predicted image was generated, pixel by pixel, and generates a prediction error image based on the difference in each pixel value between the two. The example in Fig. 5 shows an example in which a prediction error image at time t0 is generated based on the difference in each pixel value between the predicted image at time t0 and the actual image (correct image). The value of each pixel in this prediction error image indicates the value of the respective prediction error.

[0032] Next, after generating the prediction error image in this manner, the potential danger determination unit 31 divides the entire image area of ​​the generated prediction image into a total of 25 divided areas, for example, 5 x 5 vertical x 5 horizontal.The potential danger determination unit 31 then calculates the sum of the prediction error values, which are the values ​​of each pixel, for each divided area.The number of frames of past images used to calculate the prediction error may be any number depending on the design, such as 30 frames.

[0033] Next, in step S2 of Fig. 4, the potential danger determination unit 31 compares the sum of prediction errors of each divided region with a first threshold (High) for the upper limit of the sum of prediction errors shown in Fig. 6. Furthermore, in step S5 of Fig. 4, the potential danger determination unit 31 compares the sum of prediction errors of each divided region with a second threshold (Low) for the lower limit of the sum of prediction errors shown in Fig. 6. Then, of the divided regions, a divided region having a sum of prediction errors that exceeds the first threshold (High) and a divided region having a sum of prediction errors that is below the second threshold (Low) are detected. The first and second thresholds shown in Fig. 6 are examples of prediction error thresholds.

[0034] If the sum of prediction errors exceeds the first threshold (High), it means that the divided area is an area that the driver is unconsciously aware of, and if the sum of prediction errors is below the second threshold (Low), it means that the divided area is an area that the driver is not even aware of, even if unconsciously.

[0035] The information presentation control unit 32 changes the driving assistance information (presentation information) to be presented based on the divided area having a sum of prediction errors that exceeds the first threshold (High) (Step S2: Yes → Step S3). The information presentation control unit 32 also changes the driving assistance information (presentation information) to be presented based on the divided area having a sum of prediction errors that is below the second threshold (Low) (Step S5: Yes → Step S6). The information presentation control unit 32 also changes the driving assistance information (presentation information) to be presented based on the divided area having a sum of prediction errors that is less than the first threshold (High) and less than the second threshold (Low) (Step S5: No → Step S7). Then, the information presentation control unit 32 presents the driving assistance information changed in Step S3, Step S6, or Step S7 to the information presentation unit 5.

[0036] FIG. 7 is a diagram schematically illustrating a flow up to displaying such driving assistance information on the information presentation unit 5. As shown in FIG. 7, a predicted image is generated based on a deep learning prediction model (=driver visual model) from multiple forward images captured by the outside camera unit 23, and a prediction error image is generated by taking the difference with a forward image captured at the same time. For simplicity of explanation, the following description will be given assuming that the driver visual model outputs the prediction error image. Furthermore, of the divided regions of the prediction error image, divided regions having a sum of prediction errors exceeding a first threshold (High) and divided regions having a sum of prediction errors below a second threshold (Low) are detected. Then, based on the detection results, driving assistance information is presented on the information presentation unit 5.

[0037] Deep learning prediction models are described in detail in, for example, "Kato, Emura, and Watanabe, 'Analysis of Factors behind Traffic Near-Miss Events Using Deep Learning to Simulate Human Vision,' Proceedings of the Academic Lectures of the 2021 Spring Meeting of the Society of Automotive Engineers of Japan."

[0038] FIG. 8 is a diagram illustrating the inconveniences that occur when driving support information is not presented. In FIG. 8(a), the dotted line graph shows the transition of the prediction error value of the first divided region, the dashed line graph shows the transition of the prediction error value of the second divided region, and the solid line graph shows the transition of the prediction error value of the third divided region. FIG. 8(b) shows an image captured at time t1 when the prediction error value of the first divided region exceeds the first threshold (High). FIG. 8(c) shows an image captured at time t2 when the prediction error value of the second divided region exceeds the first threshold (High). FIG. 8(d) shows an image captured at time t3 when the prediction error value of the third divided region exceeds the first threshold (High).

[0039] As shown in FIG. 8(b), at time t1, the first divided region with a prediction error value exceeding the first threshold (High) is located on the left side, so the driver unconsciously pays attention to the left side. Next, as shown in FIG. 8(c), at time t2, the second divided region with a prediction error value exceeding the first threshold (High) is located on the right side, so the driver unconsciously pays attention to the right side. Similarly, as shown in FIG. 8(d), at time t3, the third divided region with a prediction error value exceeding the first threshold (High) is located on the right side, so the driver unconsciously pays attention to the right side. In this state where the driver is unconsciously paying attention to the right side, it is difficult for the driver to notice another vehicle appearing from the left side, as shown in FIG. 8(d).

[0040] In contrast, Fig. 9 is a diagram showing an example of presentation of driving assistance information in the driving assistance device 1 of the embodiment. In Fig. 9(a), the dotted line graph shows the transition of the prediction error value of the first divided region, the dashed line graph shows the transition of the prediction error value of the second divided region, and the solid line graph shows the transition of the prediction error value of the third divided region. Fig. 9(b) shows an image captured at time t1 when the prediction error value of the first divided region exceeds the first threshold (High). Fig. 9(c) shows an image captured at time t2 when the prediction error value of the second divided region exceeds the first threshold (High). Fig. 9(d) shows an image captured at time t3 when the prediction error value of the third divided region exceeds the first threshold (High).

[0041] 9(b), at time t1, the first divided area having a prediction error value exceeding the first threshold (High) is on the left side, so the driver unconsciously pays attention to the left side. In this case, the information presentation control unit 32 presents driving assistance information such as "Pay attention to the right side as well" on the display screen of the HUD to encourage the driver to pay attention to the right side as well.

[0042] 9(c), at time t2, the second divided area having a prediction error value exceeding the first threshold (High) is present on the right side, so the driver unconsciously pays attention to the right side. In this case, the information presentation control unit 32 presents driving assistance information such as "Pay attention to the left side" on the display screen of the HUD to encourage the driver to also pay attention to the left side.

[0043] Also, as shown in FIG. 9(d), at time t3, the third divided region with a prediction error value exceeding the first threshold (High) is located on the right side, so the driver unconsciously pays attention to the right side. In this case, the information presentation control unit 32 presents driving assistance information such as "Pay attention to the left side" on the display screen of the HUD to encourage the driver to also pay attention to the left side. This allows the driver to notice another vehicle appearing from the left side as shown in FIG. 9(d), and to avoid inconveniences such as a collision by driving operations.

[0044] 10 is a flowchart showing a flow of determining the display position of the driving support information for the divided area with a large prediction error. In step S31, the potential danger determination unit 31 extracts a prediction error, which is an example of an extraction amount, for each divided area of ​​the captured image ahead of the vehicle.

[0045] Next, in step S32, the information presentation control unit 32 determines whether or not the first threshold (High) is exceeded for each divided area. For ease of understanding, an example of determining whether or not the first threshold (High) is exceeded for each divided area will be described, but the information presentation control unit 32 also determines whether or not the second threshold (Low) is exceeded for each divided area.

[0046] Specifically, the information presentation control unit 32 assigns a number "i" in order starting from "0" to each divided area for which it is determined whether the feature value exceeds the first threshold (High) (step S33 or step S36). The number assigned to each divided area is then incremented by one in step S35. For each divided area with each number, the information presentation control unit 32 determines whether the feature value (prediction error value) exceeds the first threshold (High) (step S32).

[0047] Next, the information presentation control unit 32 determines whether the numbers assigned to the divided areas are equal to or greater than the total number of divided areas (step S34). For example, in the above example, the entire area of ​​the captured image is divided into 25 divided areas. In this case, the information presentation control unit 32 determines whether the number "i" assigned to the divided areas while incrementing it by one in step S34 has reached "25". If the number "i" assigned to each divided area has not reached "25" (step S34: Yes), this means that the determination of whether all divided areas exceed the first threshold (High) has not been completed. Therefore, the information presentation control unit 32 repeatedly executes the processes of steps S32 to S36 described above.

[0048] On the other hand, when the number "i" assigned to each divided area reaches "25," this means that the determination of whether or not the first threshold (High) is exceeded for all divided areas has been completed. In this case, the information presentation control unit 32 counts the number of divided areas having prediction error values ​​exceeding the first threshold (High) based on the captured image, dividing the number into left and right areas of the information acquisition screen (step S37). Then, the information presentation control unit 32 determines whether the number of the divided areas located on the right side of the information acquisition screen is greater than the number of the divided areas located on the left side of the information acquisition screen (step S38).

[0049] The information presentation control unit 32 determines that the number of divided areas located on the right side of the information acquisition screen is greater than the number of divided areas located on the left side of the information acquisition screen, which means that the driver's unconscious attention is biased toward the right side of the screen and the driver is likely to overlook the left side of the screen. In this case (step S38: Yes), the information presentation control unit 32 presents driving assistance information such as "Pay attention to the left side as well" via the information presentation unit 5 (step S39). This can draw the driver's conscious attention to the left side, which the driver is likely to overlook.

[0050] On the other hand, if the number of divided areas located on the right side of the information acquisition screen is smaller than the number of divided areas located on the left side of the information acquisition screen, it means that the driver's unconscious attention is biased toward the left side of the screen and the driver is likely to overlook the right side of the screen. In this case (step S38: No), the information presentation control unit 32 presents driving assistance information such as "Pay attention to the right side as well" via the information presentation unit 5 (step S40). This can attract the driver's conscious attention to the right side, which the driver is likely to overlook.

[0051] (Information presentation according to how prediction errors occur) Next, an example of information presentation according to the manner in which prediction errors occur will be described. Fig. 11 shows a case in which prediction errors occur disproportionately on either the left or right side of the information acquisition screen. Figs. 11(a) to 11(c) show a case in which prediction errors occur disproportionately on the left side of the information acquisition screen. In this case, the information presentation control unit 32 presents driving assistance information to attract the driver's attention on the right side of the information acquisition screen.

[0052] Furthermore, when many prediction errors occur biased toward the upper side of the information acquisition screen, the information presentation control unit 32 presents driving assistance information to attract the driver's attention on the lower side of the information acquisition screen. When many prediction errors occur biased toward the right side of the information acquisition screen, the information presentation control unit 32 presents driving assistance information to attract the driver's attention on the left side of the information acquisition screen. In other words, the information presentation control unit 32 presents driving assistance information on the side of the divided area that is positioned relative to the divided area on the information acquisition screen where many prediction errors occur.

[0053] 12A and 12B are diagrams showing examples of display of driving assistance information. Fig. 12A shows an example in which many prediction errors occur on the left side of the information acquisition screen. In this case, as shown in Fig. 12B, the information presentation control unit 32 displays, for example, a yellow triangle icon or an arrow on the right side of the information acquisition screen, which is likely to be overlooked by the driver, to draw the driver's attention to the right side of the information acquisition screen.

[0054] FIG. 13 is a diagram showing another example of the display of driving assistance information. FIG. 13(a) shows an example in which many prediction errors occur on the left side of the information acquisition screen. In the example of FIG. 12, in such a case, driving assistance information to attract the driver's attention is displayed on the right side of the information acquisition screen. However, as shown in FIG. 13(b), driving assistance information to attract the driver's attention may also be displayed in the center of the information acquisition screen. In the example of FIG. 13(b), the information presentation control unit 32 displays straight lines extending from the front side to the depth direction on the left and right sides of the information acquisition screen. By displaying such lines, the driver's attention can be attracted to the center of the information acquisition screen.

[0055] Note that a segmented area in which the prediction error value is below the second threshold (Low) shown in FIG. 6 is a segmented area that the driver is likely to overlook (pay less attention to). Such a segmented area that the driver is likely to overlook occurs, for example, when the driver is driving the vehicle following a vehicle in front during a traffic jam. This means that the driver is paying less attention to the vehicle in front and is driving carelessly. If this continues, there is a risk of an inconvenience such as a rear-end collision with the vehicle in front.

[0056] For this reason, the information presentation control unit 32 displays the line shown in Fig. 13(b) to guide the driver's line of sight to the center of the information acquisition screen and call the driver's attention. Alternatively, the information presentation control unit 32 displays driving assistance information (message or voice) such as "Pay attention to what's ahead" in the center of the information acquisition screen to guide the driver's line of sight to the center of the screen and call the driver's attention. This makes it possible to prompt the driver to pay attention to what's ahead, preventing inconveniences such as rear-end collisions and ensuring safety.

[0057] Next, Fig. 14 shows an example of presenting driving assistance information by voice. Fig. 14(a) shows an example in which many prediction errors occur on the left side of the information acquisition screen. In this case, the information presentation control unit 32 presents a voice message such as "Pay attention to the right side as well," via the speaker unit 5, as shown in Fig. 14(b). This makes it possible to draw the driver's attention to the right side of the information acquisition screen.

[0058] Next, Figures 15(a) to 15(d) show examples in which there is no bias in the prediction errors that occur in each divided area of ​​the information acquisition screen. This means that the driver is unconsciously paying attention to the entire area. Therefore, the information presentation control unit 32 does not present the driving assistance information.

[0059] 16(a) to 16(d) show an example in which the number of divided areas in which prediction errors exceeding the threshold value occur gradually increases. This means that the cognitive load on the driver is increasing. Therefore, the information presentation control unit 32 presents driving assistance information indicating that the cognitive load is increasing.

[0060] The opposite is also true: if the number of divided areas where prediction errors exceed the threshold value are gradually decreasing, this indicates that the prediction error value of each divided area falls between the first threshold value (High) and the second threshold value (Low) shown in Fig. 6, and the cognitive load is decreasing. In this case, the information presentation control unit 32 presents driving assistance information indicating that the cognitive load is decreasing.

[0061] Next, Figures 17(a) to 17(d) show an example in which no segmented regions with prediction errors exceeding the threshold value occur overall. This occurs when there is little change in the scenery ahead, such as when the driver is driving their own vehicle following a vehicle in front during a traffic jam. In such a case, the driver's attention may become distracted, leading to careless driving and causing inconvenience such as a rear-end collision with the vehicle in front.

[0062] For this reason, the information presentation control unit 32 displays a line, for example, as shown in Fig. 13(b), at the center of the information acquisition screen to guide the driver's line of sight to the center of the screen and call the driver's attention. Alternatively, the information presentation control unit 32 displays driving assistance information (message or voice), such as "Pay attention to what's ahead," at the center of the information acquisition screen to guide the driver's line of sight to the center of the screen and call the driver's attention. This makes it possible to prompt the driver to pay attention to what's ahead, preventing inconveniences such as rear-end collisions and ensuring safety.

[0063] (Driving assistance information presentation period) Next, Fig. 18 is a diagram for explaining the presentation period of driving assistance information. As shown in Fig. 6, when the sum of prediction errors of each divided area exceeds a first threshold (High) or a second threshold (Low), the information presentation control unit 32 presents the driving assistance information for a certain period (or while it exceeds either threshold). Fig. 18 shows that the sum of prediction errors exceeds either threshold at time t1, and in this case, the information presentation control unit 32 presents the driving assistance information for a certain period from time t1.

[0064] 18 also shows that the sum of prediction errors exceeds one of the thresholds at time t2 and time t3, in which case the information presentation control unit 32 presents the first driving support information for a certain period from time t2 and presents the second driving support information for a certain period from time t3. When presenting the second driving support information, the presentation of the first driving support information may be terminated.

[0065] In this way, by presenting driving assistance information for a predetermined period of time (or while any of the thresholds is exceeded), it is possible to prevent the presentation of driving assistance information from becoming too long and causing inconvenience to the driver.

[0066] (Change control of presented driving assistance information) Next, presenting a large amount of driving support information to the information presentation unit 5 is cumbersome for the driver and is not desirable from the perspective of safe driving. For this reason, the information presentation control unit 32 changes and controls the presence, number, position, type, etc. of the driving support information to be presented depending on the situation.

[0067] Fig. 19 shows an example of changing the presence or absence of driving support information. For example, as shown in Fig. 19(a), an arrow indicating the direction of travel, a circle indicating a pedestrian, and a view behind the vehicle are usually displayed on the image presentation screen of the information presentation unit 5 in a rectangular area at the top right. If the amount of driving support information presented in this way increases, the driver's attention may be distracted.

[0068] For this reason, the information presentation control unit 32 hides the driving assistance information when the vehicle is stopped or while the vehicle is traveling at a low speed, such as less than 15 km / h, as shown in Fig. 19(b). This prevents the inconvenience of displaying unnecessary driving assistance information and causing annoyance to the driver.

[0069] Next, Fig. 20 shows an example of changing the number of pieces of driving support information (number of pieces of presentation). For example, as shown in Fig. 20(a), an arrow indicating the direction of travel, a circle indicating a pedestrian, and a scene behind the vehicle are usually displayed on the image presentation screen of the information presentation unit 5 in a rectangular area at the top right. If the number of pieces of driving support information presented in this way increases, the driver's attention will be distracted.

[0070] For this reason, the information presentation control unit 32 preferentially presents only driving support information for targets with a high degree of risk, such as an area where a prediction error occurs as shown in Fig. 20(b). This makes it possible to present only truly necessary driving support information and draw the driver's attention.

[0071] Next, Fig. 21 shows an example of changing the position of driving support information. For example, as shown in Fig. 20(a), an arrow indicating the direction of travel, a circle indicating a pedestrian, and a view behind the vehicle are usually displayed on the image presentation screen of the information presentation unit 5 in a rectangular area at the top right. If the amount of driving support information presented in this way increases, the driver's attention will be distracted.

[0072] Therefore, the information presentation control unit 32 changes the position of the presented driving support information as shown in FIG. 21(b) or FIG. 21(c) based on the divided area having a prediction error exceeding the threshold.

[0073] This allows driving assistance information to be presented at a necessary location to attract the driver's attention, depending on the known situation of the vehicle.

[0074] Next, Fig. 22 shows an example of changing the type of driving assistance information. As shown in Fig. 22(a), when there are many pedestrians and stopped vehicles ahead of the vehicle, many divided areas with prediction errors exceeding the threshold value will occur, as shown in Fig. 22(b). In this case, the information presentation control unit 32 changes the driving assistance information in the form of an arrow to a message calling attention, such as "Slow down," and presents it. This allows the driving assistance information to be presented to the driver more accurately.

[0075] (Effects of the first embodiment) As is clear from the above description, in the driving assistance device 1 of the first embodiment, the potential danger determination unit 31 detects a potential recognition area that a person unconsciously recognizes from the entire captured area of ​​the captured image, based on an image of the vehicle traveling direction captured by the vehicle's exterior camera unit 23. Then, the information presentation control unit 32 presents predetermined driving assistance information in an area on the information presentation unit that is estimated according to the detected potential recognition area.

[0076] This makes it possible to draw the driver's attention to areas that the driver is likely to overlook, other than areas that the driver is unconsciously paying attention to, and to present appropriate driving assistance information according to the situation around the vehicle.

[0077] Furthermore, the potential danger determination unit 31 detects, as potential recognition areas, areas that have a feature amount (prediction error, etc.) exceeding a predetermined threshold value among the divided areas obtained by dividing the entire imaging area into a plurality of areas. This makes it possible to accurately detect areas that the driver is unconsciously paying attention to, and to present more appropriate driving support information.

[0078] Furthermore, the information presentation control unit 32 presents driving assistance information separately while the feature amount (prediction error, etc.) is below a predetermined threshold and while the feature amount exceeds the threshold. Specifically, the information presentation control unit 32 does not present driving assistance information while the feature amount (prediction error, etc.) is below a predetermined threshold, but presents driving assistance information when the feature amount exceeds the threshold. This makes it possible to clearly identify the divided area that the driver is unconsciously paying attention to and present driving assistance information that calls attention to the relative area, thereby ensuring greater safety.

[0079] Furthermore, the information presentation control unit 32 presents the driving assistance information for a predetermined period of time (or while the feature amount exceeds any of the thresholds), thereby enabling the driver to properly recognize the driving assistance information (the driver will not miss the driving assistance information due to the short presentation time).

[0080] Furthermore, the information presentation control unit 32 changes one or more of the following in accordance with the detected latent recognition area: whether or not to present driving assistance information, the presentation position, the number of pieces of information to be presented, or the type of information to be presented. This allows the driver's attention to be drawn by selectively presenting necessary driving assistance information in accordance with the known situation of the vehicle. This prevents the inconvenience of displaying too much driving assistance information and causing annoyance to the driver.

[0081] [Second embodiment] Next, a driving assistance device according to a second embodiment will be described. The driving assistance device according to the second embodiment is an example in which the driving assistance information to be presented is changed based on the vehicle's traveling speed. Note that the first embodiment described above and the second embodiment described below differ only in this respect. Therefore, only the differences between the two will be described below, and redundant explanations will be omitted.

[0082] Fig. 23 is a functional block diagram showing the functional configuration of a driving assistance device according to the second embodiment. As shown in Fig. 23, in the driving assistance device according to the second embodiment, the potential danger determination unit 31 acquires a forward image captured by the outside-vehicle camera unit 23, as well as the vehicle traveling speed (host vehicle speed) detected by the vehicle speed sensor 21 shown in Fig. 2.

[0083] 24 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device of the second embodiment. In this flowchart, the potential danger determination unit 31 extracts feature quantities such as prediction errors for each divided area based on the captured image of the area ahead of the vehicle, as described above (step S11). The information presentation control unit 32 compares the extracted feature quantities of each divided area with the first threshold (High) and the second threshold (Low) shown in FIG. 6, and determines whether the feature quantities of each extracted divided area exceed either of the thresholds (step S12).

[0084] If it is determined that the feature amount of each extracted divided region exceeds any of the thresholds (step S12: Yes), the potential danger determination unit 31 acquires the vehicle traveling speed (host vehicle speed) detected by the vehicle speed sensor 21 (step S13). In step S14, the potential danger determination unit 31 determines whether the acquired vehicle traveling speed exceeds a traveling speed threshold, such as 30 km / h.

[0085] If the feature amount of each divided area exceeds one of the thresholds (step S12: Yes) and the vehicle's traveling speed exceeds the traveling speed threshold (step S14: Yes), it is necessary to be careful of pedestrians or other vehicles suddenly appearing. Therefore, in step S15, the information presentation control unit 32 presents driving assistance information recommending "driving in a possible situation."

[0086] FIG. 25 is a diagram showing an example of the presentation of driving assistance information recommending "driving in a possible danger." As shown in FIG. 25(a), when the prediction error in the divided area on the left side of the information acquisition screen exceeds a threshold, the area on the right side of the information acquisition screen indicates that the driver is likely to overlook an obstacle, leading to a tendency to delay in responding to a pedestrian or other vehicle suddenly appearing. For this reason, the information presentation control unit 32 presents driving assistance information recommending "driving in a possible danger," such as "drive slowly," on the image presentation screen of the information presentation unit 5, as shown in FIG. 25(b). This allows the driver to reduce the vehicle's speed and continue driving while paying attention to pedestrians or other vehicles suddenly appearing.

[0087] In contrast to this, in the flowchart of FIG. 24, if the feature value of each divided area exceeds one of the thresholds (step S12: Yes), but the vehicle's traveling speed does not exceed the threshold for traveling speed (step S14: No), this means that the vehicle is traveling at a low speed and is being operated while paying attention to pedestrians or other vehicles suddenly appearing.

[0088] In this case, the information presentation control unit 32 presents driving support information praising the "potential driving" in step S16. Fig. 26 is a diagram showing an example of the presentation of driving support information praising the "potential driving." As shown in Fig. 26(a), when the prediction error in the divided area on the left side of the information acquisition screen indicates a value exceeding the threshold, the area on the right side of the information acquisition screen indicates that the driver is likely to overlook something, and the driver is likely to be slow to respond to a pedestrian or another vehicle suddenly appearing.

[0089] However, even in such cases, if the vehicle is driven slowly, an accident can be prevented. For this reason, the information presentation control unit 32 presents driving support information praising the "possible driving," such as "You are driving at an appropriate slow speed," on the image presentation screen of the information presentation unit 5, as shown in FIG. 26(b). This allows the driver to recognize that the current vehicle driving operation is correct, and to continue driving the vehicle while paying attention to pedestrians or other vehicles suddenly appearing. This ensures the safety of the driver and surrounding pedestrians.

[0090] (Effects of the second embodiment) As is clear from the above description, the driving assistance device of the second embodiment detects the vehicle's traveling speed together with the feature amount and presents driving assistance information. This makes it possible to change the driving assistance information to be presented according to the vehicle speed, and to present appropriate driving assistance information according to the vehicle speed, while also achieving the same effects as those of the first embodiment described above.

[0091] [Third embodiment] Next, a driving assistance device according to a third embodiment will be described. The driving assistance device according to the third embodiment performs, for example, braking control in accordance with the vehicle's traveling speed, and presents the control content as driving assistance information. Note that this is the only point that differs between the second embodiment described above and the third embodiment described below. Therefore, only the differences between the two will be described below, and redundant explanations will be omitted.

[0092] Fig. 27 is a functional block diagram showing the functional configuration of a driving assistance device according to the third embodiment. As shown in Fig. 27, in the driving assistance device according to the third embodiment, the potential danger determination unit 31 acquires the vehicle's traveling speed (host vehicle speed) detected by the vehicle speed sensor 21 shown in Fig. 2 together with a forward image captured by the exterior camera unit 23. If the vehicle's traveling speed (host vehicle speed) acquired from the vehicle speed sensor 21 exceeds a predetermined threshold (an example of a travel speed threshold), the potential danger determination unit 31 notifies the vehicle control unit 6 of this detection result, causing the vehicle control unit 6 to perform braking control of the vehicle, and the information presentation control unit 32 to present driving assistance information indicating that braking control has been performed.

[0093] 28 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device of the third embodiment. In this flowchart, the potential danger determination unit 31 extracts feature quantities such as prediction errors for each divided area based on the captured image of the area ahead of the vehicle, as described above (step S21). The information presentation control unit 32 compares the extracted feature quantities of each divided area with the first threshold (High) and the second threshold (Low) shown in FIG. 6, and determines whether the feature quantities of each extracted divided area exceed either of the thresholds (step S22).

[0094] If it is determined that the feature amount of each extracted divided region exceeds any of the thresholds (step S22: Yes), the potential danger determination unit 31 acquires the vehicle traveling speed (host vehicle speed) detected by the vehicle speed sensor 21 (step S23). In step S24, the potential danger determination unit 31 determines whether the acquired vehicle traveling speed exceeds a traveling speed threshold, such as 30 km / h, and notifies the vehicle control unit 6 of this determination result.

[0095] When the vehicle control unit 6 obtains a judgment result from the potential danger judgment unit 31 indicating that the vehicle's traveling speed exceeds the traveling speed threshold, in step S25, the vehicle control unit 6 automatically operates the vehicle's brakes or automatically releases the accelerator pedal, thereby performing deceleration control of the vehicle.

[0096] When such deceleration control is performed, the information presentation control unit 32 presents driving support information indicating that deceleration control has been performed on the image presentation screen of the information presentation unit 5 in step S26. Fig. 29 is a diagram showing an example of presentation of driving support information indicating that such deceleration control has been performed. Fig. 29(a) shows that the area on the right side of the information acquisition screen has a large prediction error value, indicating that the driver is unconsciously paying attention, while the opposite area on the left side indicates a situation in which the driver is not paying attention.

[0097] In such a situation, if the vehicle's traveling speed exceeds the threshold for traveling speed, it becomes difficult to deal with a pedestrian or another vehicle suddenly appearing from the left area, so the vehicle control unit 6 automatically operates the vehicle's brakes or automatically releases the accelerator pedal to control the deceleration of the vehicle.

[0098] Furthermore, when such vehicle deceleration control is performed, the driver may mistake the unintended vehicle deceleration control for a vehicle malfunction, etc. For this reason, the information presentation control unit 32 presents driving assistance information indicating that braking control has been performed, such as "Slow down," on the image presentation screen of the information presentation unit 5, as shown in Fig. 29(b). This allows the driver to recognize that automatic deceleration control is being performed because the driving speed is fast, and prevents the driver from mistaking this for a vehicle malfunction, etc.

[0099] Although the example described above concerns the case where braking control of a vehicle is performed, other vehicle operation controls such as lane adjustment control by automatic steering operation and turn signal operation may also be performed.

[0100] (Effects of the third embodiment) As is clear from the above description, the driving assistance device of the third embodiment detects vehicle information (such as traveling speed) along with feature quantities, and automatically brakes the vehicle when the traveling speed exceeds a threshold, for example. Furthermore, the information presentation control unit 32 presents driving assistance information indicating that braking control has been performed on the image presentation screen of the information presentation unit 5. This forcibly brakes the vehicle, making it possible to prepare for a pedestrian or another vehicle suddenly appearing from an area that the driver is likely to overlook, ensuring safety and providing the same effects as those of the first and second embodiments described above.

[0101] [Fourth embodiment] Next, a driving assistance device according to a fourth embodiment will be described. The driving assistance device according to the fourth embodiment is an example in which the driving assistance information to be presented is changed taking into consideration the time the driver's line of sight is maintained. Note that this is the only point that differs between the first embodiment described above and the fourth embodiment described below. Therefore, only the differences between the two will be described below, and redundant explanations will be omitted.

[0102] Fig. 30 is a functional block diagram showing the functional configuration of a driving assistance device according to the fourth embodiment. As shown in Fig. 30, in the driving assistance device according to the fourth embodiment, potential danger determination unit 31 acquires a forward image captured by exterior camera unit 23 and an image of the driver's face captured by driver camera unit 24 shown in Fig. 2.

[0103] 31 is a flowchart showing the flow of the operation of presenting driving support information in the driving support device of the fourth embodiment. In this flowchart, the potential danger determination unit 31 extracts feature quantities such as prediction errors for each divided area based on a captured image of the area ahead of the vehicle, as described above (step S61). The information presentation control unit 32 compares the feature quantities of each extracted divided area with the first threshold (High) and the second threshold (Low) shown in FIG. 6, and determines whether the feature quantities of each extracted divided area exceed either of the thresholds (step S62).

[0104] If it is determined that the feature amount of each extracted divided area exceeds one of the thresholds (step S62: Yes), the potential danger determination unit 31 functions as a gaze detection unit and acquires a facial image of the driver captured by the driver camera unit 24 (step S63). Based on the acquired facial image of the driver, the potential danger determination unit 31 detects the position and duration on the information acquisition screen where the driver's gaze is fixed. Then, the potential danger determination unit 31 determines whether the duration of time the driver's gaze is fixed on a divided area having a prediction error that exceeds the threshold exceeds a threshold for duration (step S64).

[0105] If a segmented area has a large feature value such as a prediction error (step S62: Yes) and the gaze dwell time on that segmented area exceeds the dwell time threshold (step S64: Yes), this means that the driver's gaze is fixed on that segmented area and the driver's attention is biased. Therefore, the information presentation control unit 32 presents a message recommending slowing down, such as "Drive slowly" as shown in FIG. 25(b), as driving assistance information on the image presentation screen of the information presentation unit 5, to enable the driver to deal with pedestrians or other vehicles suddenly appearing, and encourages the driver to "drive in a possible accident" (step S65). This allows the driver's biased attention to be redirected back to a state of paying attention to the entire area of ​​the information acquisition screen, enabling the driver to deal with pedestrians or other vehicles suddenly appearing, ensuring safety.

[0106] On the other hand, even if a segmented area has a large feature amount such as a prediction error (step S62: Yes), the fact that the gaze dwell time on that segmented area does not exceed the dwell time threshold (step S64: No) means that the driver is also paying attention to other areas. Therefore, the information presentation control unit 32 presents a message praising the driver for "possible driving," such as "You are driving at an appropriate slow speed," as illustrated in FIG. 26(b) on the image presentation screen of the information presentation unit 5 as driving assistance information (step S66). This can improve the driver's motivation for safe driving and ensure safety.

[0107] The first threshold (High) and the second threshold (Low) used in this embodiment may be either one of them, or three or more thresholds may be provided.

[0108] (Effects of the Fourth Embodiment) As is clear from the above description, the driving assistance device of the fourth embodiment detects the gaze dwell time of the driver as well as the feature amount, and presents driving assistance information. This makes it possible to change the driving assistance information to be presented depending on the gaze dwell time of the driver, and to present appropriate driving assistance information, as well as to obtain the same effects as the above-mentioned embodiments.

[0109] [Fifth embodiment] Next, a driving assistance device according to a fifth embodiment will be described. In the driving assistance devices according to the above-described embodiments, a prediction error is used as a feature of each divided region. In contrast, the driving assistance device according to the fifth embodiment is an example in which a saliency map is used as a feature of each divided region. Note that this is the only point that differs between the above-described embodiments and the fifth embodiment described below. Therefore, only the different parts will be described below, and redundant explanations will be omitted.

[0110] 32 is a schematic diagram showing a flow until the driving assistance device of the fifth embodiment presents driving assistance information based on a saliency map. In this case, the potential danger determination unit 31 extracts edge portions (contour portions) of people, objects, etc. captured in the captured image taken by the outside camera unit 23, for example, based on brightness data. The potential danger determination unit 31 then generates a saliency map indicating the content of edge portions in each region of the captured image. The potential danger determination unit 31 also divides the entire region of this saliency map into, for example, 25 divided regions as described above, and generates an attention prediction map in which divided regions in which the proportion of edge portions contained exceeds a predetermined saliency threshold are detected as potential recognition regions.

[0111] The divided regions in which the content of edge portions exceeds the saliency threshold are regions to which the driver unconsciously pays attention. Therefore, as described above, the information presentation control unit 32 presents, on the image presentation screen of the information presentation unit 5, driving assistance information that calls the driver's attention to regions relative to the divided regions in which the content of edge portions exceeds the saliency threshold.

[0112] This allows the driver to pay attention to areas that the driver is likely to overlook, other than the areas that the driver is unconsciously paying attention to, and it is possible to provide appropriate driving assistance information according to the situation around the vehicle, thereby achieving the same effects as the above-mentioned embodiments.

[0113] [Sixth embodiment] Next, a driving assistance device according to a sixth embodiment will be described. In the driving assistance devices according to the above-described embodiments, a prediction error or a saliency map is used as the feature of each divided region. In contrast, the driving assistance device according to the sixth embodiment is an example in which both a prediction error and an edge content (saliency) are used as the feature of each divided region. Note that this is the only point that differs between the above-described embodiments and the sixth embodiment described below. Therefore, only the different parts will be described below, and redundant explanations will be omitted.

[0114] Fig. 33 is a schematic diagram showing a flow until the driving support device of the sixth embodiment presents driving support information based on the prediction error and saliency map, and Fig. 34 is a flowchart showing a flow of the operation of presenting driving support information based on the prediction error and saliency map in the driving support device of the sixth embodiment.

[0115] First, in steps S41 to S44 of the flowchart in Fig. 34, as explained using the flowchart in Fig. 10, the potential danger determination unit 31 detects a prediction error (first feature amount) for each divided area based on the image captured by the vehicle exterior camera unit 23. Similarly, in steps S45 to S48, the potential danger determination unit 31 detects the edge content (saliency: second feature amount) for each divided area based on the image captured by the vehicle exterior camera unit 23.

[0116] In steps S49 to S51, the potential danger determination unit 31 detects, as a potential recognition area, a divided area in which the prediction error is larger than a predetermined threshold and the content (saliency) of edge parts is larger than a predetermined threshold, as shown in Fig. 33, and generates an attention prediction map. By detecting a potential recognition area using both the prediction error and the saliency, it is possible to detect a potential recognition area more accurately.

[0117] In step S52, the potential danger determination unit 31 determines whether or not the detection process for such a latent recognition area has been completed for all 25 divided areas. If the detection process for the latent recognition area for all 25 divided areas has not been completed, in step S53, the number of the divided area for which the detection process for the latent recognition area is to be performed is incremented by one, and the process returns to step S42.

[0118] On the other hand, if the latent recognition area detection process for all 25 divided areas is completed, the process proceeds to step S54. In step S54, the information presentation control unit 32 counts the number of latent recognition areas located in the right area of ​​the information acquisition screen and the number of latent recognition areas located in the left area of ​​the information acquisition screen. In step S55, the information presentation control unit 32 determines whether the number of latent recognition areas located in the right area of ​​the information acquisition screen is greater than the number of latent recognition areas located in the left area of ​​the information acquisition screen.

[0119] Then, if the information presentation control unit 32 determines that the number of latent recognition areas located in the right area of ​​the information acquisition screen is greater than the number of latent recognition areas located in the left area of ​​the information acquisition screen (step S55: Yes), in step S56, it presents driving assistance information that draws attention to the left area of ​​the information acquisition screen.

[0120] In contrast, if the information presentation control unit 32 determines that the number of latent recognition areas located in the right area of ​​the information acquisition screen is less than the number of latent recognition areas located in the left area of ​​the information acquisition screen (step S55: No), in step S57, it presents driving assistance information that draws attention to the right area of ​​the information acquisition screen.

[0121] This allows the driver's attention to be drawn to an area relative to the area to which the driver is unconsciously paying attention, thereby ensuring safety, and other effects similar to those of the above-mentioned embodiments can be obtained. Furthermore, in the case of the sixth embodiment, the latent recognition area is detected using both the prediction error and the content (saliency) of the edge portion, so that the latent recognition area can be detected more accurately.

[0122] [Seventh embodiment] Next, a driving assistance device according to a seventh embodiment will be described. The driving assistance device according to the sixth embodiment described above is an example in which a latent recognition area is detected using both prediction error and saliency as feature quantities of each divided area. In contrast, the driving assistance device according to the seventh embodiment is an example in which driving assistance information is presented based on prediction error and the driver's gaze retention time. Note that this is the only point that differs between the above-described embodiments and the seventh embodiment described below. Therefore, only the different parts will be described below, and duplicate explanations will be omitted.

[0123] 35 is a schematic diagram showing the flow until the driving assistance device of the seventh embodiment presents driving assistance information based on the prediction error and the gaze dwell time. In this case, the potential danger determination unit 31 detects the prediction error for each divided area based on the image captured by the outside-vehicle camera unit 23. In addition, the potential danger determination unit 31 detects the driver's gaze dwell time for each divided area based on the image of the driver's face captured by the driver camera unit 24.

[0124] The potential danger determination unit 31 then generates an attention prediction map that indicates, among the divided areas, those divided areas where the prediction error is less than a predetermined threshold and the gaze retention time exceeds the predetermined threshold. A divided area where the prediction error is less than the predetermined threshold and the driver's gaze retention time on that divided area exceeds the predetermined threshold means that the driver is looking at an area that is easy to overlook without paying attention to it.

[0125] For this reason, the information presentation control unit 32 presents driving assistance information that calls the driver's attention to the divided area where the prediction error is less than the predetermined threshold and the gaze dwell time exceeds the predetermined threshold on the image presentation screen of the information presentation unit 5. This makes it possible to obtain the same effects as the above-mentioned embodiments, such as calling the driver's attention to the divided area where the prediction error is less than the predetermined threshold and the gaze dwell time exceeds the predetermined threshold, and ensuring safety.

[0126] [Eighth embodiment] Next, a driving assistance device according to an eighth embodiment will be described. The driving assistance device according to the seventh embodiment described above is an example in which driving assistance information is presented using a prediction error and a dwell time of the driver's gaze. In contrast, the driving assistance device according to the eighth embodiment described below is an example in which driving assistance information is presented using saliency and a dwell time of the driver's gaze. Note that this is the only point that differs between the above-described embodiments and the eighth embodiment described below. Therefore, only the different parts will be described below, and redundant explanations will be omitted.

[0127] 36 is a schematic diagram showing the flow until the driving assistance device of the eighth embodiment presents driving assistance information based on saliency and gaze dwell time. In this case, the potential danger determination unit 31 detects the saliency, which is the content of the above-mentioned edge portion, for each divided area based on the image captured by the outside-vehicle camera unit 23. In addition, the potential danger determination unit 31 detects the driver's gaze dwell time for each divided area based on the image of the driver's face captured by the driver camera unit 24.

[0128] The potential danger determination unit 31 then generates an attention prediction map that indicates divided regions where the content of edge portions is less than a predetermined threshold (low salience) and the gaze retention time exceeds a predetermined threshold. A divided region with "low" salience and where the driver's gaze retention time on that divided region exceeds a predetermined threshold means that the driver is looking at an easily overlooked region without paying attention to it.

[0129] For this reason, the information presentation control unit 32 presents driving assistance information that calls the driver's attention to divided areas where the saliency is "low" and the gaze dwell time exceeds a predetermined threshold on the image presentation screen of the information presentation unit 5. This makes it possible to obtain the same effects as the above-mentioned embodiments, such as calling the driver's attention to divided areas where the prediction error is less than the predetermined threshold and the gaze dwell time exceeds the predetermined threshold, thereby ensuring safety.

[0130] Although the embodiments of the present disclosure have been described above, these embodiments are merely examples and are not intended to limit the scope of the invention. These embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0131] 1 Driving assistance devices 2. Detection Device 3. Control Unit 4 Storage section 5 Information presentation section 6 Vehicle control unit 21 Vehicle speed sensor 22 GPS sensor 23 Exterior camera unit 24 Driver Camera Unit 25 Microphone section 31 Potential Hazard Judgment Section 32 Information presentation control unit

Claims

1. a detection unit that detects a latent recognition area that a human being unconsciously recognizes from an entire image capture area of ​​a captured image, based on a predetermined feature amount included in a captured image of at least a moving direction of the mobile object, captured by an external camera unit provided on the mobile object; an information presentation control unit that presents predetermined driving assistance information in an area on the information presentation unit that is estimated according to the detected latent recognition area; a gaze detection unit that detects the driver's gaze, the detection unit detects, as the latent recognition area, a divided area having a predetermined feature amount exceeding a predetermined threshold, from among each of a plurality of divided areas obtained by dividing the entire imaging area; the detection unit detects, for each divided region, a prediction error that is a difference between each pixel of a predicted image predicted from a past image captured in the past by the external camera unit and each pixel of an actually captured image actually captured by the external camera unit at the time of the predicted image, and detects the divided region having the prediction error that exceeds a predetermined threshold as the latent recognition region; The information presentation control unit presents the driving assistance information to the divided area where the gaze detection unit detects a gaze retention length that is equal to or longer than a predetermined retention length and where the prediction error is less than the threshold. A driving assistance device characterized by the above.

2. A detection unit that detects a potential recognition area that a human being unconsciously recognizes from the entire image capture area of ​​a captured image based on a predetermined feature contained in an image of at least the moving direction of the moving body, the image being captured by an external camera unit provided on the moving body; an information presentation control unit that presents predetermined driving assistance information in an area on the information presentation unit that is estimated according to the detected latent recognition area; a gaze detection unit that detects the driver's gaze, the detection unit detects, as the latent recognition area, a divided area having a predetermined feature amount exceeding a predetermined threshold, from among each of a plurality of divided areas obtained by dividing the entire imaging area; the detection unit detects an edge portion of an object from the captured image captured by the external camera unit, and detects, among the divided areas, a divided area in which a ratio of the edge portion to the captured image exceeds a predetermined threshold as the latent recognition area; The information presentation control unit presents the driving assistance information to the divided area where a gaze retention length detected by the gaze detection unit is equal to or longer than a predetermined retention length and a ratio of the edge portion included in the divided area is less than the predetermined threshold. A driving assistance device characterized by the above.

3. The information presentation control unit presents the driving assistance information while the feature amount is less than the threshold value and while the feature amount exceeds the threshold value.

3. The driving assistance device according to claim 1 or 2,

4. The information presentation control unit presents the driving assistance information while the feature amount exceeds the threshold, or presents the driving assistance information for a certain period after the feature amount exceeds the threshold. The driving assistance device according to any one of claims 1 to 3, characterized in that:

5. The detection unit detects an edge portion of an object from an image captured by the external camera unit, and detects, as the latent recognition area, a divided area in which a ratio of the edge portion included in the divided area exceeds a predetermined threshold. The driving assistance device according to any one of claims 1 to 4, characterized in that:

6. The detection unit detects, for each divided area, a prediction error indicating a difference between each pixel of a predicted current image corresponding to a current captured image generated by predicting from a captured image previously captured by the external camera unit and each pixel of the current captured image captured by the external camera unit, and detects a divided area including an edge portion of an object from the captured image captured by the external camera unit, and detects the divided area in which the prediction error exceeds a predetermined threshold and which includes the edge portion exceeding the predetermined threshold as the latent recognition area. The driving assistance device according to any one of claims 1 to 4, characterized in that:

7. The information presentation control unit changes one or more of the following in accordance with the detected latent recognition area: whether or not to present the driving support information; a presentation position; and a change in the number or type of presentation. The driving assistance device according to any one of claims 1 to 6, characterized in that:

8. Further, a moving speed detection unit is provided to detect the moving speed of the moving object, The information presentation control unit changes at least the type of the driving assistance information to be presented depending on whether the moving speed of the moving object detected by the moving speed detection unit is equal to or greater than a predetermined threshold value or is less than the threshold value. The driving support device according to claim 7,

9. a movement speed detection unit that detects the movement speed of the moving object; and a movement control unit that controls the movement of the moving object in accordance with the moving speed of the moving object detected by the moving speed detection unit. The driving assistance device according to any one of claims 1 to 8, characterized in that:

10. a detection step in which a detection unit detects a latent recognition area that a human being unconsciously recognizes from an entire image capture area of ​​a captured image of at least a moving direction of the moving body, the latent recognition area being captured by an external camera unit provided on the moving body; an information presentation control step in which an information presentation control unit presents predetermined driving assistance information in an area on the information presentation unit estimated according to the detected latent recognition area; a gaze detection step in which the gaze detection unit detects the gaze of the driver, the detection unit detects, as the latent recognition area, a divided area having a predetermined feature amount exceeding a predetermined threshold, from among each of a plurality of divided areas obtained by dividing the entire imaging area; the detection unit detects, for each divided region, a prediction error that is a difference between each pixel of a predicted image predicted from a past image captured in the past by the external camera unit and each pixel of an actually captured image actually captured by the external camera unit at the time of the predicted image, and detects the divided region having the prediction error that exceeds a predetermined threshold as the latent recognition region; The information presentation control unit presents the driving assistance information to the divided area where the gaze detection unit detects a gaze retention length that is equal to or longer than a predetermined retention length and where the prediction error is less than the threshold. A driving assistance method characterized by:

11. Computer, a detection unit that detects a potential recognition area that a person unconsciously recognizes from an entire image capture area of ​​a captured image of at least a moving direction of the moving body, the potential recognition area being captured by an external camera unit provided on the moving body; an information presentation control unit that presents predetermined driving assistance information in an area on the information presentation unit that is estimated according to the detected latent recognition area; It functions as a gaze detection unit that detects the driver's gaze, the detection unit detects, as the latent recognition area, a divided area having a predetermined feature amount exceeding a predetermined threshold, from among each of a plurality of divided areas obtained by dividing the entire imaging area; the detection unit detects, for each divided region, a prediction error that is a difference between each pixel of a predicted image predicted from a past image captured in the past by the external camera unit and each pixel of an actually captured image actually captured by the external camera unit at the time of the predicted image, and detects the divided region having the prediction error that exceeds a predetermined threshold as the latent recognition region; The information presentation control unit presents the driving assistance information to the divided area where the gaze detection unit detects a gaze retention length that is equal to or longer than a predetermined retention length and where the prediction error is less than the threshold. A driving assistance program featuring:

Images