Driving support system, driving support method, and program

Abstract

This driving assistance system comprises: an acquisition unit; a detection unit; and an output control unit. The acquisition unit acquires a current road video. The detection unit detects a difference between a video when the road situation is good and the current road video. The output control unit displays, for a user, a notification based on a difference, the display being at a position which is in a display region of a head-up display and which corresponds to the difference.

Description

Driving assistance system, driving assistance method, and recording medium

[0001] The present disclosure relates to a driving assistance system and the like.

[0002] There are times when the amount of rainfall is heavy, such as when a linear rain band occurs, or when the amount of snowfall is heavy. For example, Patent Document 1 describes a method for calculating the depth of a flooded area and displaying a warning image corresponding to the calculated depth overlaid on the image captured by the camera.

[0003] Furthermore, Patent Document 2 describes a technique in which a digital tachograph displays a message urging the driver to be careful of changes in the shape of the road.

[0004] Furthermore, Patent Document 3 describes that when an abnormality is found in the measurement results of the road plane, a reference image and an abnormality candidate image are displayed.

[0005] JP 2021-152917 A JP 2022-133816 A JP 2015-041188 A

[0006] Heavy rainfall and snowfall can still make driving difficult and dangerous.

[0007] An example of an object of the present disclosure is to provide a driving assistance system that aims to make driving safer.

[0008] A driving assistance system according to one aspect of the present disclosure includes an acquisition means for acquiring an image of a current road, a detection means for detecting the difference between the image of the current road and an image of the road when the road is in good condition, and an output control means for displaying a notification to a user based on the difference at a position in the display area of ​​a head-up display that corresponds to the difference.

[0009] A driving assistance method in one aspect of the present disclosure acquires an image of a current road, detects a difference between the image when the road conditions are good and the image of the current road, and displays a notification to the user based on the difference in a position corresponding to the difference in the display area of ​​a head-up display.

[0010] A program in one aspect of the present disclosure causes a computer to acquire an image of a current road, detect the difference between the image when the road conditions are good and the image of the current road, and display a notification to the user based on the difference at a position in the display area of ​​a head-up display that corresponds to the difference.

[0011] Each program may be stored in a non-transitory computer-readable recording medium.

[0012] According to the present disclosure, driving can be made safer.

[0013] 1 is a block diagram showing an example of the configuration of a driving assistance system according to a first embodiment. FIG. 2 is a flowchart showing an example of the operation of the driving assistance system according to the first embodiment. FIG. 3 is an explanatory diagram showing an example of connection between the driving assistance system and other devices. FIG. 4 is a block diagram showing an example of the configuration of a driving assistance system according to a second embodiment. FIG. 5 is an explanatory diagram showing an example of a current road with puddles. FIG. 6 is an explanatory diagram showing a comparison between an image of the current road and an image of the road when the road condition is good. FIG. 7 is an explanatory diagram showing an example of a display when the risk level is determined to be low based on the depth of puddles. FIG. 8 is an explanatory diagram showing an example of a display when the risk level is determined to be high based on the depth of puddles. FIG. 9 is an explanatory diagram showing an example of a current road with snow accumulation. FIG. 10 is an explanatory diagram showing a comparison between an image of the current road and an image of the road when the road condition is good. FIG. 11 is an explanatory diagram showing an example of displaying a supplementary lane. FIG. 12 is an explanatory diagram showing an example of displaying guidance to a detour. FIG. 13 is a flowchart showing an example of the operation of the driving assistance system according to the second embodiment. FIG. 14 is an explanatory diagram showing an example of a driving assistance system configured by a plurality of servers. FIG. 15 is an explanatory diagram showing an example of a hardware configuration of a computer.

[0014] Hereinafter, with reference to the drawings, embodiments of a driving assistance system, a driving assistance method, a program, and a non-transitory recording medium for recording the program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0015] In each embodiment, a head-up display is installed in the vehicle, and the type of the head-up display is not particularly limited. For example, the head-up display displays information on the windshield by projecting the information onto the windshield.

[0016] First Embodiment First, in the first embodiment, basic functions of a driving assistance system will be described. Fig. 1 is a block diagram showing an example of the configuration of a driving assistance system according to the first embodiment. The driving assistance system 10 includes an acquisition unit 101, a detection unit 102, and an output control unit 103.

[0017] The acquisition unit 101 acquires an image of the current road. Specifically, for example, the acquisition unit 101 acquires an image of the current road from an imaging device installed in the vehicle. For example, the imaging device captures an image of the road ahead of the vehicle through the windshield of the vehicle. Alternatively, for example, the acquisition unit 101 acquires an image of the current road from an imaging device installed on the road. For example, the imaging device may be provided in a roadside unit installed on the road. When using an image from an imaging device installed in a roadside unit, it is possible to check the current road condition a short distance ahead in the vehicle's traveling direction from the current position of the vehicle.

[0018] The detection unit 102 detects the difference between an image of a good road condition and an image of the current road. The image of a good road condition is stored in a storage means or the like and prepared in advance. Specifically, the detection unit 102 aligns the image of the current road with the image of a good road condition and detects the difference. For example, the detection unit 102 detects puddles, accumulated snow, fallen rocks, etc. The detection method used by the detection unit 102 may be any existing technology. The detection unit 102 may also detect the size of puddles, the depth of accumulated snow, and the size of fallen rocks. Note that the size of a puddle can be measured by its depth, its length and width relative to the ground, etc., but in the following description, the depth of a puddle will be used as an example of the size of a puddle.

[0019] The output control unit 103 displays a notification based on the difference for the user at a position corresponding to the difference in the display area of ​​the head-up display. The user is, for example, a driver. In the following description, a driver will be used as an example. For example, the position corresponding to the difference is a position where there is a difference visible to the driver or a position near the position where there is a difference. The notification may be, for example, information to call attention.

[0020] For example, the output control unit 103 may display information indicating the difference as a notification. Furthermore, the information indicating the difference may be information indicating which difference it is. Whether the difference is a puddle, accumulated snow, or fallen rocks may be represented by at least one graphic of color, shape, or pattern. For example, a puddle may be represented by blue, accumulated snow by yellow, and fallen rocks by red. The information indicating the difference may be the depth of the detected puddle, the depth of accumulated snow, or the size of the fallen rocks. Furthermore, the depth of the detected puddle, the depth of accumulated snow, or the size of the fallen rocks may be represented by at least one graphic of color, shape, or pattern. For example, if the depth of the puddle is less than 4 centimeters (cm), it may be represented by a blue circle, and if it is 4 cm or more, it may be represented by a blue triangle.

[0021] For example, the output control unit 103 may display, as a notification, driving instructions corresponding to the difference. For example, if a puddle is detected, the output control unit 103 may display driving instructions corresponding to the depth of the puddle. For example, if snow is detected, the output control unit 103 may display driving instructions corresponding to the depth of the snow. For example, if falling rocks are detected, the output control unit 103 may display driving instructions corresponding to the size of the falling rocks. Driving instructions may be represented by text such as "Stop," "Slow Down," "Detour," "Slow Down," "Stop," "Caution," or "Watch Out for Falling Rocks." Driving instructions may be represented by graphics of at least one of a color, a shape, and a pattern. For example, in terms of color, a blue graphic may represent "Slow Down," and a red graphic may represent "Stop." In terms of shape, a triangular graphic may represent "Slow Down," and a square graphic may represent "Stop." Taking a pattern as an example, a striped graphic may represent "slow down," and a dotted graphic may represent "stop." For example, when characters used on road signs are displayed as driving instructions, the colors, shapes, and patterns used on road signs may also be used.

[0022] Note that information indicating which difference it is, information about size and shape, such as the size of a puddle, the depth of snow, or the size of a fallen rock, and driving instructions may be combined and displayed graphically. For example, the difference may be indicated by color, information about size and shape may be displayed by pattern, and driving instructions may be displayed by shape. In this way, a combination of differences and driving instructions corresponding to the differences may be displayed using universal design. This allows the driver to recognize danger at a glance.

[0023] 2 is a flowchart showing an example of the operation of the driving assistance system 10 according to the first embodiment. The acquisition unit 101 acquires an image of the current road (step S101). Next, the detection unit 102 detects a difference between an image of a good road condition and the image of the current road (step S102). The output control unit 103 displays a notification based on the difference to the driver at a position corresponding to the difference within the display area of ​​the head-up display (step S103).

[0024] As described above, in the first embodiment, the driving assistance system 10 displays a notification based on the difference between an image of a road in good condition and an image of the current road in a position corresponding to the difference in the display area of ​​the head-up display. This makes it possible to notify the driver of dangers such as puddles, accumulated snow, and falling rocks. This makes it possible to improve driving safety.

[0025] (Embodiment 2) Next, embodiment 2 will be described in detail with reference to the drawings. In embodiment 2, a specific example of a display on a head-up display and an example of determining a risk level based on a detected difference will be described. Below, a description of content that overlaps with the above description will be omitted to the extent that the description of embodiment 2 is not unclear.

[0026] 3 is an explanatory diagram showing an example of a connection between the driving assistance system 20 and other devices. Here, an example will be described in which the driving assistance system 20 is realized by an edge server in a vehicle. For this reason, it is assumed that the driving assistance system 20 is installed in the vehicle.

[0027] For example, the driving assistance system 20 is connected to an imaging device 21, a head-up display 22, and the like via a communication network within the vehicle. The imaging device 21 captures an image of the road ahead of the vehicle, for example, through the windshield of the vehicle. Note that the number of imaging devices 21 is not particularly limited.

[0028] In addition, in Figure 3, an example has been described in which the driving assistance system 20 is realized by an edge server within the vehicle, but the driving assistance system 20 may also be realized by an edge server within the vehicle and a server outside the vehicle, etc.

[0029] 4 is a block diagram showing an example of a configuration of a driving assistance system 20 according to the second embodiment. The driving assistance system 20 includes an acquisition unit 201, a detection unit 202, an output control unit 203, a determination unit 204, and an identification unit 205.

[0030] The driving assistance system 20 is obtained by adding a determination unit 204 and an identification unit 205 to the driving assistance system 20 in the first embodiment. The acquisition unit 201 may have the function of the acquisition unit 101 described in the first embodiment as a basic function. The detection unit 202 may have the function of the detection unit 102 described in the first embodiment as a basic function. The output control unit 203 may have the function of the output control unit 103 described in the first embodiment as a basic function.

[0031] The driving assistance system 20 includes, for example, a captured image DB (Database) 2001 and a comparison image DB 2002. The captured image DB 2001 stores, for example, images of the current road captured in real time. The comparison image DB 2002 stores, for example, images of the road captured in the past when the road was in good condition.

[0032] The acquisition unit 201 acquires a video of the current road. As described in the first embodiment, for example, the acquisition unit 201 acquires a video of the current road from the imaging device 21 installed in the vehicle. For example, the acquisition unit 201 stores the acquired video in the captured video DB 2001. Alternatively, as described in the first embodiment, for example, the acquisition unit 201 acquires a video of the current road from the imaging device 21 installed on the road via a communication network.

[0033] The detection unit 202 detects the difference between an image of a good road condition and an image of the current road. The image of a good road condition is stored in the comparison image DB 2002. The difference is a difference in road conditions that may cause driving problems when traveling on the road. For example, the detection unit 202 detects puddles, accumulated snow, fallen rocks, etc. as differences. Furthermore, the detection unit 202 detects the depth of puddles, the depth of accumulated snow, the size of fallen rocks, etc. The detection method used by the detection unit 202 may be any existing technology.

[0034] The output control unit 203 displays a notification based on the difference for the driver at a position corresponding to the difference in the display area of ​​the head-up display 22. As described in the first embodiment, for example, the position corresponding to the difference is a position where the difference is visible to the driver or a position near the position where the difference is.

[0035] Here, in the display devices of car navigation systems and terminal devices such as smartphones, other images may be superimposed on images such as captured images and maps. In such cases, since the images are not superimposed on the road that the driver is looking at, it is thought that even if a large image is superimposed, the impact on driving is minimal. On the other hand, the head-up display 22 displays information superimposed on the road that the driver is actually looking at. Unlike the display devices of car navigation systems and terminal devices such as smartphones, when a large image is displayed on the head-up display 22, the driver's field of vision may be obstructed. Therefore, depending on the display, there is a risk that the image may interfere with driving.

[0036] Therefore, the output control unit 203 may display a predetermined number of characters representing the notification. If the notification is text, it is necessary to prevent the driver from being distracted by reading a long message while driving. Therefore, the predetermined number is a number that can identify the warning and is smaller, such as 2 to 4 or the number of characters used on road signs. For example, characters with a small number of characters include "Stop," "Slow Down," and "Detour." Furthermore, for example, the characters may be characters used on road signs. Examples of characters used on road signs include "Slow Down" and "Stop." Specifically, for example, when driving instructions are displayed according to a difference as described in the first embodiment, the output control unit 203 may display a road sign corresponding to the driving instructions. As described in the first embodiment, the driving instructions may be represented by at least one graphic of a color, a shape, and a pattern. Furthermore, as described in the first embodiment, whether the difference is a puddle, accumulated snow, or fallen rocks may be represented by at least one graphic of a color, a shape, and a pattern. The detected depth of puddles, depth of snow, and size of fallen rocks may be displayed graphically using colors, shapes, and / or patterns, allowing the driver to check for hazards and driving instructions at a glance.

[0037] <Notification According to Risk Level> As described above, the head-up display 22 displays information so as to be superimposed on the road that the driver is actually viewing. To prevent the display on the head-up display 22 from interfering with driving, the display position of the notification or the notification itself may be changed according to the current road risk level. For example, the determination unit 204 determines the risk level based on the difference between an image of a road in good condition and an image of the current road. The method of expressing the risk level is not particularly limited. Then, the output control unit 203 displays the notification in a position within the display area of ​​the head-up display 22 according to the risk level based on the risk level. For example, the higher the risk level, the more conspicuous the notification is displayed to the driver, and the lower the risk level, the more unobstructed the driver's view is displayed. Furthermore, the output control unit 203 may provide a difference-based notification based on the risk level. Furthermore, the output control unit 203 may not provide a difference-based notification when the risk level is low.

[0038] Here, an example in which the risk level is divided into two levels, high and low, will be taken as an example, and an example in which the display position of the notification and the notification itself change depending on the risk level will be specifically described.

[0039] First, an example of detecting the depth of a puddle will be described. Fig. 5 is an explanatory diagram showing an example of a current road with a puddle. There is a puddle on the current road. For example, an imaging device 21 installed in a vehicle captures an image of the road while the vehicle is traveling.

[0040] Fig. 6 is an explanatory diagram showing a comparison example between a current road image and a road image when the road condition is good. In Fig. 6, the detection unit 202 detects the difference between the road image when the road condition is good and the current road image. In Fig. 6, the detection unit 202 detects puddles. Furthermore, the detection unit 202 detects the depth of the puddles.

[0041] For example, the determination unit 204 may determine the level of danger according to the result of comparing the depth of the puddle with a threshold. The determination unit 204 determines that the deeper the puddle, the higher the level of danger, and that the shallower the puddle, the lower the level of danger. If the level of danger is divided into two levels, high and low, the determination unit 204 determines that the level of danger is high if the depth of the puddle is equal to or greater than the threshold, and determines that the level of danger is low if the depth of the puddle is less than the threshold.

[0042] When the risk level is low, the output control unit 203 displays a notification based on the difference for the driver in a predetermined area of ​​the display area of ​​the head-up display 22. The predetermined area is an area that is unlikely to obstruct the field of view, such as an area within 0 cm to several tens of cm from the hood side of the windshield. Several tens of cm is approximately 30 cm to 50 cm. The area that is unlikely to obstruct the field of view may be an area that is determined individually for each driver based on the position of the driver's head, face, and eyes.

[0043] 7 is an explanatory diagram showing an example of a display when the risk level is determined to be low based on the depth of a puddle. In Fig. 7, the output control unit 203 displays "slow down" as a notification to the driver based on the difference in the display area of ​​the head-up display 22 within a range of 0 cm to several tens of cm from the hood side of the windshield.

[0044] On the other hand, for example, when the risk level is high, the output control unit 203 may display a notification based on the difference to the driver at a position corresponding to the difference in the display area of ​​the head-up display 22. The position corresponding to the difference is, for example, a position that overlaps with the difference when viewed by the driver. Also, the position corresponding to the difference is a position near the position where the difference is located.

[0045] FIG. 8 is an explanatory diagram showing an example of a display when the risk level is determined to be high based on the depth of a puddle. In FIG. 8, the output control unit 203 displays "Stop" in a position next to the puddle that is visible to the driver in the display area of ​​the head-up display 22 as a notification based on the difference. The output control unit 203 also displays the depth of the puddle, "10 cm," along with "Stop." For example, the characters for "Stop" and "10 cm" in FIG. 8 are larger than the characters for "Slow down" in FIG. 7 when the risk level is low.

[0046] In this way, in the event of a danger, the notification can be displayed in a position that is more likely to catch the driver's eye. In other words, when the depth of the puddle is detected, the notification is displayed at a display position according to the depth of the puddle.

[0047] Furthermore, when the risk level is high, the output control unit 203 may display the image at a position corresponding to the difference within an area that is easily visible to the driver within the display area of ​​the head-up display 22. For example, the area that is easily visible to the driver may be a predetermined area, or may be an area that is determined individually for each driver based on the position of the driver's head, face, and eyes.

[0048] Next, an example will be described in which the depth of snow is detected. For example, the determination unit 204 determines that the deeper the snow depth is, the higher the risk level, and the shallower the snow depth is, the lower the risk level. If the risk level is divided into two levels, high and low, the determination unit 204 may determine the risk level based on the result of comparing the snow depth with a threshold value. If the snow depth is equal to or greater than the threshold value, the determination unit 204 determines that the risk level is high, and if the snow depth is less than the threshold value, the determination unit 204 determines that the risk level is low. The display position may be the same as in the example of a puddle. In other words, if the snow depth is detected, the notification will be displayed at a display position corresponding to the snow depth.

[0049] In addition, an example will be described in which the depth of a falling rock is detected. For example, the determination unit 204 determines that the larger the size of the falling rock, the higher the risk level, and the smaller the size of the falling rock, the lower the risk level. If the risk level is divided into two levels, high and low, the determination unit 204 may determine the risk level based on the result of comparing the size of the falling rock with a threshold value. The determination unit 204 determines that the risk level is high if the depth of the accumulated snow is equal to or greater than the threshold value, and determines that the risk level is low if the depth of the accumulated snow is less than the threshold value. The display position may be set in the same manner as in the example of a puddle.

[0050] Furthermore, the output control unit 203 may change the size, color, etc. of the notification depending on the level of risk. For example, the output control unit 203 may display the notification larger the higher the level of risk, and smaller the lower the level of risk. For example, the characters shown in FIG. 8 for a high level of risk are larger than the characters shown in FIG. 7 for a low level of risk. When the notification is displayed in text, the output control unit 203 may display the notification in more characters the higher the level of risk and in fewer characters the lower the level of risk. When the notification is displayed in color, the output control unit 203 may display the notification in a more noticeable color the higher the level of risk and in a less noticeable color the lower the level of risk.

[0051] The output control unit 203 may also change the notification method depending on the level of risk. The output control unit 203 may also increase the number of notification methods as the level of risk increases. For example, when the level of risk is low, the output control unit 203 may display a notification on the head-up display 22, and when the level of risk is high, the output control unit 203 may output a notification by voice and also display the notification on the head-up display 22.

[0052] <Identifying Display Position> If the output control unit 203 displays a notification near an area where the driver can see puddles, snow, fallen rocks, etc., or in an area that the driver can easily see, the notification may overlap with important information that the driver can see ahead, such as information related to dangerous driving. Important information is information that may affect the driver's judgment while driving. For example, important information includes vehicles, traffic lights, pedestrians, road signs, etc. ahead.

[0053] Therefore, the specifying unit 205 specifies an area that does not interfere with driving by the driver within the display area of ​​the head-up display 22. Specifically, for example, the specifying unit 205 specifies an area within the display area of ​​the head-up display 22 that does not overlap with any of a vehicle, traffic light, pedestrian, or road sign ahead. Then, the output control unit 203 displays a notification within the specified area.

[0054] <Lane Completion> When it is snowing, the driver may not be able to see the lanes on the road. The detection unit 202 detects that at least a portion of the lane on the current road has disappeared by calculating the difference between an image of a road in good condition and an image of the current road. FIG. 9 is an explanatory diagram showing an example of a current road with snow accumulation. There is snow accumulation on the current road. For example, the imaging device 21 installed in the vehicle captures an image of the road while the vehicle is traveling. As a result, part of the lane is hidden by the snow.

[0055] 10 is an explanatory diagram showing a comparison example between an image of a current road and an image of a road when the road is in good condition. In Fig. 10, the detection unit 202 detects that at least a part of a lane mark has disappeared on the current road as a difference between the image of the road when the road is in good condition and the image of the current road.

[0056] Next, when it is detected that at least a part of the lane markings are missing on the current road, the output control unit 203 displays the lane markings of the road based on the image of the road in good condition. That is, the output control unit 203 may display lane markings to complement the actual lane markings. Specifically, for example, the output control unit 203 displays the lane markings several meters ahead.

[0057] Here, the displayed lane color may be the same as the actual lane color or may be different from the actual lane color. For example, the displayed lane color may be a color that does not appear on the current road or scenery, or a color that does not appear in nature, so that the driver can identify the lane as a supplemented lane. For example, red is a color that appears in nature, such as a sunset, and green is a color that appears in nature, such as a tree. For example, an unnatural color may be blue.

[0058] Fig. 11 is an explanatory diagram showing an example of the display of supplementary lanes. In Fig. 11, the supplementary lanes are displayed in a color different from that of the actual lanes. This allows the driver to drive while distinguishing between the lanes that are actually visible and the supplementary lanes.

[0059] In addition, the more lanes are filled, the more drivers will be aware that the snow is having a significant impact. In this way, lane filling also serves as a warning to drivers to be more careful when driving.

[0060] <Display of Detour Route> For example, the output control unit 203 may display guidance to a detour route based on the difference as a notification according to the difference. Specifically, for example, the output control unit 203 displays guidance to a detour route when at least one of a puddle, accumulated snow, or fallen rocks is detected. In particular, when a fallen rock is detected, there is a possibility that further rockfalls will occur or that the road will be impassable due to the fallen rocks. Therefore, the output control unit 203 displays guidance to a detour route when a fallen rock is detected.

[0061] Alternatively, the output control unit 203 displays guidance to a detour route depending on the size of puddles, the depth of snow, and the size of falling rocks. For example, the level of danger may be used. When the level of danger is divided into two levels, high and low, the output control unit 203 displays guidance to a detour route when the level of danger is high, and displays guidance to a detour route when the level of danger is low.

[0062] An example of a display of guidance to a detour route will be described with reference to Fig. 12. Fig. 12 is an explanatory diagram showing an example of a display of guidance to a detour route. For example, the output control unit 203 displays an arrow urging a right turn as guidance to a detour route. This arrow indicates that if you go straight, you will come to a section where you need to be careful of falling rocks, so you should turn right before that. In addition, the output control unit 203 displays "Watch out for falling rocks" as a warning against falling rocks.

[0063] For example, a "Beware of falling rocks" message is displayed in an area within a predetermined length from the hood side of the windshield, and an arrow encouraging a right turn is displayed in an area that is easy for the driver to see.

[0064] When displaying multiple notifications, not limited to notifications regarding falling rocks, the output control unit 203 may display the notifications in different positions. Furthermore, the output control unit 203 may set priorities for the multiple notifications, and the higher the priority, the more easily the driver can see the notification displayed.

[0065] For example, when both a falling rock and a puddle are detected, the priority of the notification regarding the size of the falling rock and the notification regarding the depth of the puddle may be determined based on the respective risk levels of the size of the falling rock and the depth of the puddle, and the notification with the higher priority may be displayed preferentially in a position that is easily visible to the driver.

[0066] Furthermore, the traffic system 23 may be able to suggest a route depending on the rockfall situation. Therefore, the output control unit 203 notifies the traffic system 23 that a rockfall has been detected. This allows other vehicles to be suggested an alternative route in advance.

[0067] In addition, the example is not limited to detection of falling rocks, and the output control unit 203 may notify the transportation system 23 when snow accumulation exceeding a predetermined depth or a puddle of water exceeding a predetermined size is detected.

[0068] (Flowchart) Fig. 13 is a flowchart showing an example of the operation of the driving assistance system 20 according to the embodiment 2. Fig. 13 illustrates an example in which the risk level is divided into three levels: "low", "medium", and "high".

[0069] The acquisition unit 201 acquires an image of the current road (step S201). The detection unit 202 detects the difference between the image of the current road and an image of the road in good condition (step S202). The determination unit 204 determines the degree of risk based on the difference (step S203).

[0070] The determination unit 204 determines whether the risk level is "low" (step S204). If the risk level is "low" (step S204: Yes), the process returns to step S201. If the risk level is not "low" (step S204: No), the determination unit 204 determines whether the risk level is "high" (step S205).

[0071] If the risk level is "high" (step S205: Yes), the output control unit 203 identifies a display area of ​​the head-up display 22 that is easy for the driver to see and that does not overlap with important information (step S206). Next, the output control unit 203 displays a notification based on the difference at a position included in the display area (step S207), and proceeds to step S209.

[0072] On the other hand, if the risk level is not "high" (step S205: No), the output control unit 203 displays a notification based on the difference within a predetermined range of the display area of ​​the head-up display 22 (step S208), and proceeds to step S209.

[0073] For example, the output control unit 203 determines whether the portion where the difference was detected has been passed (step S209). If the portion has been passed (step S209: Yes), the process returns to step S201. If the portion has not been passed (step S209: No), the process returns to step S209.

[0074] As described above, in the second embodiment, the driving assistance system 20 determines the risk level of the current road based on the difference between an image of the current road and an image of the road when the road conditions are good. The higher the risk level, the more the system displays a notification in the display area of ​​the head-up display 22 at a position that overlaps with the difference or is close to the difference. Furthermore, when the risk level is low, the driving assistance system 20 displays the notification in a predetermined area of ​​the display area of ​​the head-up display 22 that is less likely to obstruct the view, such as an area within a few tens of centimeters, such as 0 to 30 cm or 50 cm, from the hood side of the windshield. This allows the position of the notification to be changed depending on the risk level. Therefore, driving can be made safer.

[0075] The driving assistance system 20 identifies an area of ​​the display area of ​​the head-up display 22 that will not interfere with the driver's driving, and displays a notification at a position within the identified area that corresponds to the difference. This makes it possible to notify the driver of imminent danger without interfering with the driver's field of vision, thereby making driving safer.

[0076] Furthermore, as mentioned above, in the case of the head-up display 22, information is superimposed on the actual road that the driver is looking at, which may obstruct the driver's view while driving. Furthermore, if a long sentence is displayed, the driver may be distracted while driving by reading the sentence. Therefore, the driving assistance system 20 may display at least one graphic of a color, shape, or pattern that represents the detected difference. This allows danger to be represented using a universal design, such as a color, shape, or pattern. Therefore, the driver can see at a glance what kind of danger is in front of them, thereby improving driving safety.

[0077] Furthermore, if the difference is a puddle, the driving assistance system 20 displays at least one graphic of the color, shape, and pattern that represents the size of the puddle. This makes it possible to represent the danger of a puddle using universal design such as color, shape, and pattern. Therefore, the driver can check the size of the puddle at a glance, thereby making driving safer.

[0078] Furthermore, if the difference is snow accumulation, the driving assistance system 20 displays at least one graphic of a color, shape, or pattern that indicates the depth of snow. This allows the danger of snow accumulation to be indicated using universal design such as color, shape, or pattern. Therefore, the driver can check the depth of snow at a glance, thereby making driving safer.

[0079] Furthermore, if the difference is a falling rock, the driving assistance system 20 displays at least one graphic of the color, shape, and pattern that represents the size of the falling rock. This allows the driver to check the size of the falling rock at a glance, and thus the risk of falling rocks can be represented by a universal design such as color, shape, and pattern, thereby improving driving safety.

[0080] Furthermore, the driving assistance system 20 displays driving instructions corresponding to the difference as a notification based on the difference. The driving assistance system 20 may also display road signs corresponding to the driving instructions as instructions. This allows the driver to immediately understand the instructions and the road signs corresponding to the instructions, thereby making driving safer.

[0081] Furthermore, the driving assistance system 20 displays road lanes based on the image when the road conditions are good. This allows the driver to drive more safely by following the displayed lanes even if the actual lanes on the road are hidden.

[0082] Furthermore, the driving assistance system 20 displays guidance to a detour route according to the difference. This allows the driver to drive along the guidance when the guidance to a detour route is displayed, thereby enabling the driver to avoid danger.

[0083] Furthermore, when a rockfall is detected, the driving assistance system 20 notifies the traffic system 23 of the detection of the rockfall. This allows other vehicles to be notified of the rockfall situation in advance, thereby improving driving safety.

[0084] This concludes the description of each embodiment. The embodiments may be used in combination as appropriate. For example, various display methods have been described in each embodiment, but the display methods may be combined as appropriate. Furthermore, the output control unit 203 may perform output on another output device in addition to displaying on the head-up display 22. For example, the output control unit 203 may control audio output from an audio output device in addition to displaying on the head-up display 22.

[0085] Furthermore, the embodiments are not limited to the examples described above and can be modified in various ways. Furthermore, the configuration of the driving assistance systems 10 and 20 in the embodiments is not particularly limited. For example, the driving assistance systems 10 and 20 may be realized by a single device, such as a single server. When each functional unit of the driving assistance systems 10 and 20 is realized by a single device, the single device may be called, for example, a driving assistance device, an information processing device, or the like, and is not particularly limited. Alternatively, the driving assistance systems 10 and 20 in the embodiments may be realized by different devices for different functions or data. For example, each functional unit may be configured by multiple servers and realized as the driving assistance systems 10 and 20.

[0086] 14 is an explanatory diagram showing an example in which the driving assistance system 20 is configured with a plurality of servers. In FIG. 14, the driving assistance system 20 has an edge server 31, a roadside device 32, and another edge server 31.

[0087] There are no particular limitations on which functional units and DBs each device included in the driving assistance system 20 has.

[0088] For example, the edge server 31 in the vehicle has an output control unit 203. The roadside device 32 may have a comparison image DB 2002. The cloud server 33 may have an acquisition unit 201, a detection unit 202, a determination unit 204, an identification unit 205, and an imaged image DB 2001. In such a case, the edge server 31 in the vehicle sequentially transmits images captured by the imaging device 21 to the cloud server 33 and controls display on the head-up display 22. The cloud server 33 acquires current images from the edge server 31 and acquires images of a road in good condition from the roadside device 32, and performs various processes.

[0089] For example, the edge server 31 in the vehicle has an output control unit 203. The roadside device 32 may have a comparison image DB 2002 and may be equipped with an imaging device 21 that captures images of the current road. The cloud server 33 may have an acquisition unit 201, a detection unit 202, a determination unit 204, an identification unit 205, and an imaged image DB 2001. In such a case, the edge server 31 in the vehicle controls the display on the head-up display 22. The roadside device 32 sequentially delivers images captured by the imaging device 21 and images of a road in good condition. The cloud server 33 acquires the current image and the images of a road in good condition from the roadside device 32 and performs various processes.

[0090] 14 , the driving assistance system 20 may be configured with an edge server 31 and a cloud server 33. Furthermore, the driving assistance system 20 may be configured with a roadside device 32 and an edge server 31. For example, the edge server 31 in a vehicle has an output control unit 203. The roadside device 32 may have an acquisition unit 201, a detection unit 202, a determination unit 204, an identification unit 205, a captured image DB 2001, and a comparison image DB 2002, and may also include an imaging device 21 that captures an image of the current road. The edge server 31 in the vehicle controls the display on the head-up display 22. The roadside device 32 performs various processes and instructs the edge server 31 to output the results.

[0091] In each embodiment, each piece of information or each DB may include a portion of the information described above. Each piece of information or each DB may also include information other than the information described above. Each piece of information or each DB may be divided into multiple DBs or multiple pieces of information in more detail. Thus, the method for realizing each piece of information or each DB is not particularly limited.

[0092] Furthermore, each display is an example and is not particularly limited. Information not shown may be added to each display.

[0093] Furthermore, the process of generating information to be displayed on the head-up display 22 may be performed by the output control unit 203. Furthermore, this process may be performed by the head-up display 22.

[0094] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration when each device, such as the driving assistance systems 10 and 20, described in each embodiment is implemented by a computer will be described. Fig. 15 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be implemented using any combination of a computer 80 and a program as shown in Fig. 15.

[0095] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0096] The processor 801 controls the entire computer 80. Examples of the processor 801 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The computer 80 includes a storage unit such as a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801 .

[0097] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0098] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0099] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0100] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 15 . The computer 80 may have components other than those shown in FIG. 15 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0101] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0102] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0103] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. Furthermore, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0104] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0105] The driving assistance method described in each embodiment is realized by a driving assistance system, or by a computer such as a server or a terminal device executing a prepared program.

[0106] The programs described in each embodiment are recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then read from the recording medium and executed by a computer. The programs may also be distributed via a communication network NT.

[0107] The functions of each of the components of the driving assistance system in each embodiment described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0108] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0109] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0110] (Supplementary Note 1) A driving assistance system comprising: an acquisition means for acquiring an image of a current road; a detection means for detecting a difference between an image of the road when the road is in good conditions and the current image of the road; and an output control means for displaying a notification to a user based on the difference in a display area of ​​a head-up display at a position corresponding to the difference. (Supplementary Note 2) The driving assistance system according to Supplementary Note 1, further comprising: a determination means for determining a level of danger of the current road based on the difference, wherein the output control means causes the notification to be displayed in a position in the display area of ​​the head-up display that overlaps with the difference the higher the level of danger. (Supplementary Note 3) The driving assistance system according to Supplementary Note 1 or 2, further comprising: an identification means for identifying an area in the display area of ​​the head-up display that will not interfere with driving by the user, wherein the output control means causes the notification to be displayed in a position within the identified area in the display area of ​​the head-up display that corresponds to the difference. (Supplementary Note 4) The driving assistance system according to any one of Supplements 1 to 3, wherein the output control means displays a graphic of at least one of a color, a shape, and a pattern that represents the detected difference. (Supplementary Note 5) The driving assistance system according to Supplementary Note 4, wherein, if the difference is a puddle, the detection means detects a size of the puddle from the difference, and the output control means displays a graphic of at least one of a color, a shape, and a pattern that represents the size of the puddle. (Supplementary Note 6) The driving assistance system according to Supplementary Note 4, wherein, if the difference is accumulated snow, the detection means detects a depth of the accumulated snow from the difference, and the output control means displays a graphic of at least one of a color, a shape, and a pattern that represents the depth of the accumulated snow. (Supplementary Note 7) The driving assistance system according to Supplementary Note 4, wherein, if the difference is a falling rock, the detection means detects a size of the falling rock from the difference, and the output control means displays a graphic of at least one of a color, a shape, and a pattern that represents the size of the falling rock. (Supplementary Note 8) The driving assistance system according to any one of Supplementary Notes 1 to 7, wherein the output control means displays an instruction regarding driving corresponding to the difference.(Supplementary Note 9) The driving assistance system according to Supplementary Note 8, wherein the output control means displays road signs corresponding to the driving instructions. (Supplementary Note 10) The driving assistance system according to any of Supplements 1 to 9, wherein the output control means displays lanes of the road based on the image when the road condition is good. (Supplementary Note 11) The driving assistance system according to any of Supplements 1 to 10, wherein the output control means displays guidance to a detour route according to the difference. (Supplementary Note 12) The driving assistance system according to any of Supplements 1 to 11, wherein, when a rockfall is detected, the output control means notifies a traffic system that the rockfall has been detected. (Supplementary Note 13) The driving assistance system according to any of Supplements 1 to 12, wherein the acquisition means acquires the image of the current road from an imaging device attached to a vehicle. (Supplementary Note 14) The driving assistance system according to any of Supplements 1 to 13, wherein the acquisition means acquires the image of the current road from an imaging device installed on the road. (Supplementary Note 15) A driving assistance method comprising: acquiring an image of a current road; detecting a difference between an image of the road when the road conditions are good and the current image of the road; and displaying a notification to the user based on the difference at a position corresponding to the difference in a display area of ​​a head-up display. (Supplementary Note 16) A non-transitory computer-readable recording medium having recorded thereon a program that causes a computer to execute the following processes: acquiring an image of the current road; detecting a difference between an image of the road when the road conditions are good and the current image of the road; and displaying a notification to the user based on the difference at a position corresponding to the difference in a display area of ​​a head-up display. (Supplementary Note 17) A program that causes a computer to execute the following processes: acquiring an image of the current road; detecting a difference between an image of the road when the road conditions are good and the current image of the road; and displaying a notification to the user based on the difference at a position corresponding to the difference in a display area of ​​a head-up display.

[0111] 10, 20 Driving assistance system 21 Imaging device 22 Head-up display 23 Transportation system 31 Edge server 32 Roadside device 33 Cloud server 80 Computer 101, 201 Acquisition unit 102, 202 Detection unit 103, 203 Output control unit 204 Determination unit 205 Identification unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus 2001 Captured image DB 2002 Comparison image DB NT Communication network

Claims

1. An acquisition means for acquiring an image of the current road, A detection means for detecting a difference between an image when the condition of the road is good and the image of the current road, An output control means for displaying a notification based on the difference to the user at a position corresponding to the difference in the display area of the head-up display, A driving support system comprising the above.

2. A determination means for determining the degree of danger of the current road based on the difference, Comprising, The output control means displays the notification at a position overlapping with the difference in the display area of the head-up display, the higher the degree of danger. The driving support system according to claim 1.

3. A specifying means for specifying an area in the display area of the head-up display that does not interfere with the driving by the user, Comprising, The output control means, Displays the notification at a position within the specified area in the display area of the head-up display and corresponding to the difference. The driving support system according to claim 1 or 2.

4. The output control means displays at least one graphic of a color, shape, and pattern representing the detected difference. The driving support system according to claim 1 or 2.

5. When the difference is a puddle, the detection means detects the size of the puddle from the difference, The output control means displays at least one graphic of a color, shape, and pattern representing the size of the puddle. The driving support system according to claim 4.

6. When the difference is snow accumulation, the detection means detects the depth of the snow accumulation from the difference, The output control means displays at least one graphic of a color, shape, and pattern representing the depth of the snow accumulation. The driving support system according to claim 4.

7. When the difference is a rockfall, the detection means detects the size of the rockfall from the difference, The output control means displays at least one graphic of a color, shape, and pattern representing the size of the rockfall. The driving support system according to claim 4.

8. The output control means displays an instruction regarding driving corresponding to the difference. The driving support system according to claim 1 or 2.

9. A computer, Acquires an image of the current road, Detect the difference between the video when the road condition is good and the current video of the road, and display a notification based on the difference to the user at a position corresponding to the difference in the display area of the head-up display. A driving support method.

10. Cause a computer to acquire a video of the current road, detect the difference between the video when the road condition is good and the current video of the road, and display a notification based on the difference to the user at a position corresponding to the difference in the display area of the head-up display. A program for executing the process.