Driving support device, driving support system, driving support method, and program

Abstract

A driving assistance device according to the present disclosure comprises: an image acquisition means that acquires an image obtained by imaging a crosswalk in the travel direction from a host vehicle; a pedestrian detection means that detects a pedestrian within a predetermined distance from the crosswalk on the basis of the image; a motion acquisition means that acquires a facial state of the pedestrian and a change in a motion vector of the pedestrian; an estimation means that estimates a motion of the pedestrian with respect to the crosswalk on the basis of the facial state and the change in the motion vector; and an output means that outputs information relating to the motion of the pedestrian based on a result of the estimation.

Description

Driving assistance device, driving assistance system, driving assistance method, and recording medium

[0001] The present disclosure relates to a driving assistance device, a driving assistance system, a driving assistance method, and a recording medium.

[0002] To prevent traffic accidents, it is urgent to improve the functionality of ADAS (Advanced Driver-Assistance Systems). Although there are legal regulations requiring drivers to stop when pedestrians are crossing the street, in reality, it is largely dependent on the driver's judgment, and there are many cases where drivers overlook the situation, so the system is not being implemented thoroughly.

[0003] For this reason, there are technologies that provide drivers with information about pedestrians near crosswalks. For example, Patent Literature 1 describes a technology that estimates the state of a moving object related to crossing a road, such as crossing, waiting to cross, or near a crosswalk, based on the position of the bottom edge of a bounding box that surrounds a detected object such as a person or bicycle.

[0004] Japanese Patent Application Laid-Open No. 2022-027306

[0005] However, it can be difficult to determine how pedestrians will move near a crosswalk based on their positions.

[0006] An example of an object of the present disclosure is to provide a driving assistance device or the like that can effectively alert a driver to the presence of pedestrians near a crosswalk.

[0007] A driving assistance device in one aspect of the present disclosure includes an image acquisition means that acquires an image of a crosswalk in the direction of travel from the vehicle, a pedestrian detection means that detects pedestrians within a predetermined distance from the crosswalk based on the image, a motion acquisition means that acquires changes in the pedestrian's motion vector and facial state, an estimation means that estimates the pedestrian's motion relative to the crosswalk based on the changes in the motion vector and the facial state, and an output means that outputs information related to the pedestrian's motion based on the estimation results.

[0008] A driving assistance system in one aspect of the present disclosure includes a driving assistance device provided in the host vehicle and a driving assistance device provided in another vehicle traveling in the oncoming lane, the driving assistance device provided in the other vehicle having an image acquisition means that acquires an image of a crosswalk in the direction of travel from the other vehicle, a pedestrian detection means that detects pedestrians within a predetermined distance from the crosswalk based on the image, a motion acquisition means that acquires changes in the pedestrian's motion vector and facial state, an estimation means that estimates the pedestrian's motion relative to the crosswalk based on the changes in the motion vector and the facial state, and an output means that transmits information related to the pedestrian's motion based on the estimation results to the driving assistance device of the host vehicle, and the driving assistance device of the host vehicle notifies the driver of the host vehicle of the information related to the pedestrian's motion.

[0009] A driving assistance system in one aspect of the present disclosure includes a driving assistance device provided in the host vehicle and a video recording device provided near a crosswalk, the video recording device having an image acquisition means that acquires an image of a crosswalk within a predetermined distance, a pedestrian detection means that detects a pedestrian within a predetermined distance from the crosswalk based on the image, a motion acquisition means that acquires changes in the pedestrian's motion vector and facial state, an estimation means that estimates the pedestrian's motion relative to the crosswalk based on the changes in the motion vector and the facial state, and an output means that transmits information regarding the pedestrian's motion based on the estimation results to the driving assistance device of the host vehicle, and the driving assistance device of the host vehicle notifies the driver of the host vehicle of the information regarding the pedestrian's motion.

[0010] In one aspect of the present disclosure, a driving assistance method involves a computer acquiring an image of a crosswalk in the direction of travel from the vehicle, detecting pedestrians within a predetermined distance from the crosswalk based on the image, acquiring changes in the pedestrian's movement vectors and facial state, estimating the pedestrian's movement relative to the crosswalk based on the changes in the movement vectors and the facial state, and outputting information related to the pedestrian's movement based on the estimation results.

[0011] A recording medium according to one aspect of the present disclosure stores a program that causes a computer to execute a process of acquiring an image of a crosswalk in the direction of travel from the vehicle, detecting pedestrians within a predetermined distance from the crosswalk based on the image, acquiring changes in the pedestrian's movement vector and facial state, estimating the pedestrian's movement relative to the crosswalk based on the changes in the movement vector and the facial state, and outputting information related to the pedestrian's movement based on the estimated results.

[0012] According to the present disclosure, it is possible to effectively draw attention to the presence of pedestrians near a crosswalk.

[0013] FIG. 1 is a block diagram showing an example of the configuration of a driving assistance device in a first embodiment. FIG. 2 is a diagram showing a hardware configuration in which the driving assistance device in the first embodiment is realized by a computer device and its peripheral devices. FIG. 3 is a diagram explaining the position of a pedestrian with respect to a crosswalk in the first embodiment. FIG. 4 is an example of a screen showing information related to the movement of a pedestrian output by an output unit in the first embodiment. FIG. 5 is a flowchart showing the operation of the driving assistance device in the first embodiment. FIG. 6 is a block diagram showing an example of the configuration of a driving assistance system in a second embodiment. FIG. 7 is a diagram explaining the position of a pedestrian near a crosswalk in the second embodiment. FIG. 8 is a flowchart showing the operation of the driving assistance system in the second embodiment.

[0014] Hereinafter, with reference to the drawings, embodiments of a driving assistance device, a driving assistance system, a driving assistance method, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0015] First Embodiment A driving assistance system 10 according to this modification includes a driving assistance device 100 and a display device 200. Examples of the display device 200 include a terminal device carried by a pedestrian, or a signage or road sign provided on a road.

[0016] The driving assistance device 100 according to this embodiment performs processing to detect pedestrians near a crosswalk based on image information obtained from a dashcam of the vehicle being driven by the driver or from a video recording device installed on the road, such as a traffic light. In this embodiment, the device particularly detects pedestrians who may cross the crosswalk in the direction of travel of the vehicle.

[0017] 1 is a block diagram showing an example of the configuration of an information provision system according to the first embodiment. Referring to FIG. 1, a driving assistance device 100 includes an image acquisition unit 101, a pedestrian detection unit 102, a motion acquisition unit 103, an estimation unit 104, and an output unit 105.

[0018] FIG. 2 is a diagram illustrating an example of a hardware configuration in which the driving assistance device 100 according to the first embodiment of the present disclosure is realized by a computer device 500 including a processor. As shown in FIG. 2, the driving assistance device 100 includes a CPU (Central Processing Unit) 501, memories such as a ROM (Read Only Memory) 502 and a RAM (Random Access Memory) 503, a storage device 505 such as a hard disk for storing a program 504, a communication interface 508 for network connection, and an input / output interface 509 for inputting and outputting data. In the first embodiment, the driving assistance device 100 is connected to each component via a bus 510. The driving assistance device 100 according to the first embodiment shown in FIG. 1 can also be configured using cloud computing or the like.

[0019] The CPU 501 runs an operating system to control the entire driving assistance device 100 according to the first embodiment of the present invention. The CPU 501 also reads programs and data into memory from a recording medium 506 attached to, for example, a drive device 507. The CPU 501 also functions as the image acquisition unit 101, the pedestrian detection unit 102, the motion acquisition unit 103, the estimation unit 104, and the output unit 105 in the first embodiment, or as part of these units, and executes processing or commands in the flowchart shown in FIG. 5 (described later) based on the program.

[0020] The recording medium 506 is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory. The semiconductor memory or the like that is part of the recording medium is a non-volatile storage device that stores the program. The program may also be downloaded from an external computer (not shown) that is connected to a communication network.

[0021] As described above, the first embodiment shown in FIG. 1 is realized by the computer hardware shown in FIG. 2. However, the means for realizing each unit included in the driving assistance device 100 of FIG. 1 is not limited to the configuration described above. Furthermore, the driving assistance device 100 may be realized by a single physically combined device, or may be realized by a system consisting of two or more physically separated devices connected by wire or wirelessly. In particular, the driving assistance device 100 and the display device may be configured as a system realized by separate devices.

[0022] The image acquisition unit 101 is a means for acquiring captured images taken in the traveling direction from the host vehicle. The image acquisition unit 101 acquires image data of a moving image captured by a front camera of the host vehicle, for example. The image acquisition unit 101 outputs the acquired image data to the pedestrian detection unit 102.

[0023] The pedestrian detection unit 102 is a means for detecting pedestrians within a predetermined distance from a crosswalk based on an image. The pedestrian detection unit 102 analyzes image data captured by the image acquisition unit 101 to detect a crosswalk and determine whether a pedestrian is present within a predetermined distance (for example, within 3 m) from the crosswalk. The distance from the crosswalk may be the distance from the nearest end of the crosswalk or the distance from the center of the crosswalk. The pedestrian detection unit 102 may use existing technology to detect pedestrians within a predetermined distance from the crosswalk.

[0024] The motion acquisition unit 103 is a means for acquiring changes in the motion vectors and facial states of a pedestrian. The motion vector indicates the magnitude and direction of the change in the position of the pedestrian over a predetermined time period. Specifically, the motion acquisition unit 103 obtains the magnitude and direction of the change in the position of the pedestrian over a predetermined time period, for example, by using image data of a moving image, and divides these values ​​by the time interval between images captured, thereby calculating the magnitude and direction of the change in the position of the pedestrian over a predetermined time period.

[0025] The motion acquisition unit 103 also extracts the facial state of the pedestrian from the captured image data. In this embodiment, the facial state of the pedestrian includes the orientation of the pedestrian's face, such as the front, back (back of the head), or side (profile) as seen from the vehicle, facial movements above the neck, such as looking left and right, the pedestrian's line of sight, the pedestrian's mouth movement, the state of complexion due to drinking or poor health, and the state of wearing audio output devices, such as earphones or headphones. The motion acquisition unit 103 may also estimate these facial states based on the captured image data.

[0026] The estimation unit 104 estimates the pedestrian's movement relative to the crosswalk based on the change in the pedestrian's movement vector and the state of the pedestrian's face. Specifically, the estimation unit 104 estimates whether the pedestrian is about to cross the crosswalk based on whether the change in the pedestrian's movement vector is equal to or smaller than a predetermined value.

[0027] The estimation unit 104 also estimates the pedestrian's movement by analyzing whether the pedestrian's face is in a state in which the pedestrian is about to cross a crosswalk. For example, the estimation unit 104 extracts feature amounts in advance from a sample image of a pedestrian about to cross a crosswalk, and estimates whether the pedestrian is about to cross a crosswalk based on the extracted feature amounts.

[0028] Here, a method for estimating the behavior of a pedestrian relative to a crosswalk by the estimation unit 104 will be specifically described using drawings. FIG. 3 is a diagram illustrating the position of a pedestrian relative to the crosswalk in this embodiment. As shown in FIG. 3, there are three possible cases for a pedestrian: (1) a case where the pedestrian is further back than the crosswalk as viewed from the vehicle, (2) a case where the pedestrian is directly beside the crosswalk, and (3) a case where the pedestrian is closer to the vehicle than the crosswalk. In FIG. 3, d is an example showing a predetermined distance.

[0029] In (1), (2), and (3), if the change in the pedestrian's motion vector is equal to or less than a predetermined value and the pedestrian's face is in a state that suggests the pedestrian is about to cross the crosswalk, the estimation unit 104 estimates that the pedestrian is likely to cross the crosswalk. Examples of a face state that suggests the pedestrian is about to cross the crosswalk include a face facing the crosswalk, or a pedestrian facing the crosswalk and moving their face from side to side. However, the face state of the pedestrian detected by the estimation unit 104 is not limited to this, and any face state that suggests the pedestrian is about to cross the crosswalk may be used.

[0030] The estimation unit 104 estimates that the pedestrian is unlikely to cross the crosswalk if the pedestrian's motion vector is moving in a direction away from the crosswalk. That is, the estimation unit 104 estimates that the pedestrian is unlikely to cross the crosswalk if the direction of the pedestrian's motion vector is changing in a direction different from the direction of the crosswalk. Specifically, the estimation unit 104 estimates that the pedestrian will not cross the crosswalk if, in (1), the pedestrian's face is facing backward as seen from the driver of the vehicle and the pedestrian is moving in a direction away from the vehicle. Furthermore, the estimation unit 104 estimates that the pedestrian will not cross the crosswalk if, in (3), the pedestrian's face is facing forward (toward the driver's face) as seen from the driver of the vehicle and the pedestrian is moving in a direction approaching the vehicle.

[0031] If the change in the motion vector is greater than a predetermined value, the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk. For example, in (1) and (3), if the pedestrian is moving in a direction approaching the crosswalk by more than a predetermined amount, the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk. Furthermore, even if the change in the motion vector is less than a predetermined value, the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk if the pedestrian's face is not in a state indicating that the pedestrian is about to cross the crosswalk. For example, if the motion acquisition unit 103 estimates that the pedestrian is talking to someone based on mouth movement and the pedestrian's face is not in a state indicating that the pedestrian is about to cross the crosswalk, the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk. Furthermore, if the motion acquisition unit 103 estimates that the pedestrian is looking at the terminal device they are carrying based on the user's line of sight and the pedestrian's face is not in a state indicating that the pedestrian is about to cross the crosswalk, the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk. This is because in these cases, it is not possible to estimate whether the pedestrian will cross the crosswalk.

[0032] The output unit 105 is a means for outputting information about the pedestrian's behavior based on the estimation result. The output unit 105 warns the driver by setting a higher notification level when it is estimated that the pedestrian is likely to cross the crosswalk than when it is not estimated by the estimation unit 104 whether the pedestrian will cross the crosswalk. Note that a higher notification level means a higher level of warning. The output unit 105 may warn the driver by tightening the seat belt. In this case, the output unit 105 controls the seat belt to be tightened more tightly when it is estimated that the pedestrian is likely to cross the crosswalk than when it is not estimated whether the pedestrian will cross the crosswalk.

[0033] The output unit 105 may provide information about the pedestrian's behavior by voice via a speaker of the host vehicle, or by text or images on the screen of a display device such as a meter panel or a car navigation system of the host vehicle. In this case, if the estimation unit 104 estimates that a pedestrian may cross the crosswalk, the output unit 105 notifies the driver of the host vehicle that the pedestrian will cross the crosswalk. The output unit 105 may also notify the driver to slow down before the crosswalk. On the other hand, if the estimation unit 104 does not estimate whether the pedestrian will cross the crosswalk, the output unit 105 simply notifies the driver of the host vehicle that a pedestrian is present near the crosswalk.

[0034] The output destination of the output unit 105 is not limited to a display device in the vehicle, but may also be, for example, a display device such as a signage or road sign installed near a crosswalk. In this case, the output unit 105 transmits information about the pedestrian's movement to a display device near the crosswalk based on the V2I (Vehicle to Infrastructure) standard. The output unit 105 may also transmit information about the vehicle to a terminal device of the pedestrian using the V2P (Vehicle to Pedestrian) standard. In this case, the pedestrian can also be alerted.

[0035] FIG. 4 is an example of a screen displaying information about pedestrian movement output by the output unit in the first embodiment. The example of FIG. 4 is an example in which information about pedestrian movement is output to the screen of a car navigation system of the host vehicle. As shown in FIG. 4, a map showing the current location of the host vehicle is displayed on the left side of the screen, and an enlarged view of the crosswalk ahead of the host vehicle is displayed on the right side of the screen. In addition, the right side of the screen displays the view of pedestrians near the crosswalk. As shown in FIG. 4, the output unit 105 may visually display pedestrians near the crosswalk. In the example of FIG. 4, the pedestrian is shown standing and looking toward the crosswalk. In addition, in FIG. 4, changes in the pedestrian's movement vector and the state of the pedestrian's face may also be visually displayed.

[0036] 5 is a flowchart showing an outline of the operation of the driving assistance device 100 in the first embodiment. The processing according to this flowchart may be executed based on program control by the processor described above. The processing according to this flowchart is executed every time the pedestrian detection unit 102 detects a pedestrian within a predetermined distance from a crosswalk.

[0037] 5 is a flowchart showing the operation of the driving assistance device 100 in the first embodiment. The image acquisition unit 101 acquires an image taken from the host vehicle in the traveling direction (step S101). Next, the pedestrian detection unit 102 detects a pedestrian within a predetermined distance from the crosswalk based on the image (step S102). Next, the motion acquisition unit 103 acquires changes in the pedestrian's motion vector and facial state (step S103).

[0038] Next, the estimation unit 104 estimates the pedestrian's movement relative to the crosswalk based on the change in the pedestrian's movement vector and the state of the pedestrian's face (step S104). If the movement vector is changing in a direction away from the crosswalk (step S105; YES), the estimation unit 104 ends the process. On the other hand, if the movement vector is not changing in a direction away from the crosswalk (step S105; NO), the estimation unit 104 proceeds to step S106.

[0039] If the estimation unit 104 determines that the change in the pedestrian's motion vector is equal to or less than a predetermined value and that the pedestrian's face is in a state indicating that the pedestrian is about to cross the crosswalk (step S106; YES), the output unit 105 outputs that the pedestrian is crossing the crosswalk (step S107). On the other hand, if the estimation unit 104 determines that the change in the pedestrian's motion vector is equal to or less than a predetermined value and that the pedestrian's face is not in a state indicating that the pedestrian is about to cross the crosswalk (step S106; NO), the output unit 105 outputs that the pedestrian is present near the crosswalk (step S108). With this, the driving assistance device 100 ends the processing.

[0040] As described above, in the first embodiment, the driving assistance device 100 has the estimation unit 104 estimate the pedestrian's movement relative to the crosswalk based on the change in the vector of the pedestrian's movement and the state of the face, and the output unit 105 outputs information about the pedestrian's movement based on the estimation result. In this case, for example, when there is a possibility that a pedestrian will cross the crosswalk, the driver can be notified of content urging further caution, thereby strongly calling the driver's attention. Therefore, the presence of a pedestrian near the crosswalk can be effectively called.

[0041] [Modification of First Embodiment] Next, a modification of the first embodiment of the present disclosure will be described. Below, explanations of the modification that overlap with the above explanation will be omitted to the extent that the explanation of this modification is not unclear.

[0042] In this modification, the estimation unit 104 further estimates the recognition state of the vehicle based on the change in the motion vector and the state of the face. For example, if a pedestrian near a crosswalk is not looking in the direction of the vehicle or is distracted by something else, it is highly likely that the pedestrian is not aware of the presence of the vehicle. On the other hand, if a pedestrian near a crosswalk is looking in the direction of the vehicle, it is highly likely that the pedestrian is aware of the presence of the vehicle.

[0043] The estimation unit 104 first estimates whether a pedestrian is approaching a crosswalk based on a change in the motion vector. Then, when a pedestrian is approaching a crosswalk, the estimation unit 104 estimates whether the pedestrian is likely to be aware of the presence of the vehicle based on the orientation of the pedestrian's face. That is, when the pedestrian's face is facing the terminal device carried by the pedestrian and not facing the crosswalk or the vehicle, the estimation unit 104 estimates that the pedestrian is likely not aware of the presence of the vehicle. Then, the output unit 105 increases the notification level and outputs information related to the pedestrian's motion. On the other hand, when the pedestrian's face is facing the crosswalk or the vehicle, the estimation unit 104 estimates that the pedestrian is likely aware of the presence of the vehicle. Then, the output unit 105 outputs information related to the pedestrian's motion without increasing the notification level.

[0044] As another example, the estimation unit 104 may estimate whether or not a pedestrian is likely to be aware of the presence of the vehicle based on the state of the pedestrian's face other than the direction of the face. That is, the estimation unit 104 may similarly estimate that a pedestrian whose face is flushed due to drinking or a pedestrian wearing earphones is likely to be unaware of the presence of the vehicle. Then, the output unit 105 increases the notification level and outputs information about the pedestrian's behavior. In this modification, a specific example of a method for changing the notification level of the output unit 105 is the same as in the first embodiment, and the degree to which the notification level is increased is arbitrary. For example, even if the estimation unit 104 does not estimate whether a pedestrian will cross the crosswalk in the first example, information about the pedestrian's behavior may be output at the same notification level as when the estimation unit 104 estimates that the pedestrian is likely to cross the crosswalk.

[0045] In this modification, when it is estimated that the pedestrian may not be aware of the presence of the vehicle, the output unit 105 may transmit information regarding the presence of the vehicle to the pedestrian's terminal device. Furthermore, the terminal device that receives the information regarding the presence of the vehicle may transmit location information of the terminal device, such as GPS, to the driving assistance device 100. Furthermore, the output unit 105 of the driving assistance device 100 may display the location information of the terminal device on a map or a navigation screen. In this case, the accurate location information of the pedestrian can be displayed to the driver.

[0046] As described above, in the modified example of the first embodiment, the driving assistance device 100 has the estimation unit 104 estimate the recognition state of the vehicle based on changes in the movement vector and the state of the face. Then, the output unit 105 outputs information about the pedestrian's movement while changing the notification level depending on the recognition state of the vehicle. This makes it possible to notify the driver of pedestrian information at a higher notification level when greater attention is required.

[0047] [Second Embodiment] Next, a second embodiment of the present disclosure will be described in detail with reference to the drawings. Below, the description of the second embodiment will be omitted to the extent that it does not make the description of the present embodiment unclear.

[0048] 6 is a block diagram showing an example of the configuration of a driving assistance system 11 according to the second embodiment. The driving assistance system 11 according to this embodiment includes a driving assistance device 110 mounted on the vehicle, a driving assistance device 120 mounted on another vehicle, and a video recording device 130 installed on road infrastructure such as a traffic light near a pedestrian crossing, all of which are connected via a communication network.

[0049] For example, communication between the devices is performed based on the V2V (Vehicle to Vehicle) standard for vehicle-to-vehicle communication between the subject vehicle and another vehicle, and based on the V2I standard for communication between a vehicle and road infrastructure. In this embodiment, it is not necessary for both the driving assistance device 120 and the video recording device 130 to be present; it is sufficient for either device to be present.

[0050] In this embodiment, the driving assistance device 110, the driving assistance device 120, and the video recording device 130 can realize their functions not only by hardware but also by a computer device or software based on program control, similar to the computer device shown in FIG. 2 .

[0051] In this embodiment, it is assumed that when driving assistance devices 110, 120 and video recording device 130 detect a pedestrian within a predetermined distance from a crosswalk based on an image of the crosswalk, information about the pedestrian's behavior is transmitted to other driving assistance devices, etc. In addition to the configuration of driving assistance device 100 in the first embodiment, driving assistance device 110, driving assistance device 120 and video recording device 130 each include vehicle detection units 115, 125 and 135 that detect vehicles that require notification of information about pedestrian behavior.

[0052] The vehicle detection units 115, 125, and 135 are means for detecting vehicles approaching a crosswalk when a pedestrian is detected within a predetermined distance from the crosswalk. The vehicle detection units 115 and 125 in the driving assistance devices 110 and 120 analyze images of the crosswalk in the direction of travel from the vehicle and detect vehicles traveling within a predetermined distance in front of the crosswalk. The vehicle detection unit 135 in the video recording device 130 analyzes images of the crosswalk photographed from above and detects vehicles traveling within a predetermined distance in front of the crosswalk. The vehicle detection units 115, 125, and 135 may use existing technology to detect vehicles. In particular, the vehicle detection units 115, 125, and 135 detect vehicles that are in blind spots and cannot detect pedestrians.

[0053] In this embodiment, the output units 116, 126, and 136 transmit information about the pedestrian's movement to the detected vehicle. The output units 116 and 126 of the received vehicle notify the driver of the information. The method of notifying the driver is the same as in the first embodiment.

[0054] Here, an example of transmitting information regarding a pedestrian's behavior between the driving assistance devices 110 and 120 will be described with reference to FIG. 7 . FIG. 7 is a diagram illustrating the position of a pedestrian near a crosswalk. In the example of FIG. 7 , the host vehicle V1 is equipped with the driving assistance device 110, and another vehicle V2 is traveling in the oncoming lane of the host vehicle and equipped with the driving assistance device 120. As shown in FIG. 7 , there is a pedestrian P and an obstacle O in front of the pedestrian P on the oncoming lane of the host vehicle V1. In this case, the pedestrian P may not be detected in an image captured from the host vehicle V1 due to the blind spot of the obstacle O. In this case, the vehicle detection unit 125 in the driving assistance device 120 detects the host vehicle V1, and the output unit 126 transmits information regarding the behavior of the pedestrian P to the driving assistance device 110. The driving assistance device 110 then notifies the driver of the information regarding the pedestrian P's behavior received from the driving assistance device 120 of the other vehicle V2 via the output unit 116. 7 illustrates an example in which information about a pedestrian's movement is transmitted from the driving assistance device 120 to the driving assistance device 110, but this is not limited to this in the present embodiment. For example, information about a pedestrian's movement may be transmitted from the video recording device 130 to the driving assistance device 110 or the driving assistance device 120.

[0055] 8 is a flowchart showing an outline of the operation of the driving assistance system 11 in the second embodiment. The processing according to this flowchart may be executed based on program control by the processor described above. While this flowchart illustrates an example in which information about a pedestrian's movement is transmitted from the driving assistance device 120 to the driving assistance device 110, the present invention is not limited to this.

[0056] As shown in FIG. 8 , the processing of steps S201 to S204 is the same as the operation of the driving assistance device 100 in the first embodiment. That is, the image acquisition unit 121 acquires an image captured from another vehicle in the direction of travel (step S201). Next, the pedestrian detection unit 122 detects a pedestrian within a predetermined distance from the crosswalk based on the image (step S202). Next, the motion acquisition unit 123 acquires changes in the pedestrian's motion vector and facial state (step S203). Next, the estimation unit 124 estimates the pedestrian's motion relative to the crosswalk based on changes in the pedestrian's motion vector and facial state (step S204). Next, the vehicle detection unit 125 detects a vehicle approaching the crosswalk (step S205). Next, the output unit 126 transmits information about the pedestrian's motion based on the estimation result to the driving assistance device 110 of the host vehicle (step S206). Meanwhile, in the driving assistance device 110 of the subject vehicle, the output unit 116 notifies the driver of the subject vehicle of the information relating to the pedestrian's movement (step S207). With this, the driving assistance system 11 ends the processing.

[0057] As described above, in the second embodiment, the driving assistance system 11 notifies the driver of the own vehicle of information related to the behavior of a pedestrian detected by the driving assistance device 120 or the video recording device 130 of the other vehicle. As a result, even if the pedestrian is located in a blind spot from the own vehicle, for example, the driver can recognize information related to the pedestrian's behavior.

[0058] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0059] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0060] (Supplementary Note 1) A driving assistance device comprising: an image acquisition means that acquires an image of a crosswalk in the traveling direction from the host vehicle; a pedestrian detection means that detects a pedestrian who is within a predetermined distance from the crosswalk based on the image; a motion acquisition means that acquires a change in the motion vector and a facial state of the pedestrian; an estimation means that estimates the motion of the pedestrian with respect to the crosswalk based on the change in the motion vector and the facial state; and an output means that outputs information related to the motion of the pedestrian based on the estimation result.

[0061] (Supplementary Note 2) The driving assistance device according to Supplementary Note 1, wherein the estimation means estimates that there is a possibility that the pedestrian will cross the crosswalk when a change in the pedestrian's movement vector is equal to or less than a predetermined value and the pedestrian's face is in a state that indicates that the pedestrian is about to cross the crosswalk.

[0062] (Supplementary Note 3) The driving assistance device according to Supplementary Note 1 or Supplementary Note 2, wherein the estimation means estimates that the pedestrian is unlikely to cross the crosswalk when the movement vector of the pedestrian is changing in a direction away from the crosswalk.

[0063] (Supplementary Note 4) The driving assistance device according to any one of Supplementary Notes 1 to 3, wherein the estimation means does not estimate whether the pedestrian will cross the crosswalk if the change in the pedestrian's motion vector is greater than a predetermined value, or if the change in the motion vector is equal to or less than a predetermined value and the pedestrian's face does not appear to be in a state indicating that the pedestrian is about to cross the crosswalk.

[0064] (Supplementary Note 5) The driving assistance device according to Supplementary Note 4, wherein the output means outputs information to alert the driver of the host vehicle to the pedestrian's actions so that a notification level is higher when it is estimated that the pedestrian is likely to cross the crosswalk than when it is not estimated whether the pedestrian will cross the crosswalk.

[0065] (Supplementary Note 6) The driving assistance device according to Supplementary Note 4, wherein the output means outputs that the pedestrian will cross the crosswalk if it is estimated that there is a possibility that the pedestrian will cross the crosswalk, and outputs that the pedestrian is near the crosswalk if it is not estimated whether the pedestrian will cross the crosswalk.

[0066] (Supplementary Note 7) The driving assistance device according to any one of Supplementary Notes 1 to 6, wherein the output means outputs information about the pedestrian's movement to a display device including a signage or a road sign provided near the crosswalk.

[0067] (Supplementary Note 8) The driving assistance device according to any one of Supplementary Notes 1 to 7, wherein the output means outputs information relating to the presence of the vehicle to a terminal device carried by the pedestrian.

[0068] (Supplementary Note 9) The driving assistance device according to any one of Supplementary Notes 1 to 8, wherein the estimation means estimates a recognition state of the pedestrian to the vehicle based on a change in the movement vector and a state of the face, and the output means outputs information relating to the pedestrian's movement by changing a notification level depending on the recognition state.

[0069] (Supplementary Note 10) A driving assistance device according to any one of Supplementary Notes 1 to 9, further comprising: a vehicle detection means for detecting a vehicle approaching the crosswalk when a pedestrian is detected within a predetermined distance from the crosswalk; and the output means outputs information relating to the movement of the pedestrian to the vehicle.

[0070] (Supplementary Note 11) A driving assistance system including a driving assistance device provided in a host vehicle and a driving assistance device provided in another vehicle traveling in an oncoming lane, wherein the driving assistance device provided in the other vehicle comprises: image acquisition means that acquires an image of a crosswalk in the traveling direction from the other vehicle; pedestrian detection means that detects a pedestrian who is within a predetermined distance from the crosswalk based on the image; movement acquisition means that acquires changes in the movement vector and the facial state of the pedestrian; estimation means that estimates the movement of the pedestrian with respect to the crosswalk based on the changes in the movement vector and the facial state; and output means that transmits information related to the movement of the pedestrian based on the estimation result to the driving assistance device of the host vehicle, and the driving assistance device of the host vehicle notifies the driver of the host vehicle of the information related to the movement of the pedestrian.

[0071] (Supplementary Note 12) A driving assistance system including a driving assistance device provided in the host vehicle and a video recording device provided near the crosswalk, wherein the video recording device comprises: image acquisition means that acquires an image of the crosswalk within a predetermined distance; pedestrian detection means that detects a pedestrian within a predetermined distance from the crosswalk based on the image; motion acquisition means that acquires changes in the pedestrian's motion vector and a facial state; estimation means that estimates the pedestrian's motion with respect to the crosswalk based on the changes in the motion vector and the facial state; and output means that transmits information regarding the pedestrian's motion based on the estimation result to the driving assistance device of the host vehicle, and the driving assistance device of the host vehicle notifies the driver of the host vehicle of the information regarding the pedestrian's motion.

[0072] (Supplementary Note 13) A driving assistance method in which a computer acquires an image of a crosswalk in the traveling direction from the host vehicle, detects a pedestrian within a predetermined distance from the crosswalk based on the image, acquires changes in the movement vectors and the facial state of the pedestrian, estimates the movement of the pedestrian relative to the crosswalk based on the changes in the movement vectors and the facial state, and outputs information related to the movement of the pedestrian based on the estimation result.

[0073] (Supplementary Note 14) A recording medium storing a program that causes a computer to execute the following processes: acquiring an image of a crosswalk in the traveling direction from one's own vehicle; detecting a pedestrian within a predetermined distance from the crosswalk based on the image; acquiring changes in the movement vectors and facial states of the pedestrian; estimating the movement of the pedestrian relative to the crosswalk based on the changes in the movement vectors and the facial states; and outputting information related to the movement of the pedestrian based on the estimation results.

[0074] 10, 11 Driving assistance system 100, 110, 120 Driving assistance device 130 Video recording device 101, 111, 121, 131 Image acquisition unit 102, 112, 122, 132 Pedestrian detection unit 103, 113, 123, 133 Action acquisition unit 104, 114, 124, 134 Estimation unit 105, 116, 126, 136 Output unit 115, 125, 135 Vehicle detection unit 200 Display device 500 Computer device 501 CPU 502 ROM 503 RAM 504 Program 505 Storage device 506 Recording medium 507 Drive device 508 Communication interface 509 Input / output interface 510 Bus

Claims

1. An image acquisition means for acquiring an image of a crosswalk in the traveling direction from the host vehicle; A pedestrian detection means for detecting a pedestrian within a predetermined distance from the crosswalk based on the image; An action acquisition means for acquiring a change in the action vector of the pedestrian and the state of the face; An estimation means for estimating the action of the pedestrian with respect to the crosswalk based on the change in the action vector and the state of the face; An output means for outputting information regarding the action of the pedestrian based on the estimated result A driving support device comprising the above.

2. When the estimation means detects that the change in the action vector of the pedestrian is equal to or less than a predetermined value and the state of the face is a state of attempting to cross the crosswalk, the driving support device according to claim 1, which estimates that there is a possibility of crossing the crosswalk.

3. The estimation means does not estimate whether or not to cross the crosswalk in any of the cases where the change in the action vector of the pedestrian is greater than a predetermined value, and the case where the amount of change in the action vector is equal to or less than a predetermined value and the state of the face is not a state of attempting to cross the crosswalk. The driving support device according to claim 1 or 2.

4. When the pedestrian is estimated to have a possibility of crossing the crosswalk rather than when it is not estimated whether the pedestrian crosses the crosswalk, the output means outputs information for prompting the driver of the host vehicle to pay attention to the action of the pedestrian so that the notification level becomes higher. The driving support device according to claim 3.

5. The output means outputs information regarding the action of the pedestrian to a display device including a sign or a road sign provided near the crosswalk. The driving support device according to claim 1 or 2.

6. The estimation means estimates the recognition state of the host vehicle based on the change in the action vector and the state of the face, The output means changes the notification level according to the recognition state and outputs information regarding the action of the pedestrian. The driving support device according to claim 1 or 2.

7. When detecting a pedestrian within a predetermined distance from the crosswalk, it further includes a vehicle detection means for detecting a vehicle approaching the crosswalk, The output means outputs information regarding the action of the pedestrian to the vehicle. The driving support device according to claim 1 or 2.

8. Including a driving support device provided in the host vehicle and a video recording device, The image recording device is either a driving assistance device provided in another vehicle traveling in the oncoming lane or a device provided near a crosswalk, an image acquisition means for acquiring an image of a crosswalk within a predetermined distance, a pedestrian detection means for detecting a pedestrian within a predetermined distance from the crosswalk based on the image, a motion acquisition means for acquiring a change in the motion vector of the pedestrian and the state of the face, an estimation means for estimating the motion of the pedestrian with respect to the crosswalk based on the change in the motion vector and the state of the face, and an output means for transmitting information regarding the motion of the pedestrian based on the estimated result to a driving assistance device of the host vehicle, and the driving assistance device of the host vehicle is a driving assistance system that notifies the driver of the host vehicle of information regarding the motion of the pedestrian.

9. A computer acquires an image of a crosswalk in the traveling direction from the host vehicle, detects a pedestrian within a predetermined distance from the crosswalk based on the image, acquires a change in the motion vector of the pedestrian and the state of the face, estimates the motion of the pedestrian with respect to the crosswalk based on the change in the motion vector and the state of the face, and outputs information regarding the motion of the pedestrian based on the estimated result. A driving assistance method.

10. Acquires an image of a crosswalk in the traveling direction from the host vehicle, detects a pedestrian within a predetermined distance from the crosswalk based on the image, acquires a change in the motion vector of the pedestrian and the state of the face, estimates the motion of the pedestrian with respect to the crosswalk based on the change in the motion vector and the state of the face, and a program for causing a computer to execute a process of outputting information regarding the motion of the pedestrian based on the estimated result.