Driving assistance devices

Abstract

The present invention aims to provide a driving assistance device that can accurately estimate a passenger's line of sight. To address this issue, the driving assistance device 11 includes: a peripheral information acquisition unit 40 for acquiring peripheral information about the vehicle 1; a driver monitoring camera 50 for acquiring passenger information related to a passenger in the vehicle 1; an external information detection unit 201 for detecting external information, including the position and velocity of objects around the vehicle 1, based on the peripheral information; a line of sight estimation unit 202 for estimating the line of sight of a passenger in the vehicle 1 based on the passenger information; and a line of sight correction unit 203 for correcting the estimated line of sight based on the external information detected by the external information detection unit 201.

Description

Technical Field

[0001] The present invention relates to a driving assistance device. Background Art

[0002] A technology for detecting whether a driver is looking at a vehicle mirror is currently known (for example, see Patent Document 1). The vehicle mirror control device described in Patent Document 1 determines whether the driver is looking at the vehicle mirror and adjusts the mirror angle of the vehicle mirror based on the determination result.

[0003] [Prior art literature]

[0004] (Patent Document)

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-047019 Summary of the Invention

[0006] [Problems to be solved by the invention]

[0007] However, determining whether a passenger is looking at the mirror is technically difficult and requires high-precision estimation of the passenger's line of sight.

[0008] Therefore, an object of the present invention is to provide a driving assistance device that can estimate a passenger's line of sight with high accuracy.

[0009] A driving assistance device according to one aspect of the present disclosure (e.g., the driving assistance device 11 described later) includes: a peripheral information acquisition unit (e.g., the peripheral information acquisition unit 40 described later) for acquiring peripheral information about a vehicle (e.g., the vehicle 1 described later); a passenger information acquisition unit (e.g., the driver monitoring camera 50 described later) for acquiring passenger information related to passengers of the aforementioned vehicle; an external information detection unit (e.g., the external information detection unit 201 described later) for detecting external information including the position and speed of objects around the aforementioned vehicle based on the aforementioned peripheral information; a line of sight estimation unit (e.g., the line of sight estimation unit 202 described later) for estimating the line of sight of the passenger of the aforementioned vehicle based on the aforementioned passenger information; and a line of sight correction unit (e.g., the line of sight correction unit 203 described later) for correcting the estimated line of sight based on the aforementioned external information detected by the aforementioned external information detection unit.

[0010] The aforementioned object is another vehicle (for example, the other vehicle 300 described later). When the aforementioned external information includes the aforementioned other vehicle traveling at a position offset from the traveling direction of the aforementioned vehicle toward the lateral direction, the aforementioned external information detection unit calculates the relative speed between the aforementioned other vehicle and the aforementioned vehicle, and the aforementioned line of sight correction unit corrects the aforementioned line of sight based on the aforementioned relative speed detected by the aforementioned external information detection unit.

[0011] The sight line estimation unit determines whether the passenger has visually confirmed a specific range in the left and right directions from the side view mirror for a specific time based on the relative speed. When the passenger has visually confirmed a specific range in the left and right directions from the side view mirror for a specific time, the sight line correction unit corrects the sight line.

[0012] When the relative speed of the other vehicle is greater than or equal to a specific value, the line of sight estimating unit determines whether the passenger's line of sight follows the other vehicle within the specific range in the left and right directions from the side-view mirror. When it is determined that the passenger's line of sight does not follow the other vehicle, it is estimated that the passenger is looking in the direction of the side-view mirror. The line of sight correcting unit uses the error between the reference side-view mirror line of sight direction representing the reference of the line of sight direction of the side-view mirror and the line of sight in the estimated direction of the side-view mirror as a correction value to correct the line of sight.

[0013] The driving assistance device also includes a passenger condition estimation unit (for example, the passenger condition estimation unit 204 described later), which estimates the passenger condition of the vehicle based on the passenger information. When the passenger condition estimation unit estimates that the passenger is wearing glasses, the line of sight correction unit corrects the line of sight.

[0014] Furthermore, the visual line information acquiring unit is arranged at the center of the vehicle in the left-right direction.

[0015] The aforementioned object is another vehicle (for example, the other vehicle 300 described later). When the aforementioned external information includes the aforementioned other vehicle existing in the traveling direction of the aforementioned vehicle, the aforementioned external information detection unit detects the relative speed between the aforementioned other vehicle and the aforementioned vehicle. When the relative speed of the aforementioned other vehicle is greater than or equal to a specific amount, the aforementioned line of sight estimation unit determines whether the line of sight of the aforementioned passenger has changed. When it is determined that the line of sight of the aforementioned passenger has not changed, it is estimated that the aforementioned passenger has turned his line of sight from the driver's seat toward the traveling direction of the vehicle. The aforementioned line of sight correction unit uses the error between the reference traveling direction representing the reference of the traveling direction of the aforementioned vehicle and the aforementioned estimated passenger's traveling direction line of sight as a correction value to correct the aforementioned line of sight.

[0016] (Effects of the Invention)

[0017] According to the present invention, it is possible to provide a driving assistance device that can estimate a passenger's line of sight with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram illustrating the configuration of a vehicle according to the present embodiment.

[0019] Figure 2 This is a diagram illustrating the functional configuration of the vehicle driving assistance device according to the present embodiment.

[0020] Figure 3 This is a diagram illustrating a specific example of correcting the sight line of a passenger of a vehicle when another vehicle is traveling at a position offset from the vehicle's traveling direction toward the side.

[0021] Figure 4 1 is a diagram illustrating a specific example of correcting the sight line of a passenger of a vehicle when another vehicle is traveling ahead in the vehicle's traveling direction.

[0022] Figure 5 This is a flowchart illustrating a process of correcting the sight line of a passenger of a vehicle when another vehicle is traveling at a position offset in a lateral direction from the vehicle's traveling direction.

[0023] Figure 6 This is a flowchart illustrating a process of correcting the sight line of a passenger of a vehicle when another vehicle is traveling ahead in the vehicle's traveling direction. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the driving assistance device according to the present invention will be described with reference to the accompanying drawings.

[0025] Figure 1 This is a block diagram illustrating the configuration of the vehicle 1 according to the present embodiment. Figure 1 The outline of the vehicle 1 is shown in combination with a plan view and a side view. The vehicle 1 is, for example, a sedan-type four-wheeled passenger vehicle.

[0026] Vehicle 1 includes a control device 2. Control device 2 includes multiple electronic control units (ECUs) (autopilot ECU 20 to stop control ECU 29) that can be communicatively connected via an in-vehicle network. Each ECU functions as a computer, including a processor, typically a CPU, storage devices such as semiconductor memory, and interfaces for external devices. The storage devices store programs executed by the processor and data processed by the processor. Each ECU may also include multiple processors, storage devices, and interfaces.

[0027] The following describes the functions of each of the automatic driving ECU 20 to the stop control ECU 29. Furthermore, the number of ECUs and the functions they are responsible for can be appropriately designed, and the ECUs shown in this embodiment can be further divided or integrated.

[0028] The automatic driving ECU 20 performs control related to automatic driving of the vehicle 1. In the automatic driving, the automatic driving ECU 20 automatically controls at least any one of steering, acceleration, and deceleration of the vehicle 1.

[0029] The steering ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism for steering the front wheels based on the driver's steering operation (steering operation) of the steering wheel 31. The electric power steering system 3 also includes an electric motor for assisting the steering operation or applying the driving force required for automatic steering of the front wheels, and a sensor for detecting the steering angle. When the vehicle 1 is in automatic driving mode, the steering ECU 21 automatically controls the electric power steering system 3 in response to instructions from the automatic driving ECU 20, thereby controlling the direction of travel of the vehicle 1.

[0030] The driving assistance ECUs 22 and 23 control the camera 41, LIDAR 42, and millimeter-wave radar 43 that detect the vehicle's surroundings and process the detection results. The camera 41 captures images of the front, sides, and rear of the vehicle 1. In this embodiment, two cameras 41 are installed at the front of the vehicle 1, and one camera 41 is installed at the side and rear of the vehicle 1. The driving assistance ECUs 22 and 23 analyze the images captured by the cameras 41 to extract the outline of objects and lane markings (such as white lines) on the road.

[0031] The LIDAR 42 is a light detection and ranging (LIDAR) system used to detect objects around the vehicle 1 and measure the distance to them. In this embodiment, five LIDARs 42 are installed: one at each front corner of the vehicle 1, one at the center of the rear, and one at each rear side.

[0032] The millimeter wave radar 43 detects objects around the vehicle 1 and measures the distance to the objects. In this embodiment, five millimeter wave radars 43 are provided: one at the front center of the vehicle 1, one at each front corner, and one at each rear corner.

[0033] The driving assistance ECU 22 controls a camera 41 and various LIDARs 42 on the front of the vehicle 1 and processes information about their detection results. The driving assistance ECU 23 controls another camera 41 and various millimeter-wave radars 43 on the front of the vehicle 1 and processes information about their detection results. Having two ECUs for detecting the surrounding conditions of the vehicle 1 improves the reliability of detection results. Furthermore, the inclusion of different types of detection units—the camera 41, LIDAR 42, and millimeter-wave radar 43—enables comprehensive analysis of the vehicle 1's surroundings.

[0034] Position recognition ECU 24 controls gyro sensor 5, GPS sensor 24b, and communication device 24c and processes information related to detection and communication results. Gyro sensor 5 detects the rotational motion of vehicle 1. Position recognition ECU 24 determines the vehicle's trajectory based on the gyro sensor 5's detection results and wheel speeds.

[0035] The GPS sensor 24b detects the current location of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information, traffic information, etc. to obtain this information. The position recognition ECU 24 can access a database 24a of map information stored in a storage device and search for a route from the current location to the destination, etc.

[0036] The communication control ECU 25 includes a communication device 25a for inter-vehicle communication. The communication device 25a wirelessly communicates with another vehicle in the vicinity to exchange information between the vehicles.

[0037] The drive control ECU 26 controls the power unit 6. The power unit 6 is a mechanism that outputs a driving force that rotates the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The drive control ECU 26 controls the output of the engine in response to, for example, the driver's driving operation (accelerator operation or acceleration operation) detected by the operation detection sensor 7D provided on the accelerator pedal 7A. In addition, the drive control ECU 26 switches the gear stage of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7C. When the driving state of the vehicle 1 is automatic driving, the drive control ECU 26 automatically controls the power unit 6 in response to instructions from the automatic driving ECU 20 to control the acceleration and deceleration of the vehicle 1.

[0038] The vehicle exterior notification control ECU 27 controls the lighting devices such as the direction indicator (turn lamp) 8. Figure 1 In the example of FIG, the direction indicator lights 8 are provided at the front, the side mirrors 12, and the rear of the vehicle 1.

[0039] The vehicle interior notification control ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives information input from the driver. The input / output device 9 includes a voice output device 91, a display device 92, and an input device 93.

[0040] The voice output device 91 notifies the driver of information by voice.

[0041] The display device 92 notifies the driver of information by displaying an image. The display device 92 is arranged, for example, in front of the driver's seat, forming an instrument panel. In addition, although voice and display are listed here, vibration and light can also be used to notify information. In addition, the input / output device 9 can also use a combination of voice, display, vibration or light to notify. Furthermore, depending on the level of the information to be notified (for example, the urgency), the input / output device 9 can have different combinations or different notification methods.

[0042] The input device 93 is disposed at a position operable by the driver and is a switch group for giving instructions to the vehicle 1 , but may also include a voice input device.

[0043] The stop control ECU 29 controls the brake device 10 and a parking brake (not shown). The brake device 10 is, for example, a disc brake device, which is provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel.

[0044] The stop control ECU 29 controls the operation of the brake device 10, for example, in response to the driver's driving operation (brake operation) detected by the operation detection sensor 7E provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, the stop control ECU 29 automatically controls the brake device 10 in response to instructions from the ECU 20 to control the deceleration and stopping of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the vehicle 1 in the stopped state. Furthermore, if the transmission of the power unit 6 is equipped with a parking lock mechanism, this parking lock mechanism can also be operated to maintain the vehicle 1 in the stopped state.

[0045] Vehicle 1 also includes a driver monitoring camera 50 that captures passenger information related to passengers in vehicle 1 as captured images. Driver monitoring camera 50 is comprised of, for example, a digital camera utilizing a solid-state imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and its type is not particularly limited. Driver monitoring camera 50 is positioned in the center of vehicle 1 in the left-right direction. For example, driver monitoring camera 50 can be positioned near display device 92 or in the center of vehicle 1 in the left-right direction.

[0046] Next, the processing of the driving assistance device 11 of the vehicle 1 according to the present embodiment will be described.

[0047] Figure 2 FIG. 1 is a diagram illustrating the functional configuration of the driving assistance device 11 of the vehicle 1 according to the present embodiment. Figure 2As shown, the driving support device 11 includes a control device 2 , a peripheral information acquisition unit 40 , and a driver monitoring camera 50 .

[0048] The control device 2 includes an external information detection unit 201, a sight line estimation unit 202, a sight line correction unit 203, and an occupant status estimation unit 204. The surrounding information acquisition unit 40 includes the camera 41, LIDAR 42, and millimeter wave radar 43 described above.

[0049] The surrounding information acquisition unit 40 acquires surrounding information about the vehicle 1. For example, the surrounding information acquisition unit 40 acquires surrounding information about the front, sides, and rear of the vehicle 1. The surrounding information is, for example, images of the front, sides, and rear of the vehicle 1 acquired by the camera 41. Alternatively, the surrounding information may be data about the front, sides, and rear of the vehicle 1 acquired by, for example, a LIDAR 42 or a millimeter-wave radar 43.

[0050] As described above, the driver monitoring camera 50 acquires passenger information related to the passenger of the vehicle 1 as a captured image.

[0051] The external information detection unit 201 detects external information including the position and speed of objects around the vehicle 1 based on the surrounding information acquired by the surrounding information acquisition unit 40 .

[0052] Specifically, the external information detection unit 201 detects external information including the position and speed of another vehicle traveling around the vehicle 1 based on the captured image of the surroundings of the vehicle 1 captured by the camera 41 and the data of the surroundings of the vehicle 1 obtained by the LIDAR 42 or the millimeter wave radar 43.

[0053] The sight line estimating unit 202 estimates the sight line of the passenger of the vehicle 1 based on the captured image as passenger information acquired by the driver monitoring camera 50 .

[0054] The sight line correction unit 203 corrects the sight line of the passenger of the vehicle 1 estimated by the sight line estimation unit 202 based on the external information detected by the external information detection unit 201 .

[0055] The passenger condition estimation unit 204 estimates the passenger condition of the vehicle 1 based on the passenger information acquired by the driver monitoring camera 50. Specifically, the passenger condition estimation unit 204 analyzes the captured image as the passenger information acquired by the driver monitoring camera 50 to estimate the condition of the passenger in the vehicle 1.

[0056] More specifically, when the detected external information includes another vehicle traveling at a position offset from the traveling direction of the vehicle 1 toward the side, the external information detection unit 201 calculates the relative speed between the other vehicle and the vehicle 1 .

[0057] The sight line correction unit 203 corrects the sight line of the passenger of the vehicle 1 estimated by the sight line estimation unit 202 based on the relative speed detected by the external information detection unit 201 .

[0058] Furthermore, the sight line estimating unit 202 determines whether the passenger of the vehicle 1 has visually checked the left and right side mirror sight lines R through the side mirrors 12 for a specific time (eg, 0.3 seconds) based on the relative speed.

[0059] When the passenger visually checks the side mirror visual area R in the left and right directions through the side mirror 12 for a specific time, the visual line correction unit 203 corrects the visual line estimated by the visual line estimation unit 202 .

[0060] Figure 3 1 is a diagram illustrating a specific example of correcting the sight line of the passenger of the vehicle 1 when another vehicle 300 is traveling at a position offset from the traveling direction of the vehicle 1 toward the side direction. Figure 3 As shown, the vehicle 1 is traveling straight, and the other vehicle 300 is traveling at a position offset to the side from the traveling direction of the vehicle 1 .

[0061] A visual line A is shown as a reference visual line of the passenger H of the vehicle 1 .

[0062] The side mirror viewing area R in the right side mirror 12 of the vehicle 1 is defined as an area between the viewing direction B1 and the viewing direction B2 .

[0063] The reference side-view mirror sight line direction D1 is a sight line that represents a reference of the sight line direction of the side-view mirror 12 .

[0064] The visual line D2 indicates the visual line of the passenger H estimated by the visual line estimating unit 202 .

[0065] Furthermore, the line of sight A and the reference side-view mirror line of sight direction D1 are known values, and therefore, the area C formed by the line of sight A and the reference side-view mirror line of sight direction D1 is also a known value.

[0066] Moreover, if Figure 3 As shown, when the relative speed of another vehicle 300 is greater than or equal to a specific value (e.g., 10 km / h), the sight line estimation unit 202 determines whether the sight line of the passenger H follows the other vehicle 300 within the range of the side mirror sight line area R. For example, when the sight line of the passenger H moves along with the movement of the other vehicle 300, the sight line estimation unit 202 determines that the sight line of the passenger H follows the other vehicle 300.

[0067] When it is determined that the passenger's line of sight is not following the other vehicle 300 , the line of sight estimating unit 202 estimates that the passenger H is directing his line of sight D2 in the direction of the side view mirror 12 .

[0068] The sight line correction unit 203 corrects the sight line using an error E1 between a reference side mirror sight line direction D1 indicating a reference of the sight line direction of the side mirror 12 and a sight line D2 of an estimated side mirror direction as a correction value.

[0069] Furthermore, when the passenger condition estimating unit 204 estimates that the passenger of the vehicle 1 is wearing glasses, the sight line correction unit 203 corrects the sight line D2 estimated by the sight line estimating unit 202 .

[0070] In the above description, the side mirror 12 , the other vehicle 300 , the reference side mirror sight line D1 , and the sight line D2 all represent situations in the same side direction (eg, the right or left direction of the side mirror 12 ).

[0071] In addition to the above-described processing, when another vehicle 300 travels ahead of the vehicle 1 in the traveling direction, the driving assistance device 11 also performs processing as shown below.

[0072] Figure 4 1 is a diagram illustrating a specific example of correcting the sight line of the passenger of the vehicle 1 when another vehicle 300 travels ahead of the vehicle 1 in its traveling direction.

[0073] like Figure 4 As shown, vehicle 1 is traveling straight ahead, and another vehicle 300 is traveling ahead of vehicle 1 in the traveling direction.

[0074] The reference traveling direction D3 is a reference sight line direction indicating the traveling direction of the vehicle 1 , and the sight line D4 indicates the sight line of the passenger H estimated by the sight line estimating unit 202 .

[0075] When the detected external information includes another vehicle 300 existing in the traveling direction of the vehicle 1 , the external information detection section 201 detects the relative speed of the other vehicle 300 and the vehicle 1 .

[0076] When the relative speed of the other vehicle 300 is greater than or equal to a specific value (e.g., 10 km / h), the sight line estimation unit 202 determines whether the sight line of the passenger H has changed. Specifically, when the sight line of the passenger H has moved by more than a certain range, the sight line estimation unit 202 determines that the sight line of the passenger H has changed.

[0077] When the sight line estimating unit 202 determines that the sight line of the passenger H has not changed, it estimates that the passenger H has directed the sight line D4 from the driver's seat toward the vehicle's travel direction (for example, the straight ahead direction).

[0078] The sight line correction unit 203 corrects the sight line D4 using the error E2 between the reference traveling direction D3 indicating the reference of the traveling direction of the vehicle 1 and the estimated sight line D4 of the passenger H as a correction value.

[0079] Furthermore, in the above-described embodiment, the sight line correction unit 203 corrects the sight lines in the left and right directions, not the up and down directions of the passenger's perspective.

[0080] Figure 5 1 is a flowchart illustrating a process of correcting the sight line of the passenger of the vehicle 1 when another vehicle 300 is traveling at a position offset from the traveling direction of the vehicle 1 toward the side.

[0081] In step S1 , the surrounding information acquisition unit 40 acquires surrounding information around the vehicle 1 .

[0082] In step S2 , the driver monitoring camera 50 acquires passenger information related to a passenger of the vehicle 1 as a captured image.

[0083] In step S3, the external information detection unit 201 detects external information including the position and speed of another vehicle around the vehicle 1 based on the surrounding information acquired by the surrounding information acquisition unit 40. Furthermore, when the detected external information includes another vehicle 300 traveling at a position offset in a lateral direction from the direction of travel of the vehicle 1, the external information detection unit 201 calculates the relative speed between the other vehicle 300 and the vehicle 1.

[0084] In step S4, the passenger condition estimation unit 204 analyzes the captured image, representing passenger information, acquired by the driver monitoring camera 50 to estimate the condition of the passenger in vehicle 1. Furthermore, the line of sight estimation unit 202 determines whether the passenger condition estimation unit 204 has estimated that the passenger in vehicle 1 is wearing glasses. If the passenger in vehicle 1 is estimated to be wearing glasses (Yes), the process proceeds to step S5. On the other hand, if the passenger in vehicle 1 is not estimated to be wearing glasses (No), the process proceeds to step S10.

[0085] In step S5, the sight line estimation unit 202 determines whether the relative speed of the other vehicle 300 is greater than or equal to a specific value (e.g., 10 km / h). If the relative speed is greater than or equal to the specific value (yes), the process proceeds to step S6. On the other hand, if the relative speed is less than the specific value (no), the process proceeds to step S10.

[0086] In step S6, the sight line estimation unit 202 determines whether the passenger of the vehicle 1 has visually confirmed the left-right side mirror sight line region R through the side mirror 12 for a predetermined time (e.g., 0.3 seconds). If the passenger has visually confirmed the left-right side mirror sight line region R for the predetermined time (YES), the process proceeds to step S7. On the other hand, if the passenger has not visually confirmed the left-right side mirror sight line region R for the predetermined time (NO), the process proceeds to step S10.

[0087] In step S7, the sight line estimation unit 202 determines whether the sight line of the passenger H is following another vehicle 300 within the side mirror sight line area R. If the sight line is following another vehicle 300 (Yes), the process proceeds to step S8. On the other hand, if the sight line is not following another vehicle 300 (No), the process proceeds to step S10.

[0088] In step S8 , since the sight line estimation unit 202 determined that the passenger H's sight line was not following the other vehicle 300 in step S7 , it is estimated that the passenger H is directing the sight line D2 toward the side mirror 12 rather than toward the other vehicle 300 .

[0089] In step S9 , the sight line correction unit 203 corrects the sight line using the error E1 between the reference side mirror sight line direction D1 indicating the reference of the sight line direction of the side mirror 12 and the sight line D2 of the estimated side mirror direction as a correction value.

[0090] In step S10 , the sight line correction unit 203 identifies the sight line of the passenger H as the sight line D2 , and does not correct the sight line D2 in the estimated direction of the side mirror.

[0091] Figure 6 1 is a flowchart illustrating a process of correcting the sight line of a passenger of the vehicle 1 when another vehicle 300 is traveling ahead of the vehicle 1 in its traveling direction.

[0092] In step S11 , the surrounding information acquisition unit 40 acquires surrounding information around the vehicle 1 .

[0093] In step S12 , the driver monitoring camera 50 acquires passenger information related to a passenger of the vehicle 1 as a captured image.

[0094] In step S13, the external information detection unit 201 detects external information including the position and speed of another vehicle around the vehicle 1 based on the surrounding information acquired by the surrounding information acquisition unit 40. In addition, when the detected external information includes another vehicle 300 traveling in the vehicle travel direction of the vehicle 1, the external information detection unit 201 calculates the relative speed between the other vehicle 300 and the vehicle 1.

[0095] In step S14, the passenger condition estimation unit 204 analyzes the captured image, representing passenger information, acquired by the driver monitoring camera 50 to estimate the condition of the passenger in vehicle 1. Furthermore, the line of sight estimation unit 202 determines whether the passenger condition estimation unit 204 has estimated that the passenger in vehicle 1 is wearing glasses. If the passenger in vehicle 1 is estimated to be wearing glasses (Yes), the process proceeds to step S15. On the other hand, if the passenger in vehicle 1 is not estimated to be wearing glasses (No), the process proceeds to step S19.

[0096] In step S15, the sight line estimation unit 202 determines whether the relative speed of the other vehicle 300 is greater than or equal to a specific value (e.g., 10 km / h). If the relative speed is greater than or equal to the specific value (yes), the process proceeds to step S16. On the other hand, if the relative speed is less than the specific value (no), the process proceeds to step S19.

[0097] Alternatively, instead of using relative speed, the line of sight from the passenger's position can be used as a reference to determine whether a displacement of a specific angle relative to the other vehicle has occurred. If the displacement is greater than or equal to a specific value (e.g., 5 degrees / second in units of angle of view), processing proceeds to step S16. In other words, even if the relative speed meets the specific value, the displacement may still be small if the conversion is based on the passenger's line of sight. On the other hand, by converting the relative speed of the other vehicle 300 into a line of sight angle and setting the threshold value in units of angle of view, the line of sight estimation unit 202 can easily estimate and determine whether the line of sight is directed in the direction of the vehicle's travel.

[0098] In step S16, when the relative speed of the other vehicle 300 is greater than or equal to a specific value, the sight line estimation unit 202 determines whether the sight line of the passenger H has changed. If the sight line has changed (yes), the process moves to step S19. On the other hand, if the sight line has not changed (no), the process moves to step S17.

[0099] In step S17 , when the sight line estimation unit 202 determines that the sight line of the passenger H has not changed, it is estimated that the passenger H has directed the sight line D4 from the driver's seat toward the vehicle's travel direction (eg, the straight ahead direction).

[0100] In step S18 , the sight line correction unit 203 corrects the sight line D4 using the error E2 between the reference traveling direction D3 indicating the reference of the traveling direction of the vehicle 1 and the estimated sight line D4 of the passenger H as a correction value.

[0101] In step S19 , the sight line correction unit 203 identifies the sight line of the passenger H as the sight line D4 , and does not correct the sight line D2 in the estimated direction of the side mirror.

[0102] According to this embodiment, for example, the following effects are achieved.

[0103] The driving assistance device 11 includes a surrounding information acquisition unit 40 for acquiring surrounding information about the vehicle 1; a driver monitoring camera 50 for acquiring passenger information related to a passenger in the vehicle 1; an external information detection unit 201 for detecting external information, including the position and speed of objects surrounding the vehicle 1 (e.g., another vehicle 300), based on the surrounding information; a sight line estimation unit 202 for estimating the sight line of a passenger in the vehicle 1 based on the passenger information; and a sight line correction unit 203 for correcting the estimated sight line based on the external information detected by the external information detection unit 201. Thus, the driving assistance device 11 can accurately estimate the passenger's sight line by correcting the estimated sight line based on the external information.

[0104] Furthermore, when the object is another vehicle 300 and the external information includes another vehicle 300 traveling at a position offset laterally from the direction of travel of vehicle 1, external information detection unit 201 calculates the relative speed between another vehicle 300 and vehicle 1, and line of sight correction unit 203 corrects the line of sight based on the relative speed detected by external information detection unit 201. Thus, by correcting the estimated line of sight based on the relative speed between another vehicle 300 and vehicle 1, driving assistance device 11 can estimate the passenger's line of sight with high accuracy.

[0105] In addition, the line of sight estimation unit 202 determines whether the passenger H has visually confirmed a specific time within the range of the side mirror line of sight area R in the left and right directions from the side mirror 12 based on the relative speed. When the passenger H has visually confirmed a specific time within the range of the side mirror line of sight area R in the left and right directions from the side mirror 12, the line of sight correction unit 203 corrects the line of sight.

[0106] Thus, the driving assistance device 11 estimates the direction of the line of sight toward the side mirror 12 when the vehicle is visually confirmed within the side mirror visual area R for a specific period of time, thereby being able to estimate the line of sight with high accuracy.

[0107] Furthermore, when the relative speed of another vehicle 300 is greater than or equal to a specific value, the sight line estimating unit 202 determines whether the sight line of the passenger H is following the other vehicle 300 from the side mirror 12 within the range of the side mirror sight line region R in the left-right direction. If it is determined that the sight line of the passenger H is not following the other vehicle 300, the sight line estimating unit 202 estimates that the passenger H is looking in the direction of the side mirror 12. The sight line correcting unit 203 uses the error E1 between a reference side mirror sight line direction D1, which represents a reference for the sight line direction of the side mirror 12, and a sight line D2 in the estimated direction of the side mirror 12 as a correction value to correct the sight line D2.

[0108] Thus, the driving assistance device 11 corrects the sight line D2 using the error E1 between the reference side mirror sight line direction D1 and the sight line D2 estimated in the direction of the side mirror 12 as a correction value, thereby obtaining an error-corrected sight line.

[0109] The driving assistance device 11 also includes a passenger status estimation unit 204 that estimates the status of the occupants of the vehicle 1 based on the passenger information. When the passenger status estimation unit 204 estimates that the passenger H is wearing glasses, the line of sight correction unit 203 corrects the line of sight. In particular, when a passenger wears glasses, errors in line of sight estimation are likely to occur due to the influence of the glasses. Therefore, by correcting the line of sight when the passenger is wearing glasses, the driving assistance device 11 can estimate the line of sight with high accuracy.

[0110] Furthermore, the driver monitoring camera 50 is positioned in the center of the vehicle 1 in the left-right direction. Thus, the driving assistance device 11 can use the driver monitoring camera 50, positioned in the center of the vehicle 1 in the left-right direction, to estimate the passenger's line of sight. In particular, positioning the driver monitoring camera 50 in the center of the vehicle's width allows for wide-angle detection of the passenger. Since the camera is positioned at an angle relative to the passenger's front view, it is not affected by steering operations, etc., making it suitable for correcting the passenger's line of sight.

[0111] Furthermore, when the object is another vehicle 300 and the external information includes another vehicle 300 located in the direction of travel of vehicle 1, external information detection unit 201 detects the relative speed between another vehicle 300 and vehicle 1. When the relative speed of another vehicle 300 is greater than or equal to a predetermined value, sight line estimation unit 202 determines whether passenger H's sight line has changed. If it is determined that passenger H's sight line has not changed, it is estimated that passenger H has shifted his sight line from the driver's seat toward the direction of travel of the vehicle. Sight line correction unit 203 uses the error E2 between a reference travel direction D3, which represents a reference for the direction of travel of vehicle 1, and the estimated sight line D4 of passenger H as a correction value to correct sight line D4. Thus, driving assistance device 11 can accurately correct sight lines when vehicle 1 is traveling straight, for example.

[0112] The above describes the embodiments of the present invention. The driving assistance device 11 described above can be implemented using hardware, software, or a combination thereof. Furthermore, the control method performed by the driving assistance device 11 described above can also be implemented using hardware, software, or a combination thereof. Here, "implemented by software" means implemented by a computer reading and executing a program.

[0113] Various types of non-transitory computer readable media can be used to store the program and provide it to the computer. Non-transitory computer readable media include various types of tangible recording media. Examples of non-transitory computer readable media include magnetic recording media (e.g., hard disk drives), optical magnetic recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory)).

[0114] While one embodiment of the present invention has been described above, the present invention is not limited thereto and the detailed structure can be appropriately modified within the scope of the gist of the present invention.

[0115] Reference numerals

[0116] 1: Vehicle

[0117] 11: Driving assistance devices

[0118] 40: Peripheral Information Acquisition Department

[0119] 50: Driver monitoring camera

[0120] 201: External Information Detection Department

[0121] 202: Line of sight estimation

[0122] 203: Sight Correction Department

[0123] 204: Passenger status estimation unit

Claims

1. A driving assistance device comprising: A surrounding information acquisition unit, configured to acquire surrounding information around the vehicle; A passenger information acquisition unit, configured to acquire passenger information related to passengers of the aforementioned vehicle; an external information detection unit configured to detect external information including a position and a speed of an object around the vehicle based on the surrounding information; a sight line estimating unit for estimating a sight line of a passenger of the vehicle based on the passenger information; and, A sight line correction unit is configured to correct the estimated sight line based on the external information detected by the external information detection unit, wherein: The aforementioned object is another vehicle, When the external information includes the other vehicle traveling at a position offset to the side from the traveling direction of the vehicle, the external information detection unit calculates a relative speed between the other vehicle and the vehicle. The line of sight correction unit corrects the line of sight based on the relative speed detected by the external information detection unit. The sight line estimation unit determines whether the passenger visually checks a specific range in the left and right directions through a side mirror for a specific time based on the relative speed. When the passenger visually checks a specific range in the left and right directions through the side mirror for a specific time, the sight line correction unit corrects the sight line. When the relative speed of the other vehicle is greater than or equal to a specific value, the sight line estimating unit determines whether the passenger's sight line follows the other vehicle within the specific range in the left and right directions from the side view mirror. If it is determined that the passenger's sight line does not follow the other vehicle, the unit estimates that the passenger is looking in the direction of the side view mirror. The sight line correction unit corrects the sight line using an error between a reference side view mirror sight line direction indicating a reference of the sight line direction of the side view mirror and the estimated sight line in the direction of the side view mirror as a correction value.

2. The driving assistance device according to claim 1, wherein: The driving assistance device further includes a passenger condition estimating unit configured to estimate a passenger condition of the vehicle based on the passenger information. The sight line correction unit corrects the sight line when the passenger condition estimating unit estimates that the passenger is wearing glasses.

3. The driving assistance device according to claim 1, wherein: The passenger information acquisition unit is disposed at the center of the vehicle in a left-right direction.

4. The driving assistance device according to claim 1, wherein: When the external information includes the other vehicle existing in the traveling direction of the vehicle, the external information detecting unit detects a relative speed between the other vehicle and the vehicle. When the relative speed of the other vehicle is greater than or equal to a specific amount, the sight line estimating unit determines whether the sight line of the passenger has changed, and when it is determined that the sight line of the passenger has not changed, it is estimated that the passenger has shifted his sight line from the driver's seat toward the direction of travel of the vehicle. The sight line correction unit corrects the sight line using an error between a reference traveling direction indicating a reference of the vehicle's traveling direction and the estimated traveling direction sight line of the passenger as a correction value.

Images