Driving assistance device and driving assistance unit

Abstract

To simply and accurately output a warning.SOLUTION: A driving support device that can be attached to and detached from a vehicle via a detachable unit, comprises: an acquisition unit that acquires one or more images of a surrounding situation of the vehicle; and a notification control unit that determines a notification intensity of a notification to a driver of the vehicle based on a change related to an object in the one or more images, and causes a notification unit to make a notification of the determined notification intensity. in the case where it is predicted that the vehicle will deviate from a lane in which the vehicle is traveling based on information obtained from the one or more images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a driving assistance unit, an application program, a driving assistance method, and a program. [Background technology]

[0002] Conventionally, an alarm device has been disclosed that, when swaying of a vehicle is detected, if control is being executed to prevent the vehicle from deviating from its lane, and if the driver is not touching the steering operation unit that accepts steering operations of the vehicle, does not cause an alarm to be output from an alarm output unit (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 153606 [Patent Document 2] Patent Publication No. 2021-066197 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned techniques, there are cases where it is not possible to output a warning simply and accurately.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide a driving assistance device, a driving assistance unit, an application program, a driving assistance method, and a program that can output an alarm simply and accurately. [Means for solving the problem]

[0006] A driving assistance device, a driving assistance unit, an application program, a driving assistance method, and a program according to the present invention employ the following configuration. (1): A driving assistance device according to one aspect of the present invention is a driving assistance device that can be attached to and detached from a vehicle via an attachment / detachment unit, and includes: an acquisition unit that acquires one or more images of the situation around the vehicle; and a notification control unit that, when it is predicted that the vehicle will deviate from the lane it is traveling in based on information obtained from the one or more images, determines a notification intensity for the driver of the vehicle based on changes in objects in the one or more images, and causes the notification unit to issue a notification of the determined notification intensity.

[0007] (2): In the above aspect (1), the object is a road dividing line of the lane on which the vehicle is traveling and the driver of the vehicle, and the change in the object is a change in the position of the vehicle relative to the road dividing line and a change in the behavior of the driver.

[0008] (3): In the above aspect (1) or (2), the one or more images include an image of the face of the driver driving the vehicle, and when the direction of the driver's face or gaze based on the one or more images is outside a predetermined range, the notification control unit causes the notification unit to issue a notification with a stronger notification intensity than when the direction of the driver's face or gaze is not outside the predetermined range.

[0009] (4): In any of the above aspects (1) to (3), the one or more images include an image of a face of a driver who is driving the vehicle, and the notification control unit, when (a1) it is determined that the direction of the face or line of sight of the driver based on the one or more images is in the traveling direction of the vehicle and not facing leftward for a predetermined period of time or more, and further (a2) it is determined that the vehicle is swaying leftward based on the one or more images, causes the notification unit to issue a notification of a warning of a predetermined strength, and when one or both of (a1) and (a2) are not satisfied, causes the notification unit to issue a notification of a warning of a weaker strength than the notification of the predetermined strength. (b1) if it is determined that the direction of the driver's face or line of sight based on the one or more images is in the direction of travel of the vehicle and not facing to the right for a predetermined period of time or longer, and (b2) if it is determined that the vehicle is swaying to the right based on the one or more images, it issues a warning of a predetermined strength, and if either or both of (b1) and (b2) are not met, it issues a warning of a strength weaker than the warning of the predetermined strength, or it does not issue a warning.

[0010] (5): In the aspect of (4) above, the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, and the notification control unit, when (a1) and (a2) are satisfied and (a3) ​​a vehicle is present in the adjacent rear area on the left side of the vehicle based on the one or more images, causes the notification unit to notify a third warning that is stronger than the predetermined intensity warning, when (a1) and (a2) are satisfied but (a3) ​​is not satisfied, causes the notification unit to notify a second warning that is stronger than the predetermined intensity and weaker than the third warning, when (b1) and (b2) are satisfied and (b3) a vehicle is present in the adjacent rear area on the right side of the vehicle based on the one or more images, causes the notification unit to notify a third warning that is stronger than the predetermined intensity warning, and when (b1) and (b2) are satisfied but (b3) is not satisfied, causes the notification unit to notify the second warning.

[0011] (6): In any of the above aspects (1) to (5), the one or more images include an image of one or both of the driver's arm and hand, and the notification control unit issues a warning of a predetermined strength when (c1) it determines based on the one or more images that the hand or arm is not performing an operation to control steering to move the vehicle to the left, and (c2) it determines based on the one or more images that the vehicle is swaying to the left, and issues a warning of a weaker strength than the warning of the predetermined strength when either or both of (c1) and (c2) are not satisfied. (d1) if it is determined based on the one or more images that the hand or arm is not performing an operation to control steering to move the vehicle to the right, and (d2) if it is determined based on the one or more images that the vehicle is swaying to the right, it issues a warning of a predetermined strength, and if one or both of (d1) and (d2) are not satisfied, it issues a warning of a strength weaker than the warning of the predetermined strength, or it does not issue a warning.

[0012] (7): In any of the above-mentioned aspects (6), the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, and the notification control unit satisfies (c1) and (c2), and (c3) when a vehicle is present in the adjacent rear area on the left side of the vehicle based on the one or more images, causes the notification unit to notify a third warning notification having a strength stronger than the warning notification of the predetermined strength, and satisfies (c1) and (c2), and (c3) If the conditions are not met, a second warning notification is made that is stronger than the warning notification of the predetermined strength and weaker than the third warning notification; if (d1) and (d2) are met, and (d3) a vehicle is present in the adjacent rear area on the right side of the vehicle based on the one or more images, the notification unit is caused to issue a third warning notification that is stronger than the warning notification of the predetermined strength; if (d1) and (d2) are met and (d3) is not met, the notification unit is caused to issue a second warning notification.

[0013] (8): In any of the above aspects (1) to (7), the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, and when the notification control unit determines that a vehicle is present in the adjacent rear area based on the one or more images, it causes the notification unit to issue a warning notification of a stronger intensity than the predetermined warning intensity.

[0014] (9): A driving assistance unit according to one embodiment of the present invention includes a driving assistance device according to any one of the above embodiments (1) to (8), which includes a communication unit that transmits information for causing the notification unit to notify the notification; a first camera that is not connected to the vehicle's in-vehicle network and that captures images including the surrounding conditions in front of the vehicle, which are included in the one or more images; a second camera that is not connected to the vehicle's in-vehicle network and that captures images including the driver, the window on the driver's side of the vehicle, the window on the passenger's side of the vehicle, and the area behind the vehicle, which are included in the one or more images; and a housing that houses the driving assistance device, the first camera, and the second camera.

[0015] (10): An application program according to one aspect of the present invention is an application program installed in a mobile terminal device that is not connected to the vehicle's in-vehicle network, and causes a computer of the mobile terminal device to execute the following processes: acquiring one or more images of the situation around the vehicle; and, if it is predicted that the vehicle will deviate from the lane it is traveling in based on information obtained from the one or more images, determining the notification intensity of a notification to the driver of the vehicle based on changes in objects in the one or more images, and causing a notification unit to notify the driver of the vehicle of the determined notification intensity.

[0016] (11): A driving assistance method according to one aspect of the present invention includes a process in which a computer of a driving assistance device that is detachable from a vehicle via a detachable unit acquires one or more images of the situation around the vehicle, and when it is predicted that the vehicle will deviate from the lane it is traveling in based on information obtained from the one or more images, determines the notification intensity of a notification to the driver of the vehicle based on changes in objects in the one or more images, and causes a notification unit to notify the driver of the notification with the determined notification intensity.

[0017] (12): A control program according to one aspect of the present invention is a program for causing a computer of a driving assistance device that is detachable from a vehicle via a detachable unit to execute the following processes: acquiring one or more images of the situation around the vehicle; and, if it is predicted that the vehicle will deviate from the lane it is traveling in based on information obtained from the one or more images, determining the notification intensity of a notification to the driver of the vehicle based on changes in objects in the one or more images, and causing a notification unit to notify the driver of the notification with the determined notification intensity. [Effects of the Invention]

[0018] According to (1)-(11), a warning can be output simply and accurately. For example, the driving assistance device has a simple configuration such as a unit that can be retrofitted to a vehicle. Furthermore, the driving assistance device can accurately determine vehicle sway, deviation, etc. using images.

[0019] According to (2), (3) or (5), the driver's intention is taken into consideration, so that the sway warning can be output with higher accuracy.

[0020] According to (4), (6) or (7), surrounding vehicles are taken into consideration, so that a more appropriate warning is output and control that takes the surroundings into consideration is realized. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram showing an example of the configuration and usage environment of a driving monitoring device 1 including a driving support device according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of an operation monitoring device 1. FIG. [Figure 3] 3 is a diagram showing an example of an image acquired by the operation monitoring device 1. FIG. [Figure 4] 3 is a diagram for explaining an outline of the warning process performed by the control unit 30. FIG. [Figure 5] 3 is a diagram showing an example of a functional unit included in a departure determination unit 40. FIG. [Figure 6] 10 is a diagram for explaining the processing of an offset processing unit 40C. FIG. [Figure 7] FIG. 10 is a diagram for explaining a noise region. [Figure 8] 10 is a diagram for explaining the processing of the integration processing unit 42. FIG. [Figure 9] 10 is a flowchart showing an example of the flow of processing executed by an intention determination unit 50. [Figure 10] FIG. 10 is a diagram illustrating an example of a checklist. [Figure 11] 10 is a flowchart showing an example of the flow of processing executed by an intention determination unit 50. [Figure 12] 10 is a flowchart showing another example of the flow of the process executed by the intention determination unit 50. [Figure 13] FIG. 10 is a diagram for explaining an intention flag "TRUE." [Figure 14] 10 is a flowchart showing one routine of an example of the flow of processing executed by the alarm control unit 60. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of a driving assistance device, a driving assistance unit, an application program, a driving assistance method, and a program according to the present invention will be described with reference to the drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the backward direction of the vehicle is defined as the negative X direction, the width direction of the vehicle is defined as the right direction relative to the positive X direction, the left direction is defined as the negative Y direction, and the height direction of the vehicle, which is perpendicular to the X and Y directions, is defined as the positive Z direction.

[0023] [Overall configuration] FIG. 1 is a diagram showing an example of the configuration and usage environment of a driving monitoring device 1 (driving assistance system) including a driving assistance device according to an embodiment. The driving assistance device 20 is mounted on a driving monitoring device 1, such as a drive recorder, that is retrofitted to a vehicle (hereinafter referred to as "vehicle M"). That is, the driving monitoring device 1 (driving assistance device 20) is detachable from the vehicle M via an attachment / detachment unit. The driving assistance device 20 is, for example, a device that is not connected to an in-vehicle network of the vehicle M. The in-vehicle network is an in-vehicle network that uses a communication standard such as CAN (Controller Area Network) to which ECUs (Electronic Control Units) related to vehicle control, such as drive, braking, steering, various vehicle sensors, driving-related controls, and operation buttons of the vehicle M, are connected. The network of the vehicle M for the HMI (Human Machine Interface) may be excluded from the above-mentioned in-vehicle network.

[0024] FIG. 2 is a diagram showing an example of the driving monitoring device 1. The driving monitoring device 1 is attached, for example, to the front window glass near the rearview mirror above the front windshield. The driving monitoring device 1 stores images (moving images) captured by the front camera 10 and the rear camera 12 in, for example, a storage device (not shown). The driving monitoring device 1 may be equipped with, for example, a display device such as an LCD (Liquid Crystal Display) and configured to display the images (moving images) stored in the storage device on the display device.

[0025] The driving monitoring device 1 includes, for example, a front camera 10, a rear camera 12, a detachable unit 14, and a driving assistance device 20. Each of the front camera 10 and the rear camera 12 is a digital camera using a solid-state imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The front camera 10 repeatedly (periodically) captures images of the surroundings in front of the vehicle M from the position where the driving monitoring device 1 is attached. The rear camera 12 repeatedly (periodically) captures images of the surroundings behind the vehicle M from the position where the driving monitoring device 1 is attached. The horizontal angle of view captured by each of the front camera 10 and the rear camera 12 is, for example, 180°. In other words, the driving monitoring device 1 captures images of a 360° range around the vehicle M from the position where it is attached. Note that, although the present embodiment will be described assuming that the front camera 10 and the rear camera 12 are used, other cameras may be used instead of or in addition to these. That is, the camera may be configured to capture the images used in this embodiment (an image of the road dividing lines and an image of the driver's face). The detachable unit 14 is, for example, a member for attaching the driving monitoring device 1 to the vehicle M. The detachable unit 14 is, for example, any member such as a suction cup, a sticker, a bracket, or other supporting member.

[0026] 3 is a diagram showing an example of an image acquired by the driving monitoring device 1. The front camera 10 captures an image of the area in front of the vehicle M through the front windshield. The rear camera 12 captures images of the driver of the vehicle M, the driver's seat window, the passenger seat window, the area behind the vehicle, etc.

[0027] The image captured by the front camera 10 (hereinafter referred to as the "front image") includes, as subjects, targets in front of the vehicle M that can be seen through the front windshield and targets on the left and right (front) of the vehicle M that can be seen through the left and right side window glasses, such as other vehicles, pedestrians, bicycles, fixed objects, and road dividing lines. The driving assistance device 20 recognizes these targets through image recognition processing.

[0028] The image captured by the rear camera 12 (hereinafter referred to as the "rear image") captures the interior of the vehicle M, the rear of the vehicle M as seen through the rear window glass, and objects present on the left and right (rear) sides of the vehicle M as seen through the left and right side window glasses. Therefore, the rear image captured by the rear camera 12 also captures at least the driver of the vehicle M to which the driving monitoring device 1 is attached as a subject. The driving monitoring device 1 may be attached at any location as long as the above-mentioned area is a position where the front camera 10 and the rear camera 12 can capture images.

[0029] The driving assistance device 20 outputs a warning via the terminal device T based on the result of referring to the forward image output by the forward camera 10 and the rearward image output by the rearward camera 12.

[0030] The terminal device T is a portable terminal device, such as a smartphone or a tablet terminal, used by a driver who drives the vehicle M to which the driving monitoring device 1 is attached. The terminal device T, for example, runs an application for receiving driving assistance from the driving assistance device 20. The application displays an image on a display device or emits a sound from a speaker based on information or notification transmitted by the driving assistance device 20. The terminal device T is an example of a "notification unit." The terminal device T is, for example, detachably attached to the vehicle M and used. For example, a holder for the terminal device T having a detachable part is provided on one or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder. Instead of the terminal device T, a navigation system of the vehicle M or a display unit or speaker of the vehicle M may output an alarm based on an instruction from the driving assistance device 20. Furthermore, instead of the terminal device T, a display unit (notification unit), a speaker (notification unit), or the like may be provided in the driving monitoring device 1.

[0031] As shown in FIG. 1 , the driving assistance device 20 includes, for example, a recognition unit 22, an estimation unit 24, a control unit 30, and a communication unit 70. The recognition unit 22, the estimation unit 24, and the control unit 30 (notification control unit) each include, for example, a hardware processor such as a CPU (Central Processing Unit) and a storage device (a storage device including a non-transitory storage medium) that stores a program (software). The processor executes the program to realize the functions of each component. Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or the functions of each component may be realized by a combination of software and hardware. Some or all of these components may be realized by a dedicated LSI.

[0032] The program (software) may be stored in advance in a storage device (not shown) (storage device with a non-transitory storage medium) provided in the driving assistance device 20, such as a read-only memory (ROM), a random access memory (RAM), or a flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a memory card, and installed in the storage device by attaching the storage medium to the driving monitoring device 1. The program (software) may be downloaded in advance from another computer device via short-range communication or wide-area communication by an application running on the terminal device T, and transmitted from the terminal device T to be installed in the storage device.

[0033] The recognition unit 22 recognizes the lane in which the vehicle M is traveling, landmarks (such as the other vehicles and road dividing lines described above), and the like, based on the forward image. The other vehicles include other vehicles traveling in the same or adjacent lane, vehicles traveling in adjacent lanes and behind the vehicle M, and oncoming vehicles traveling in the oncoming lane. The recognition unit 22 may recognize landmarks by, for example, deep learning or other processing. The recognition unit 22 recognizes the state (speed in the forward direction, speed in the lateral direction, yaw rate, etc.) of the vehicle M (the driving assistance device 20) and the position of the vehicle M based on changes in the forward image or the rear image (for example, an area occupied by a landmark of interest in the image). When the recognition unit 22 is able to communicate with the vehicle M, the recognition unit 22 may identify or supplement the position or state of the vehicle M by an inertial navigation system (INS) that uses outputs from vehicle sensors (not shown) of the vehicle M.

[0034] The estimation unit 24 estimates the facial direction and gaze direction of the driver captured in the rearward image. Information representing the estimated gaze direction of the driver may be hereinafter referred to as "gaze information."

[0035] The control unit 30 includes a departure determination unit 40, an intention determination unit 50, and an alarm control unit 60. Fig. 4 is a diagram for explaining an overview of the alarm processing performed by the control unit 30. The departure determination unit 40 determines whether the vehicle M is about to deviate from the driving lane based on the state of the vehicle M and the position of the lane dividing line in the driving lane in which the vehicle M is traveling.

[0036] The intention determination unit 50 determines whether the vehicle M is about to deviate from the driving lane due to the driver's intention, based on the gaze information obtained from the estimation unit 24. The warning control unit 60 causes the terminal device T to output a warning, based on the judgment result of the departure determination unit 40 (e.g., a departure flag) and the judgment result of the intention determination unit 50 (e.g., an intention flag). For example, the warning control unit 60 transmits warning information to the terminal device T via the communication unit 70. The terminal device T outputs a warning based on information indicating the type of warning, such as the strength of the warning, included in the warning information. The communication unit 70 is a communication interface for performing wireless or wired communication with the terminal device T.

[0037] The processing of the departure determination unit 40, the intention determination unit 50, and the warning control unit 60 will be described below.

[0038] [Details of the deviation detection process] 5 is a diagram showing an example of functional units included in deviation determination unit 40. Deviation determination unit 40 includes, for example, a time processing unit 40A, a distance processing unit 40B, an offset processing unit 40C, an SMA (Simple Moving Average) processing unit 40D, and an integration processing unit 42. Deviation determination unit 40 acquires information indicating the positions of road dividing lines and the state of vehicle M from recognition unit 22. Deviation determination unit 40 repeatedly executes the various processes described below at predetermined intervals.

[0039] If all or some of the following conditions are not met, the departure determination unit 40 may not execute the following various processes and may not determine a departure. The conditions may be, for example, that the driver has performed an operation intending to stop the process, that road dividing lines cannot be detected, or that the detection accuracy of road dividing lines is equal to or less than a threshold. The condition may also be that the curvature of the travel lane in which the vehicle is traveling is equal to or greater than a threshold (that the curve is sharp). Information on the curvature of the travel lane is information provided by, for example, the terminal device T or a navigation device of the vehicle M. In this way, if the conditions for the departure determination unit 40 to execute the processes are not met, the processing load is reduced and the output of an alarm to the terminal device T is suppressed.

[0040] (Time processing section) The time processing unit 40A derives the time TLC until the vehicle M crosses the lane marking. The time TLC is, for example, a value obtained by dividing the distance from the reference position of the vehicle M to the lane marking by the lateral speed of the vehicle M. If the time TLC until the vehicle M crosses the lane marking is equal to or greater than a threshold, the time processing unit 40A outputs a red label flag (e.g., a first indicator), and if the time TLC is less than the threshold, the time processing unit 40A outputs a green label flag (e.g., a second indicator).

[0041] The time processing unit 40A may calculate the time TLC based on, for example, equations (1) and (2). "φ" is the yaw direction, which is the direction of the vehicle M relative to the direction in which the driving lane extends. "Vy" is the lateral speed of the vehicle M, and "V" is the speed obtained by combining the longitudinal speed and lateral speed of the host vehicle. "y l " is the process up to the left lane, and "y r " is the distance to the right lane. "D" is the width of the vehicle. "l f " is the distance from the center of gravity of the vehicle M (the position where the driving assistance device 20 is installed) to the axle of the front wheels. f " may be set to "zero".

[0042] If "φ>0", it indicates that the vehicle M is heading towards the left lane, and if "φ<0", it indicates that the vehicle M is heading towards the right lane. If "φ=0", the speed is zero and the time TLC is regarded as a "predetermined value (e.g., 1000)".

[0043]

number

[0044]

number

[0045] (Distance processing section) The distance processing unit 40B calculates the distance DLC from the vehicle to the road dividing line. The distance DLC is the distance from the center of the driving lane to the road dividing line minus half the width of the vehicle M. If the distance DLC is equal to or greater than a threshold, the distance processing unit 40B outputs a red label flag, and if the distance DLC is less than the threshold, it outputs a green label flag.

[0046] (Offset processing unit) The offset processing unit 40C calculates an offset index EPS. The offset index EPS is an index that increases as the vehicle M moves away from the center of the driving lane. The offset index EPS is an index that corresponds to the distance from the side of the vehicle M to a road dividing line. The offset processing unit 40C compares the offset index EPS with a threshold value, and outputs a red label flag, an orange label flag (e.g., a third index), or a green label flag based on the comparison result.

[0047] The offset processing unit 40C performs processing using an offset index EPS_L and an offset index EPS_R as shown in FIG. 6. The offset index EPS_L is the distance from the left side of the vehicle M (for example, the longitudinal center of the vehicle M or the position where the driving assistance device 20 is provided) to the left road dividing line, and the offset index EPS_R is the distance from the right side of the vehicle M to the right road dividing line. If the offset index EPS_L is greater than the distance yl or if the offset index EPS_R is greater than the distance yr, the offset processing unit 40C performs processing using a first threshold value and a second threshold value. The distance yl is the distance from the center of the driving lane to the left road dividing line, and the distance yr is the distance from the center of the driving lane to the right road dividing line. The first threshold value is, for example, a value obtained by adding a predetermined value to the vehicle width D. The second threshold value is, for example, a value obtained by adding a predetermined value greater than the predetermined value to the vehicle width D. The first threshold is, for example, greater than the distance yl and the distance yr, and the second threshold is greater than the first threshold.

[0048] For example, assume that the vehicle M is close to the lane dividing line on the right and the offset index EPS_L is greater than the distance yl. In this case, the offset processing unit 40C outputs a red label flag if the offset index EPS_L is equal to or greater than the second threshold, outputs an orange label flag if the offset index EPS_L is less than the second threshold and equal to or greater than the first threshold, and outputs a green label flag if the offset index EPS_L is less than the first threshold.

[0049] When the offset index EPS_L is equal to or less than the distance yl, the first threshold and the second threshold are not applied, and the offset index EPS_L is compared with a preset threshold that is provided separately from the first and second thresholds.

[0050] (SMA processing section) The SMA processing unit 40D calculates a noise area by statistically processing the past driving positions of the vehicle M. The noise area indicates the range in which the vehicle M deviates from the center of the driving lane while driving, and is an area that depends on the driving of the driver. The SMA processing unit 40D outputs a green label flag when the vehicle M is driving in the noise area, and outputs a red label flag when the vehicle M is driving outside the noise area.

[0051] The noise area AR_N is calculated based on the positions of multiple past sampling timings, as shown in Fig. 7. The noise area is calculated based on the average value, median value, or standard deviation of the deviation from the center of the driving lane. For example, a range of a predetermined σ from the center of the driving lane may be determined as the noise area.

[0052] (Integrated Processing Unit) 8 is a diagram for explaining the processing of the integration processing unit 42. The integration processing unit 42 acquires the processing result sequences of the time processing unit 40A, the distance processing unit 40B, the offset processing unit 40C, and the SMA processing unit 40D. The processing result sequences are the processing results of a predetermined processing cycle (one or more processing cycles). The integration processing unit 42 performs statistical processing for each processing result sequence and derives an index for each statistical processing result. The statistical processing includes calculating an average value, a median value, etc.

[0053] Next, the integration processing unit 42 performs weighted averaging using the indexes. The integration processing unit 42 multiplies each index by a coefficient corresponding to the index to perform weighted averaging, thereby deriving a deviation index. Note that other statistical processing may be applied instead of weighted averaging.

[0054] Next, the integration processing unit 42 determines whether the departure index exceeds the threshold value TH. If the departure index exceeds the threshold value TH, the integration processing unit 42 outputs the departure flag "TRUE" and the departure flag "TRUE(RIGHT)" or "TRUE(LEFT)", which is a flag indicating the direction in which the vehicle M is deviating.

[0055] If the deviation index is less than or equal to the threshold value TH, the integrated processing unit 42 outputs the deviation flag "FALSE" and the deviation flags "FALSE(RIGHT)" and "FALSE(LEFT)", which are flags indicating that the vehicle M has not deviated to the right or left.

[0056] As described above, the deviation determination unit 40 can accurately determine whether or not the vehicle M is about to deviate from a road dividing line.

[0057] [Details of the intention determination process (Pattern 1)] (Flowchart (Part 1) The intention determination unit 50 determines whether the driver's face or line of sight is facing the direction of a predetermined object. If the face or line of sight is facing the direction of the predetermined object, it is estimated that the driver is looking in the direction of the predetermined object. The predetermined object may be a side mirror, a driver's side window, or a passenger's side window. The predetermined object may be any object that is present in the line of sight when the driver is gazing sideways or toward an adjacent lane.

[0058] Fig. 9 is a flowchart showing an example of the flow of processing executed by the intention determination unit 50. The processing of steps S100-S104 and the processing of steps S106-S110 in Fig. 9 may be executed in parallel, or one may be executed first and then the other.

[0059] First, the intention determination unit 50 acquires a gaze vector included in the gaze information and determines whether the gaze vector is valid (step S100). The gaze vector is, for example, a vector defined in a two-dimensional coordinate system (e.g., a coordinate system in which a pitch direction and a yaw direction are defined) or a three-dimensional coordinate system. For example, the intention determination unit 50 determines whether the gaze vector is valid by determining whether the gaze vector conforms to a predetermined criterion (e.g., a criterion based on the previous gaze vector).

[0060] Next, the intention determination unit 50 estimates the direction in which the driver's line of sight is directed based on the pitch direction and yaw direction of the line of sight vector (step S102). Next, the intention determination unit 50 determines whether the driver's line of sight is directed toward a predetermined object (step S104). If the determination in step S100 or step S104 is negative, the processing of one routine of this flowchart ends, and if the determination in step S100 or step S104 is positive, the processing proceeds to step S112.

[0061] Next, the intention determination unit 50 acquires a face vector included in the gaze information and determines whether the face vector is valid (step S106). The face vector is, for example, a vector defined in a two-dimensional coordinate system (e.g., a coordinate system in which a pitch direction and a yaw direction are defined) or a three-dimensional coordinate system. For example, the intention determination unit 50 determines whether the gaze vector is valid by determining whether the face vector conforms to a predetermined criterion (e.g., a criterion based on the previous face vector).

[0062] Next, the intention determination unit 50 estimates the direction in which the driver's face is facing based on the pitch direction and yaw direction of the face vector (step S108). Next, the intention determination unit 50 determines whether the driver's face is facing a predetermined object direction (step S110). If the determination in step S106 or step S110 is negative, the processing of one routine of this flowchart ends, and if the determination in step S106 or step S110 is positive, the processing proceeds to step S112.

[0063] If at least one of the driver's line of sight or face is directed toward a predetermined object, the intention determination unit 50 updates the checklist (step S112). This ends one routine of this flowchart. In the above process, the checklist may be updated using only one of the line of sight direction or the face direction, or the checklist may be updated when the line of sight direction and the face direction are consistent (if both are directed in the same direction, it may be determined that that direction is being focused on).

[0064] 10 is a diagram showing an example of a checklist. The intention determination unit 50 generates information that associates time with the direction in which the driver is gazing. For example, when the driver's line of sight or face is facing the direction of an object on the right, the information that the driver is gazing to the right is associated with time, and when the driver's line of sight or face is facing the direction of an object on the left, the information that the driver is gazing to the left is associated with time.

[0065] Flowchart (part 2) FIG. 11 is a flowchart showing an example of the flow of processing executed by the intention determination unit 50. First, the intention determination unit 50 determines whether or not the departure flag "TRUE" has been acquired (step S200). The departure flag "TRUE" is output when the departure determination unit 40 determines that the vehicle M will deviate from the lane (when the vehicle is likely to deviate, or the possibility of departure is equal to or greater than a threshold). If it is determined that the vehicle M will deviate to the right of the driving lane, the departure flag "TRUE (RIGHT)" is output, and if it is determined that the vehicle M will deviate to the left of the driving lane, the departure flag "TRUE (LEFT)" is output.

[0066] Next, the intention determination unit 50 refers to the checklist (step S202). Next, when the intention determination unit 50 acquires the deviation flag "TRUE (RIGHT)", it determines whether the gaze direction of the line of sight on the checklist is to the right or not, and if the gaze direction is to the right, it generates the intention flag (DM) "TRUE (RIGHT)" (steps S204, S208).

[0067] When the deviation flag "TRUE (LEFT)" is acquired, the intention determination unit 50 determines whether the gaze direction of the checklist is to the left or not, and if the gaze direction is to the left, generates the intention flag (DM) "TRUE (LEFT)" (steps S206, S210).

[0068] If the determinations in steps S204 and S206 are negative, the intention determination unit 50 generates an intention flag (DM) "FALSE" (step 210). As described above, if the departure direction indicated by the departure flag matches the driver's gaze direction, the intention flag (DM) "TRUE" is generated.

[0069] [Details of the intention determination process (Pattern 2)] The intention determination unit 50 may generate an intention flag "TRUE" or "FALSE" based on the processing result of the SMA processing unit 40D. Figure 12 is a flowchart showing another example of the flow of processing executed by the intention determination unit 50.

[0070] First, the intention determination unit 50 acquires a departure flag and determines whether the acquired departure flag is "TRUE" (step S300). Next, the intention determination unit 50 acquires SMA information (step S300). The SMA information is information indicating the sway of the vehicle M output by the SMA processing unit 40D, and is information indicating the simple moving average value of the deviation from the center of the lane over a predetermined period. The SMA information is assigned a label such as red, orange, or green depending on the degree of sway. Note that if the SMA processing unit 40D is configured to output a red label or a green label but not an orange label, the orange label may be omitted.

[0071] The intention determination unit 50 determines whether the label of the SMA information is red or orange (step S304). In other words, the intention determination unit 50 determines whether the simple moving average value is equal to or greater than a threshold value.

[0072] If the label of the SMA information is red or orange, the intention determination unit 50 counts up (plus 1) the counter (step S308), and if the label of the SMA information is not red or orange, the intention determination unit 50 counts down (minus 1) the counter (step S310).

[0073] The intention determination unit 50 determines whether the counter value is equal to or greater than a threshold value (step S312). If the counter value is equal to or greater than the threshold value, the intention determination unit 50 generates an intention (K) flag "TRUE" (step S314), and if the counter value is less than the threshold value, the intention determination unit 50 generates an intention (K) flag "FALSE" (step S316).

[0074] As shown in 13, the warning control unit 60, which will be described later, may regard the intention flag as "TRUE" in any of states A, B, and C. State A is when the intention flag (DM) is "TRUE." State B is when the intention flag (K) is "TRUE." State C is when both the intention flag (DM) is "TRUE" and the intention flag (K) is "TRUE."

[0075] [Details of the alarm control unit processing] When it is predicted that vehicle M will deviate from the lane it is traveling in based on information obtained from one or more images (e.g., a forward image and a rearward image), the warning control unit 60 determines the notification intensity for the driver of vehicle M based on the amount of change in an object in one or more images, and causes the terminal device T to notify the driver of vehicle M of the determined notification intensity. The object may be, for example, a lane marking on the lane in which vehicle M is traveling and the driver of vehicle M. The change in the object may be a change in the position of vehicle M relative to the lane marking and a change in the driver's behavior. For example, the warning control unit 60 determines the notification intensity based on the degree to which vehicle M is approaching the lane marking and behavior suggesting that the driver is approaching the lane marking (e.g., line of sight, face direction, steering wheel operation mode). For example, the warning control unit 60 increases the notification intensity as vehicle M approaches the lane marking, and decreases the notification intensity if the driver intends to approach the lane marking.

[0076] The warning control unit 60 may determine the intensity of the notification based on the above-mentioned deviation index. For example, the larger the deviation index (the more likely it is to be a deviation index), the stronger the notification will be. Instead of the deviation index, the warning control unit 60 may determine the intensity of the notification based on one or more indices (deviation determination indices) from among an index corresponding to the processing result of the time processing unit 40A, an index corresponding to the processing result of the distance processing unit 40B, an index corresponding to the processing result of the offset processing unit 40C, or an index corresponding to the processing result of the SMA processing unit 40D, or an index obtained by statistically processing these indices.

[0077] Furthermore, the warning control unit 60 may determine the intensity of the notification by referring to the determination result of the intention determination unit 50, in addition to (or instead of) the departure determination index. For example, when the direction of the driver's face and / or line of sight based on the rear image is outside a preset range (for example, when not facing the direction of a predetermined object), the warning control unit 60 causes the terminal device T to issue a notification with a stronger notification intensity than when the direction of the driver's face or line of sight is not outside the preset range (for example, when facing the direction of a predetermined object).

[0078] More specifically, the warning control unit 60 causes the terminal device T to issue a warning notification of a predetermined intensity (a notification of intensity 2 or 3 described later) when (a1) the direction of the driver's face and / or line of sight based on the rear image is in the traveling direction of the vehicle M and not facing left for a predetermined period of time or more, and further (a2) it is determined based on the front image or the rear image that the vehicle M is swaying leftward (high possibility of departure). If one or both of (a1) and (a2) are not satisfied, the warning control unit 60 causes the terminal device T to issue a warning notification of a weaker intensity than the predetermined notification (a notification of intensity 1 described later), or does not cause the terminal device T to issue a warning notification.

[0079] The warning control unit 60 causes the terminal device T to issue a notification of a predetermined intensity (a notification of intensity 2 or 3 described later) when (b1) the direction of the driver's face and / or line of sight based on the rear image is in the traveling direction of the vehicle M and not facing to the right for a predetermined period of time or more, and further determines (b2) that the vehicle M is swaying to the right based on the front image or the rear image. When one or both of (b1) and (b2) are not satisfied, the warning control unit 60 causes the terminal device T to issue a notification of a warning weaker than the notification of the predetermined intensity (a notification of intensity 1 described later), or does not cause the terminal device T to issue a warning notification.

[0080] Furthermore, the presence or absence of a vehicle in a lane adjacent to the driving lane behind the vehicle M may also be taken into consideration. When the warning control unit 60 determines that another vehicle is present in the blind spot based on a forward image or a rearward image in which the blind spot (an adjacent lane adjacent to the driving lane and an adjacent rear area behind the vehicle M) is captured, the warning control unit 60 may cause the terminal device T to issue a warning notification (a notification of intensity 3, described later) that is stronger than a predetermined warning intensity when it is determined that no other vehicle is present in the blind spot.

[0081] The intensity of the notification may be determined as shown in the flowchart below: Figure 14 is a flowchart showing one routine of an example of the flow of processing executed by the warning control unit 60.

[0082] First, the warning control unit 60 determines whether the deviation flag is "TRUE" (step S400). If the deviation flag is not "TRUE", the warning control unit 60 does not output a warning to the terminal device T (step S402). The strength of the warning at this time is "zero (0)".

[0083] If the departure flag is "TRUE", the warning control unit 60 determines whether the intention flag is "TRUE" or not (step S404). If the intention flag is "TRUE", the warning control unit 60 outputs a warning of intensity "1" to the terminal device T (step S406). Note that if the departure direction and the direction the driver is gazing match, the process of step S406 is performed, and if they do not match, the process proceeds to step S408.

[0084] If the intention flag is not "TRUE", the warning control unit 60 determines whether the relationship between the departure flag and the other vehicle in the blind spot satisfies a predetermined condition (step S408). If the predetermined condition is not satisfied, the warning control unit 60 causes the terminal device T to output a warning of intensity "2" (step S410). If the predetermined condition is satisfied, the warning control unit 60 causes the terminal device T to output a warning of intensity "3" (step S412). The strengths increase in the order of "3", "2", and "1".

[0085] The predetermined condition is that the type of departure flag matches the position of another vehicle (or object) in the blind spot. For example, the predetermined condition is that the departure flag is "TRUE (RIGHT)" and another vehicle is present in the blind spot on the right. For example, the predetermined condition is that the departure flag is "TRUE (LEFT)" and another vehicle is present in the blind spot on the left. Note that the determination in step S408 may be made without considering the above-mentioned match. In this case, if another vehicle is present in either the left or right blind spot, the condition related to the blind spot is met.

[0086] As described above, the warning control unit 60 issues a warning of intensity 3 when (a1, b1) the driver has no intention of departing from the driving lane, (a2, b2) vehicle M is about to depart from the driving lane, and (a3, b3) there is another vehicle in the blind spot in the departure direction. The warning control unit 60 issues a warning of intensity 2 when (a1, b1) the driver has no intention of departing from the driving lane, (a2, b2) vehicle M is about to depart from the driving lane, and (a3, b3) there is no other vehicle in the blind spot in the departure direction.

[0087] An alert of intensity "1" output in the event of an intended departure is, for example, an alert that causes the HMI to output an image (or image and sound, etc.) indicating that a lane change is occurring. An alert of intensity "2" output in the event of an unintended departure is, for example, an image in which an icon flashes within the image and a low-frequency warning sound (for example, a warning sound that resembles the sound that would be produced if a vehicle were traveling on an undulating road). An alert of intensity "3" output in the event of an unintended departure and an object in the blind spot is, for example, an image in which an icon flashes within the image and a sound (for example, a high-amplitude and high-frequency warning sound) indicating a high risk.

[0088] In the above example, the intention flag is generated based on (A) the direction of the line of sight or (B) the direction of the face. However, instead of (or in addition to) this, (C) the behavior of the driver's arms or hands may also be taken into account. In this case, the rear camera 12 captures an image of the driver's arms and / or hands. For example, the intention determination unit 50 may determine that there is an intention to depart if changes in the driver's arms or hands over a predetermined period indicate a predetermined behavior. The predetermined behavior is the driver operating the steering wheel to move the vehicle M in the direction of departure. For example, in the process of step S404 in FIG. 14, the intention flag generated based on changes in the driver's arms or hands over a predetermined period may be applied to the determination. For example, the intention determination unit 50 may issue a warning of level 1 if all of the above (A), (B), and (C) indicate an intention to depart, or may issue a warning of level 1 if a predetermined number of factors indicate an intention to depart. If all of the above (A), (B), and (C) indicate no intention to depart, or if a predetermined number of the elements indicate no intention to depart, the intention determination unit 50 may proceed to the processing of step S408 in Fig. 14. Furthermore, the warning control unit 60 may determine the strength of the warning according to the number or combination of elements among the above (A), (B), and (C) that indicate an intention to depart.

[0089] In the above example, an example in which the driving assistance device 20 is applied to the driving monitoring device 1 has been described, but some or all of the functional configuration of the driving assistance device 20 may be mounted on a vehicle. Furthermore, the functions realized by the driving assistance device 20 may be realized by executing an application program installed in the terminal device T. In this case, the terminal device T is used by being detachably attached to the vehicle M, for example. For example, a holder for the terminal device T having a detachable portion is provided on either or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder.

[0090] If the driving assistance device 20 can use information acquired by the vehicle M, it can refer to this information to estimate the departure indicators of the vehicle M and the driver's intention to deviate. The information acquired by the vehicle M includes, for example, the state of the turn signals, the steering behavior (steering torque), and vehicle sensors (speed, acceleration, yaw rate). Because the driving assistance device 20 is not connected to an in-vehicle network, it may be unable to refer to the information acquired by the vehicle M, or it may be difficult to link with the in-vehicle network. In contrast, in this embodiment, the driving assistance device 20 estimates the departure indicators and the driver's intention to deviate using images captured by the front camera 10 and the rear camera 12, without using the information acquired by the vehicle M, and determines the intensity of the notification. In this way, the driving assistance device 20 can easily issue an alert to assist the driver in driving without the need for linking with an in-vehicle network. Furthermore, the driving assistance device 20 can suppress unnecessary notifications by taking the driver's intention into consideration.

[0091] According to the embodiment described above, when the driving assistance device 20 predicts that the vehicle M will deviate from the lane it is traveling in based on information obtained from an image, it determines the notification intensity of the notification to the driver of the vehicle M based on changes in the object in the image, and has the notification device notify the driver of the determined notification intensity, thereby easily and accurately outputting an alarm.

[0092] The above-described embodiment can be expressed as follows. a storage device storing a program included in a device that is detachable from the vehicle via a detachable unit; a hardware processor included in the device; The hardware processor executes the program stored in the storage device, acquiring one or more images of the surroundings of the vehicle; When it is predicted that the vehicle will deviate from the lane in which it is traveling based on information obtained from the one or more images, a notification intensity for a notification to a driver of the vehicle is determined based on a change in an object in the one or more images, and a notification unit is caused to issue a notification of the determined notification intensity. The control device is configured as follows.

[0093] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0094] 1 Operation monitoring device 10. Front camera 12 Rear camera 20 Driving assistance devices 22 Recognition part 24 Estimation part 30 Control Unit 40 Deviation Judgment Unit 50 Intention determination section 60 Alarm control unit 70 Communications Department T terminal device

Claims

1. A driving assistance device that is detachable from a vehicle via a detachable unit, an acquisition unit that acquires one or more images of the surroundings of the vehicle; a notification control unit that, when it is predicted that the vehicle will deviate from the driving lane in which it is traveling based on information obtained from the one or more images, determines a notification intensity for a notification to a driver of the vehicle based on a change in an object in the one or more images, and causes a notification unit to issue a notification of the determined notification intensity; the object is a road dividing line of a lane in which the vehicle is traveling and a driver of the vehicle; the changes in the object are changes in the position of the vehicle relative to the road dividing line and changes in the driver's behavior; the one or more images include an image of a face of a driver who is driving the vehicle; The notification control unit (a1) if it is determined that the direction of the driver's face or line of sight based on the one or more images is in the traveling direction of the vehicle and not facing left for a predetermined period of time or longer, and further (a2) it is determined that the vehicle is swaying left based on the one or more images, the notification unit is caused to issue a notification of a warning of a predetermined strength; and if either or both of (a1) and (a2) are not satisfied, the notification unit is caused to issue a notification of a warning of a weaker strength than the notification of the predetermined strength, or the notification unit is not caused to issue a notification of a warning. (b1) if the direction of the driver's face or line of sight based on the one or more images is in the direction of travel of the vehicle and not facing to the right for a predetermined period of time or longer, and (b2) if it is determined that the vehicle is swaying to the right based on the one or more images, issue a warning of a predetermined strength; and if either or both of (b1) and (b2) are not satisfied, have the notification unit issue a warning of a weaker strength than the warning of the predetermined strength, or do not issue a warning. Driving assistance device.

2. the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, The notification control unit (a3) when a vehicle is present in the adjacent rear area on the left side of the vehicle based on the one or more images, causing the notification unit to issue a third warning notification having a strength stronger than the warning notification of the predetermined strength; If the conditions (a1) and (a2) are satisfied but the condition (a3) ​​is not satisfied, a second warning is issued that is stronger than the predetermined warning strength and weaker than the third warning strength, (b3) when a vehicle is present in the adjacent rear area on the right side of the vehicle based on the one or more images, causing the notification unit to issue a third warning notification having a stronger intensity than the warning notification of the predetermined intensity; when the conditions (b1) and (b2) are satisfied but the condition (b3) is not satisfied, the notification unit is caused to notify the second alarm. The driving assistance device according to claim 1 .

3. A driving assistance device that is detachable from a vehicle via a detachable unit, an acquisition unit that acquires one or more images of the surroundings of the vehicle; a notification control unit that, when it is predicted that the vehicle will deviate from the driving lane in which it is traveling based on information obtained from the one or more images, determines a notification intensity for a notification to a driver of the vehicle based on a change in an object in the one or more images, and causes a notification unit to issue a notification of the determined notification intensity; the object is a road dividing line of a lane in which the vehicle is traveling and a driver of the vehicle; the changes in the object are changes in the position of the vehicle relative to the road dividing line and changes in the driver's behavior; the one or more images include an image of one or both of the driver's arms and hands; The notification control unit (c1) if it is determined based on the one or more images that the hand or arm is not performing an operation to control steering to move the vehicle to the left, and (c2) if it is determined based on the one or more images that the vehicle is swaying to the left, it issues a warning of a predetermined strength, and if one or both of (c1) and (c2) are not satisfied, it causes the notification unit to issue a warning of a strength weaker than the predetermined warning strength, or does not cause the notification unit to issue a warning. (d1) if it is determined based on the one or more images that the hand or arm is not performing an operation to control steering to move the vehicle to the right, and (d2) if it is determined based on the one or more images that the vehicle is swaying to the right, it issues a warning of a predetermined strength, and if one or both of (d1) and (d2) are not satisfied, it causes the notification unit to issue a warning of a weaker strength than the predetermined strength, or does not cause the notification unit to issue a warning. Driving assistance device.

4. the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, The notification control unit (c3) when a vehicle is present in the adjacent rear area on the left side of the vehicle based on the one or more images, causing the notification unit to issue a third warning notification having a strength stronger than the warning notification of the predetermined strength; and If the conditions (c1) and (c2) are satisfied but the condition (c3) is not satisfied, a second warning is issued that is stronger than the warning of the predetermined strength and weaker than the third warning, (d3) when the conditions (d1) and (d2) are satisfied and a vehicle is present in the adjacent rear area on the right side of the vehicle based on the one or more images, causing the notification unit to issue a third warning notification having a stronger intensity than the warning notification of the predetermined intensity; when the conditions (d1) and (d2) are satisfied but the condition (d3) is not satisfied, the notification unit is caused to notify the second alarm. The driving assistance device according to claim 3 .

5. the one or more images include an image of a face of a driver who is driving the vehicle; When the direction of the driver's face or line of sight based on the one or more images is outside a predetermined range, the notification control unit causes the notification unit to issue a notification with a stronger notification intensity than when the direction of the driver's face or line of sight is not outside the predetermined range. The driving assistance device according to any one of claims 1 to 4.

6. the one or more images include an image of an adjacent rear area behind the vehicle in an adjacent lane adjacent to the driving lane, When it is determined that a vehicle exists in the adjacent rear area based on the one or more images, the notification control unit causes the notification unit to issue a notification of an alarm with a stronger intensity than the predetermined intensity when it is determined that no vehicle exists in the adjacent rear area. The driving assistance device according to any one of claims 1 to 4.

7. A driving assistance unit that is detachable from a vehicle via a detachable unit, a first camera that is not connected to an in-vehicle network of the vehicle and that captures an image including a surrounding situation in front of the vehicle; a second camera not connected to the vehicle's in-vehicle network, the second camera capturing an image including a driver of the vehicle, a driver's seat side window of the vehicle, a passenger's seat side window of the vehicle, and an area behind the vehicle; an acquisition unit that acquires images acquired by the first camera and the second camera; a notification control unit that, when it is predicted that the vehicle will deviate from the lane in which it is traveling based on information obtained from the image acquired by the acquisition unit, determines a notification intensity for a notification to a driver of the vehicle based on a change in an object in the image, and causes a notification unit to issue a notification of the determined notification intensity; When it is determined that another vehicle is present in the blind spot of the vehicle for the driver based on the image acquired by the acquisition unit, the notification control unit causes the notification unit to issue a notification with a stronger notification intensity than when it is not determined that another vehicle is present in the blind spot. Driver assistance unit.

Images