Driving assistance device, road-to-vehicle driving assistance system, driving assistance method, and driving assistance program

Abstract

A driving assistance device comprising: a gaze information calculation unit (30) that calculates gaze information for a subject from a vehicle interior video of inside a vehicle, said vehicle interior video having been imaged by a imaging device that is provided with at least one camera; an object information calculation unit (40) that detects an object present in the surroundings of the vehicle from a vehicle exterior video of outside of the vehicle, said vehicle exterior video having been imaged by the imaging device, and extracts object information, which is information about the detected object; a visual recognition level calculation unit (50) that calculates a visual recognition level from the calculated gaze information and the extracted object information, said visual recognition level expressing the level to which the subject visually recognizes the detected object; a collision possibility calculation unit (60) that calculates a collision possibility from the extracted object information, said collision possibility indicating the degree of possibility that the vehicle will collide with the detected object; and an effective collision possibility calculation unit (70) that calculates an effective collision possibility from the calculated visual recognition level and the calculated collision possibility, said effective collision possibility decreasing in accordance with an increase in the calculated visual recognition level and increasing in accordance with an increase in the calculated collision possibility.

Description

[Technical field]

[0001] The present disclosure relates to driving assistance technology. [Background technology]

[0002] Patent Document 1 discloses a driving assistance device including: a visibility calculation unit that calculates a first visibility, which is the driver's degree of visibility of each of the one or more objects based on the direction of the driver's line of sight and information on one or more objects existing around the vehicle; a risk calculation unit that calculates a risk indicating the possibility of contact of the vehicle with each of the one or more objects based on the state of each of the one or more objects and the state of the vehicle; and a display control unit that, when the first visibility of a first object among the one or more objects is less than a first threshold, causes a display unit to display first warning information associated with the first object, and when the first visibility is equal to or greater than the first threshold, causes a display unit to hide the first warning information; and, when the first warning information is not displayed on the display unit and the risk level of the first object is equal to or greater than a second threshold, causes the display control unit to display second warning information for the first object on the display unit in association with the first object.

[0003] Thus, according to the driving assistance device of Patent Document 1, the display control unit determines whether the first visibility of the first object is less than the first threshold value and controls the display of first warning information, and then controls the display to display second warning information when the danger level of the first object is equal to or greater than the second threshold value when the first warning information is not displayed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-166542 A Summary of the Invention [Problem to be solved by the invention]

[0005] A configuration such as the driving assistance device of Patent Document 1, which additionally evaluates the risk level on the premise that an evaluation of visibility has been performed, has the problem that it is not possible to simultaneously evaluate visibility and risk level (likelihood of collision) and issue a warning.

[0006] The present disclosure has been made to solve such problems, and aims to provide a driving assistance technology that can simultaneously evaluate visibility and collision possibility and issue a warning. [Means for solving the problem]

[0007] One aspect of a driving assistance device according to an embodiment of the present disclosure includes a gaze information calculation unit that calculates gaze information of a subject from an in-vehicle image of the inside of the vehicle captured by an imaging device having at least one camera; an object information calculation unit that detects objects present around the vehicle from an exterior image of the outside of the vehicle captured by the imaging device and extracts object information that is information about the detected object; a visibility calculation unit that calculates a visibility indicating the degree to which the subject can see the detected object from the calculated gaze information and the extracted object information; a collision likelihood calculation unit that calculates a collision likelihood indicating the likelihood of a collision between the detected object and the vehicle from the extracted object information; and an effective collision likelihood calculation unit that calculates an effective collision likelihood from the calculated visibility and the calculated collision likelihood, which decreases as the calculated visibility increases and increases as the calculated collision likelihood increases. Effect of the Invention

[0008] According to one aspect of the driving assistance device according to an embodiment of the present disclosure, it is possible to simultaneously evaluate the visibility and the risk level and issue a warning. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a driving assistance device and a driving assistance system according to a first embodiment of the present disclosure. [Diagram 2]4 is a flowchart illustrating an operation for performing driving assistance according to the first embodiment of the present disclosure. [Diagram 3] FIG. 11 is a block diagram illustrating a driving assistance device and a driving assistance system according to a second embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram showing a specific example for explaining the concept of effective collision likelihood related to the driving assistance device and the driving assistance system according to the first and second embodiments of the present disclosure. [Diagram 5] FIG. 11 is a diagram illustrating a configuration example of a roadside system including a roadside device according to a third embodiment of the present disclosure. [Figure 6A] FIG. 2 is a diagram illustrating an example of a hardware configuration of a driving assistance device and a roadside device according to first to third embodiments of the present disclosure. [Figure 6B] FIG. 2 is a diagram illustrating an example of a hardware configuration of a driving assistance device and a roadside device according to first to third embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that components with the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted.

[0011] Embodiment 1 <Configuration> A driving assistance device 1 and a driving assistance system SYS-V1 according to a first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the driving assistance device 1 and the driving assistance system SYS-V1 according to the first embodiment of the present disclosure. As shown in Fig. 1, the driving assistance system SYS-V1 includes an imaging device 10 that captures an outside-vehicle image including a visual target and an inside-vehicle image including a target person, and the driving assistance device 1.

[0012] As an example, the driving support device 1 includes a data input unit 20 that receives an image captured by an imaging device 10, classifies the image into an in-vehicle image and an outside-vehicle image required for calculating line-of-sight information and object information, and outputs the image; a line-of-sight information calculation unit 30 that calculates the line-of-sight information of a target from the in-vehicle image; an object information calculation unit 40 that detects an object around the vehicle from the outside-vehicle image and extracts information about the object; a visibility calculation unit 50 that calculates a visibility taking into account time-series attenuation from the line-of-sight information and the object information; a collision possibility calculation unit 60 that calculates a collision possibility of the object from the object information; an effective collision possibility calculation unit 70 that calculates an effective collision possibility from the visibility and the collision possibility; and a warning unit 80 that determines and presents a warning depending on the effective collision possibility. All of the functional units of the driving support device 1 may be realized by a device mounted on the vehicle, or only some of the functional units of the driving support device 1 may be realized by a device mounted on the vehicle, and the remaining functional units may be realized by a device provided remotely. Hereinafter, a vehicle on which the driving support device 1 is mounted is referred to as a device-mounted vehicle.

[0013] (Imaging device) The imaging device 10 includes at least one camera, captures an in-vehicle image A1 including a plurality of frame images including a face image of a subject in the vehicle, and an outside-vehicle image B1 including a plurality of frame images including an object to be viewed by the subject, and outputs video data including the in-vehicle image A1 and the outside-vehicle image B1. The in-vehicle image A1 and the outside-vehicle image B1 may be captured as separate images using a single camera corresponding to each image, or may be captured using a stereo camera corresponding to each image in order to measure an accurate three-dimensional position. In addition, the in-vehicle image A1 and the outside-vehicle image B1 may be captured together in the same image using a camera that enables wide-range capture such as a spherical camera. The imaging device 10 is connected to the data input unit 20 of the driving support device 1.

[0014] (Data Entry Section) The data input unit 20 classifies the video data including the in-vehicle video A1 and the outside-vehicle video B1 into the in-vehicle video A1 required for calculating the line-of-sight information and the outside-vehicle video B1 required for calculating the object information, and outputs the classified in-vehicle video A1 and outside-vehicle video B1. The data input unit 20 is connected to the line-of-sight information calculation unit 30 and the object information calculation unit 40, and outputs the in-vehicle video A1 to the line-of-sight information calculation unit 30 and outputs the outside-vehicle video B1 to the object information calculation unit 40.

[0015] (Gaze information calculation unit) The gaze information calculation unit 30 receives the vehicle interior image A1 as an input, calculates the gaze vector of the target person from the received vehicle interior image A1, and outputs the calculated gaze vector as gaze information C1. As a method for calculating the gaze vector, a model-based gaze detection method represented by the use of a corneal reflection image may be used, or an appearance-based method using machine learning may be used. The gaze information calculation unit 30 is connected to the visibility calculation unit 50.

[0016] (Object information calculation unit) The object information calculation unit 40 receives the outside-of-vehicle image B1 as an input, detects an object that needs to be seen for proper driving from the received outside-vehicle image B1, and outputs object information D1, which is information about the detected object. As the object information D1, type information indicating the type of the object may be used, such as "fixed indicator" if the object is a traffic light, or "moving object" if the object is a car or a pedestrian. If the object is a fixed indicator, the object information D1 may further include position information indicating the position of the object, size information indicating the detected size of the object, and status information represented by the lighting form of an LED (Light Emitting Diode). On the other hand, if the object is a moving object, the object information D1 may further include position information indicating the position of the object, size information indicating the detected size of the object, and time-series change information (movement information) indicating the traveling direction and speed of the object. The object information calculation unit 40 is connected to the visibility calculation unit 50 and the collision probability calculation unit 60, and outputs the object information D1 to both of them.

[0017] (Visibility calculation section) The visibility calculation unit 50 receives the gaze information C1 of the subject and the object information D1 as input, calculates the visibility E1 of each object using the received gaze information C1 and object information D1, and outputs the calculated visibility E1. The visibility E1 is an index indicating the degree to which the subject recognizes the detected object. The visibility calculation unit 50 calculates the visibility E1, for example, as follows.

[0018] First, the visibility calculation unit 50 calculates the degree of directional agreement between vectors using the line-of-sight unit vector of the line-of-sight vector at each time and the object unit vector of the object vector from the vehicle. As the degree of directional agreement, the angle between the line-of-sight unit vector and the object unit vector may be used, or the angle between these two unit vectors in the displacement direction per unit time may be used. The degree of directional agreement may be an index smoothed per certain time.

[0019] Next, the visibility calculation unit 50 calculates the sum of the directional agreement degrees by going back a certain time from the current time to the past and giving weights for each time. The weights may be set to be higher as the time approaches the current time, or nonlinear weights may be given. If a change in the state or movement of the object is observed within the visibility measurement time, a correction coefficient is defined for each type of object, the directional agreement degree before the time when the change in the object occurred is multiplied by the correction coefficient, and the visibility is calculated by taking the sum of the corrected directional agreement degrees within the measurement time. If the object is a fixed indicator such as a traffic light, the correction coefficient may be given as a constant multiple at the time when a change in the indication occurs. If the object is a moving object such as a car, the difference between the movement at the start of the visibility measurement and the movement at the current time may be calculated, and the correction coefficient may be dynamically defined according to the amount of the difference. The visibility calculation unit 50 is connected to the effective collision probability calculation unit 70.

[0020] (Collision possibility calculation unit) The collision possibility calculation unit 60 receives the object information D1 as an input, calculates the collision possibility F1 for each object using the received object information D1, and outputs the calculated collision possibility F1. The collision possibility F1 is an index indicating the degree of possibility that the detected object and the device-equipped vehicle will collide. When calculating the collision possibility F1, the calculation method may be changed according to the type of object indicated by the type information. For example, in the case of a fixed indicator such as a traffic light, the time limit from the object's position and motion information to the stop instruction position is measured, and the collision possibility F1 is defined as an index inversely proportional to the time limit. Depending on the difference in the indication form from proceeding to stopping, a correction coefficient may be multiplied by the time limit. Here, examples of the indication form of the object include the color display of a traffic light and information on the road by a road information board (traffic jam, traffic accident, weather, etc.). In addition, the motion information is, for example, information on the speed, acceleration, and traveling direction of the object. Even if the object is a fixed object such as a traffic light, it appears to be moving relatively when viewed from a camera attached to the vehicle, so motion information exists.

[0021] If the object is a moving object such as an automobile, the time limit until contact with the device-equipped vehicle is measured from the object's position and motion information, and the collision probability F1 is defined as an index inversely proportional to the time limit. Furthermore, if the object is a moving object, the kinetic energy may be calculated and the collision probability F1 may be defined as an index proportional to the kinetic energy.

[0022] In this way, the collision likelihood calculation unit 60 selects at least one of the pointing form or the motion information of the object according to the type of the object, and calculates the collision likelihood F1. The collision likelihood calculation unit 60 is connected to the effective collision likelihood calculation unit 70.

[0023] (Effective collision probability calculation section) The effective collision likelihood calculation unit 70 receives the visibility E1 and the collision likelihood F1 as inputs, calculates the effective collision likelihood G1 using the received visibility E1 and collision likelihood F1, and outputs the calculated effective collision likelihood G1. The effective collision likelihood G1 is an index that decreases as the visibility E1 increases and increases as the collision likelihood F1 increases. The visibility E1, which is such an index, is expressed, for example, as in Equation (1), where K is a constant. The effective collision likelihood calculation unit 70 is connected to the warning unit 80. G1 = K × F1 / (1 + E1) (1)

[0024] As shown in formula (1), the effective collision probability G1 decreases if the collision probability F1 remains constant and the visibility E1 increases. Conversely, the effective collision probability G1 increases if the visibility E1 remains constant and the collision probability F1 increases. Visibility E1 and collision probability F1 are included in a single formula, so they can be evaluated simultaneously.

[0025] According to a configuration in which the risk level is additionally evaluated on the premise that an evaluation of visibility has been performed as in Patent Document 1, there is a problem in that the visibility and the risk level (collision possibility) cannot be evaluated simultaneously to issue a warning. In contrast, according to the configuration of the first embodiment including the effective collision possibility calculation unit 70, the visibility E1 and the collision possibility F1 can be evaluated simultaneously as the effective collision possibility G1. By using such an effective collision possibility G1, a warning can be issued by simultaneously evaluating the visibility E1 and the collision possibility F1.

[0026] By performing a warning that evaluates the visibility E1 and the collision possibility F1 simultaneously, it is possible to eliminate the annoyance to the user that may be caused by the configuration of Patent Document 1. That is, according to the configuration of Patent Document 1, in which the display control of the first warning information is performed by determining whether the first visibility of the first object is less than the first threshold value or not, and then the display control of the second warning information is performed when the danger level of the first object is equal to or greater than the second threshold value while the first warning information is not displayed, the determination of whether to display the warning information (first warning information) is performed based on whether the visibility (first visibility) is equal to or greater than the threshold value (first threshold value) when the danger level of the object (first object) is less than the threshold value (second threshold value), so that when the visibility of an object with a low danger level (i.e., an object with a danger level less than the second threshold value) is low (i.e., when the visibility is less than the first threshold value), a warning display is performed, causing an annoyance to the user of the device. In contrast, according to the configuration of the first embodiment including the effective collision likelihood calculation unit 70, the relationship between the visibility E1 and the collision likelihood F1 is simultaneously evaluated as the effective collision likelihood G1, so that for an object with a lower collision likelihood F1, the visibility E1 that satisfies the threshold value of the effective collision likelihood G1 becomes lower. Therefore, for an object with a lower collision likelihood F1, unnecessary warnings can be suppressed by satisfying the corresponding visibility E1.

[0027] In addition, when the collision possibility is medium and constant, if only the collision possibility is considered without considering the visibility, it is difficult to judge whether the subject (driver) should actually pay attention or whether there is no problem if it is left alone. In such a case, by considering not only the collision possibility but also the visibility, such difficulty in judgment can be eliminated. Even if the collision possibility is the same medium, if the subject is looking at the object (or was looking until just before), it is considered that the subject is sufficiently aware of the existence of the object, so there is no need to warn unnecessarily and the subject can pay attention by himself. On the other hand, if the subject is not looking at the object at all, it is sufficient to warn the subject to pay attention in advance (at least to make the subject recognize the existence of the object) before the collision possibility increases. According to the configuration according to the first embodiment including the effective collision possibility calculation unit 70, both the visibility E1 and the collision possibility F1 are evaluated, so that such difficulty in judgment can be eliminated. In this way, the index of the effective collision possibility G1 can express the difference in whether or not to take action from the vehicle side even if the collision possibility is the same medium. By threshold judgment for the effective collision probability G1, it is possible to determine whether or not to issue a warning. Furthermore, by changing the strength of the warning depending on the magnitude of the effective collision probability G1 that is equal to or greater than the threshold used in the threshold judgment, it is possible to realize a dynamic warning mode without setting fine threshold stages for different parameters as in Patent Document 1.

[0028] (Warning part) The warning unit 80 receives the effective collision possibility G1 as an input, and outputs warning information H1, which is control information for controlling a warning operation by a warning device (not shown) based on the effective collision possibility G1. For a target whose effective collision possibility G1 is lower than a threshold, the warning information H1 is not output, and when the effective collision possibility is equal to or greater than the threshold, the warning information H1 is output. The strength and form of the warning may be changed according to the magnitude of the effective collision possibility G1. Furthermore, the form of the warning may be a visual presentation form, an audio presentation form, or any other presentation method that stimulates the five human senses. When the form of the warning is a visual presentation form, a display is used as the warning device (not shown). When the form of the warning is an audio presentation form, a speaker is used as the warning device (not shown). In addition, depending on the method of presenting the warning, an appropriate device such as a vibration device is used as the warning device (not shown).

[0029] <Operation> FIG. 2 is a flowchart illustrating an operation for performing driving assistance.

[0030] In step ST1, the imaging device 10 captures an in-vehicle image A1 including a target person and an outside-vehicle image B1 including a visually recognized object.

[0031] In step ST2, the data input unit 20 receives the obtained video, classifies it into an in-vehicle video A1 and an outside-vehicle video B1 required for calculating line-of-sight information and object information, and outputs it.

[0032] In step ST3, the gaze information calculation unit 30 calculates the gaze information C1 of the target person from the in-car image A1.

[0033] In step ST4, the object information calculation unit 40 detects objects present around the vehicle from the outside-of-vehicle image, and extracts object information D1 that is information about the objects.

[0034] In step ST5, the visibility calculation unit 50 calculates the visibility E1 from the line of sight information C1 and the object information D1. The visibility E1 may be a visibility that takes into account time-series attenuation.

[0035] In step ST6, the collision likelihood calculation unit 60 calculates a collision likelihood F1 of the object from the object information D1.

[0036] In step ST7, the effective collision likelihood calculation unit 70 calculates an effective collision likelihood G1 from the visibility E1 and the collision likelihood F1.

[0037] In step ST8, the warning unit 80 determines the content of the warning depending on the effective collision possibility G1.

[0038] Embodiment 2 <Configuration> A driving assistance device 2 and a driving assistance system SYS-V2 according to a second embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the driving assistance device 2 and the driving assistance system SYS-V2 according to the second embodiment of the present disclosure. As shown in Fig. 3, the driving assistance system SYS-V2 includes an imaging device 10 that captures an outside-vehicle image including a visual target and an inside-vehicle image including a target person, and the driving assistance device 2.

[0039] As an example, the driving assistance device 2 includes a data input unit 20 that receives as input an image captured by an imaging device 10 and classifies and outputs an in-vehicle image and an outside-vehicle image required for calculating gaze information and object information, a gaze information calculation unit 30 that calculates the gaze information of a subject from the in-vehicle image, an object information calculation unit 40 that detects objects around the vehicle from the outside-vehicle image and extracts information about the objects, a visibility calculation unit 50 that calculates visibility taking into account time-series attenuation from the gaze information and object information, a collision possibility calculation unit 60 that calculates the collision possibility of the object from the object information, an effective collision possibility calculation unit 70 that calculates the effective collision possibility from the visibility and collision possibility, a warning unit 80 that determines and presents the content of a warning depending on the effective collision possibility, and a vehicle information acquisition unit 90 that acquires vehicle information from a control system installed in the vehicle equipped with the device. As in the case of embodiment 1, all of the functional units of the driving assistance device 2 may be realized by an apparatus mounted in the device-equipped vehicle, or only some of the functional units of the driving assistance device 2 may be realized by an apparatus mounted in the device-equipped vehicle, with the remaining functional units being realized by an apparatus provided remotely.

[0040] (Imaging device) The imaging device 10 includes at least one camera, captures an in-vehicle image A1 including a plurality of frame images including a face image of a subject in the vehicle, and an outside-vehicle image B1 including a plurality of frame images including an object to be viewed by the subject, and outputs video data including the in-vehicle image A1 and the outside-vehicle image B1. The in-vehicle image A1 and the outside-vehicle image B1 may be captured as separate images using a single camera corresponding to each image, or may be captured using a stereo camera corresponding to each image in order to measure an accurate three-dimensional position. In addition, the in-vehicle image A1 and the outside-vehicle image B1 may be captured together in the same image using a camera that enables wide-range capture such as a spherical camera. The imaging device 10 is connected to the data input unit 20 of the driving support device 1.

[0041] (Data Entry Section) The data input unit 20 classifies the video data including the in-vehicle video A1 and the outside-vehicle video B1 into the in-vehicle video A1 required for calculating the line-of-sight information and the outside-vehicle video B1 required for calculating the object information, and outputs the classified in-vehicle video A1 and outside-vehicle video B1. The data input unit 20 is connected to the line-of-sight information calculation unit 30 and the object information calculation unit 40, and outputs the in-vehicle video A1 to the line-of-sight information calculation unit 30 and outputs the outside-vehicle video B1 to the object information calculation unit 40.

[0042] (Vehicle information acquisition unit) The vehicle information acquisition unit 90 acquires vehicle information I1 of the device-equipped vehicle from a control system mounted in the device-equipped vehicle, and outputs the acquired vehicle information I1. For example, vehicle speed, steering control information, or pedal control information may be acquired as the vehicle information I1. The vehicle information acquisition unit 90 is connected to the object information calculation unit 40.

[0043] (Gaze information calculation unit) The gaze information calculation unit 30 receives the vehicle interior image A1 as an input, calculates the gaze vector of the target person from the received vehicle interior image A1, and outputs the calculated gaze vector as gaze information C1. As a method for calculating the gaze vector, a model-based gaze detection method represented by the use of a corneal reflection image may be used, or an appearance-based method using machine learning may be used. The gaze information calculation unit 30 is connected to the visibility calculation unit 50.

[0044] (Object information calculation unit) The object information calculation unit 40 receives the vehicle exterior image B1 and the vehicle information I1 as input, detects an object that needs to be visually recognized for proper driving from the received vehicle exterior image B1 and vehicle information I1, and outputs object information D1 and vehicle information I1, which are information about the detected object. As the object information D1, type information indicating the type of the object may be used, such as "fixed indicator" if the object is a traffic light, or "moving object" if the object is a car or a pedestrian. When the object is a fixed indicator, the object information D1 may further include position information indicating the position of the object, size information indicating the detected size of the object, and state information represented by the lighting form of an LED (Light Emitting Diode). On the other hand, when the object is a moving object, the object information D1 may further include position information indicating the position of the object, size information indicating the detected size of the object, and time-series change information (movement information) indicating the traveling direction and speed of the object. The object information calculation unit 40 is connected to the visibility calculation unit 50 and the collision likelihood calculation unit 60, outputs object information D1 to the visibility calculation unit 50, and outputs object information D1 and vehicle information I1 to the collision likelihood calculation unit 60.

[0045] (Visibility calculation section) The visibility calculation unit 50 receives the gaze information C1 of the subject and the object information D1 as input, calculates the visibility E1 of each object using the received gaze information C1 and object information D1, and outputs the calculated visibility E1. The visibility E1 is an index indicating the degree to which the subject recognizes the detected object. The visibility calculation unit 50 calculates the visibility E1, for example, as follows.

[0046] First, the visibility calculation unit 50 calculates the degree of directional agreement between vectors using the line-of-sight unit vector of the line-of-sight vector at each time and the object unit vector of the object vector from the vehicle. As the degree of directional agreement, the angle between the line-of-sight unit vector and the object unit vector may be used, or the angle between these two unit vectors in the displacement direction per unit time may be used. The degree of directional agreement may be an index smoothed per certain time.

[0047] Next, the visibility calculation unit 50 calculates the sum of the directional agreement degrees by going back a certain time from the current time to the past and giving weights for each time. The weights may be set to be higher as the time approaches the current time, or nonlinear weights may be given. If a change in the state or movement of the object is observed within the visibility measurement time, a correction coefficient is defined for each type of object, the directional agreement degree before the time when the change in the object occurred is multiplied by the correction coefficient, and the visibility is calculated by taking the sum of the corrected directional agreement degrees within the measurement time. If the object is a fixed indicator such as a traffic light, the correction coefficient may be given as a constant multiple at the time when a change in the indication occurs. If the object is a moving object such as a car, the difference between the movement at the start of the visibility measurement and the movement at the current time may be calculated, and the correction coefficient may be dynamically defined according to the amount of the difference. The visibility calculation unit 50 is connected to the effective collision probability calculation unit 70.

[0048] (Collision possibility calculation unit) The collision possibility calculation unit 60 receives the object information D1 and the vehicle information I1 as input, calculates the collision possibility F1 for each object using the received object information D1 and vehicle information I1, and outputs the calculated collision possibility F1. The collision possibility F1 is an index indicating the degree of possibility that the detected object and the device-equipped vehicle will collide. When calculating the collision possibility F1, the calculation method may be changed according to the type of object indicated by the type information. For example, in the case of a fixed indicator such as a traffic light, the time limit from the position and motion information of the object to the stop instruction position is measured, and the collision possibility F1 is defined as an index inversely proportional to the time limit. Depending on the difference in the indication form from proceeding to stopping, a correction coefficient may be multiplied by the time limit. Here, examples of the indication form of the object include the color display of the traffic light and information on the road by a road information board (traffic jam, traffic accident, weather, etc.). Also, examples of the motion information include information on the speed, acceleration, and traveling direction of the object. Even if the object is a fixed object such as a traffic light, it appears to be moving relatively when viewed from a camera attached to the vehicle, so motion information exists.

[0049] If the object is a moving object such as an automobile, the time limit until contact with the device-equipped vehicle is measured from the object's position and motion information, and the collision probability F1 is defined as an index inversely proportional to the time limit. Furthermore, if the object is a moving object, the kinetic energy may be calculated and the collision probability F1 may be defined as an index proportional to the kinetic energy.

[0050] When calculating the motion information, the motion information of the vehicle information I1 and the relative motion information of the object information as viewed from the vehicle equipped with the device may be combined and replaced with motion information in real space, and the possibility of a collision may be calculated from the replaced motion information based on a prediction of the future motion of the vehicle equipped with the device and the object.

[0051] In this way, the collision likelihood calculation unit 60 selects at least one of the pointing form or the motion information of the object according to the type of the object, and calculates the collision likelihood F1. The collision likelihood calculation unit 60 is connected to the effective collision likelihood calculation unit 70.

[0052] (Effective collision probability calculation section) The effective collision likelihood calculation unit 70 receives the visibility E1 and the collision likelihood F1 as inputs, calculates the effective collision likelihood G1 using the received visibility E1 and collision likelihood F1, and outputs the calculated effective collision likelihood G1. The effective collision likelihood G1 is an index that decreases as the visibility E1 increases and increases as the collision likelihood F1 increases. The visibility E1, which is such an index, is expressed, for example, as in the above-mentioned formula (1). The effective collision likelihood calculation unit 70 is connected to the warning unit 80.

[0053] As explained in accordance with embodiment 1, by using such an effective collision possibility G1, various effects are achieved, such as the ability to simultaneously evaluate visibility E1 and collision possibility F1 as the effective collision possibility G1.

[0054] (Warning part) The warning unit 80 receives the effective collision possibility G1 as an input, and outputs warning information H1, which is control information for controlling a warning operation by a warning device (not shown) based on the effective collision possibility G1. For a target whose effective collision possibility G1 is lower than a threshold, the warning information H1 is not output, and when the effective collision possibility is equal to or greater than the threshold, the warning information H1 is output. The strength and form of the warning may be changed according to the magnitude of the effective collision possibility G1. Furthermore, the form of the warning may be a visual presentation form, an audio presentation form, or any other presentation method that stimulates the five human senses. When the form of the warning is a visual presentation form, a display is used as the warning device (not shown). When the form of the warning is an audio presentation form, a speaker is used as the warning device (not shown). In addition, depending on the method of presenting the warning, an appropriate device such as a vibration device is used as the warning device (not shown).

[0055] Fig. 4 is a diagram for illustrating the concept of effective collision probability according to the first and second embodiments of the present disclosure with a specific example. In the driving environment shown in Fig. 4, it is assumed that the moving objects O1 to O3 and the device-equipped vehicle S1 move in the direction of the arrows attached to each object or vehicle. It is assumed that the possibility of contact with the device-equipped vehicle S1 increases within the range of region V based on the motion state of the device-equipped vehicle S1 and the motion state of the object. It is further assumed that the line of sight of the subject is shifting from the object O3 to O2.

[0056] Here, the traveling direction of the object O1 is facing away from the area V, so the collision probability is calculated to be low. On the other hand, the visibility is also calculated to be low, so the effective collision probability based on the formula (1) is calculated to be low, and no warning is issued.

[0057] Object O2 is traveling in a direction that will pass the vehicle, and the collision probability was medium, but as the traveling direction changes along the way and the time until contact becomes shorter, the collision probability increases. The subject is currently viewing object O2, and maintains a high visibility state. When object O2 was traveling straight, the effective collision probability was low to medium, but as object O2 changes direction, the collision probability increases, and the effective collision probability increases. As the effective collision probability increases, the warning shifts in the direction it is displayed, and the presentation strength gradually increases.

[0058] Object O3 is traveling outside area V and has a low collision probability, and the subject looks at object O3, which has a low collision probability, once and then looks away for a while. When the subject looks at object O3, the effective collision probability is calculated to be low and no warning is displayed, but while the subject is looking away, the motion of object O3 changes and it moves toward area V. Visibility gradually decreases due to the passage of time while the subject is looking away and the change in the object's motion, and furthermore, the collision probability increases due to the change in movement toward area V, so the effective collision probability increases and a warning is displayed. If the traveling direction of object O3 is different from area V, visibility decreases while the collision probability also decreases, and the effective collision probability remains at a low level.

[0059] Assume that object O4 is a traffic light, and that there are multiple stages of indication forms, from proceeding to stopping. When the indication form of the traffic light changes, the degree of directional agreement between the line of sight vector obtained at the time before that point and the target vector decreases instantaneously, and visibility decreases. The collision probability is calculated according to the distance to the stop position and the vehicle speed, and when the collision probability is low, the effective collision probability increases to a medium level, making it possible to give a sufficient warning. On the other hand, when the collision probability is medium, the effective collision probability increases to a high level, and a strong warning is presented.

[0060] Embodiment 3 The road-to-vehicle driving assistance system according to the third embodiment includes the driving assistance device 1 or driving assistance system SYS-V1 described in the first embodiment, or the driving assistance device 2 or driving assistance system SYS-V2 described in the second embodiment, and a roadside system SYS-R capable of communicating with the driving assistance device 1 or 2. The roadside system SYS-R includes a roadside device 101 and a communication device 102. The roadside device 101 acquires information on the device-equipped vehicle S1 and objects around the device-equipped vehicle S1 using, for example, a surveillance camera on the road or a satellite, and transmits the acquired information to the object information calculation unit 40 of the driving assistance device 1 or 2 via the communication device 102. Detection of objects around the device-equipped vehicle S1 may be performed only by the roadside device 101, or may be performed by both the roadside device 101 and a vehicle-side device or system (the driving assistance device 1 or driving assistance system SYS-V1, or the driving assistance device 2 or driving assistance system SYS-V2). In addition, the object detected by the roadside device 101 may be different from the object detected by the vehicle side device or system (driving assistance device 1 or driving assistance system SYS-V1, or driving assistance device 2 or driving assistance system SYS-V2). Furthermore, the roadside device 101 may receive at least one of the vehicle speed, steering control information, and pedal control information from the device-equipped vehicle S1, calculate the possibility of collision between the vehicle and the object, and transmit the calculated information to the collision possibility calculation unit 60.

[0061] <Hardware configuration> Next, referring to FIG. 6, an example of the hardware configuration of the driving support devices 1 and 2 and the roadside device 101 according to the first to third embodiments will be described. These devices can be realized by the control circuit shown in FIG. 6A, that is, the processor 81 and the memory 82. An example of the processor 81 is a CPU (Central Processing Unit, also called a processing device, arithmetic device, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) or a system LSI (Large Scale Integration). The memory 82 is, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disk). The driving support device 1 or 2 or the roadside device 101 is realized by the processor 81 reading and executing a program stored in the memory 82. The program executed by the processor 81 may be provided as a computer program product stored in a computer-readable storage medium in the form of an installable or executable file. The program executed by the processor 81 may also be provided to the driving assistance device via a network such as the Internet.

[0062] As another example, the driving support devices 1 and 2 and the roadside device 101 according to the first to third embodiments may be realized by a dedicated processing circuit 83 shown in Fig. 6B. The processing circuit 83 is dedicated hardware. The processing circuit 83 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0063] Note that a part of the functions of the driving assistance device 1 or 2 or the roadside device 101 may be realized by software or firmware, and the remaining part may be realized by dedicated hardware.

[0064] <Additional Notes> Certain aspects of the various embodiments described above are summarized below.

[0065] (Appendix 1) The driving assistance device of Appendix 1 includes a gaze information calculation unit (30) that calculates gaze information of a subject from an in-vehicle image of the inside of the vehicle captured by an imaging device having at least one camera, an object information calculation unit (40) that detects objects present around the vehicle from an outside-vehicle image of the outside of the vehicle captured by the imaging device and extracts object information that is information about the detected object, a visibility calculation unit (50) that calculates a visibility indicating the degree to which the subject sees the detected object from the calculated gaze information and the extracted object information, a collision possibility calculation unit (60) that calculates a collision possibility indicating the likelihood of a collision between the detected object and the vehicle from the extracted object information, and an effective collision possibility calculation unit (70) that calculates an effective collision possibility that decreases as the calculated visibility increases and increases as the calculated collision possibility increases, from the calculated visibility and the calculated collision possibility.

[0066] According to the driving assistance device of Appendix 1, a judgment is made using an effective collision probability that simultaneously evaluates visibility and collision probability. Therefore, by issuing a warning based on the effective collision probability, a warning that simultaneously evaluates visibility and collision probability is possible. In addition, by making a threshold judgment on the effective collision probability, it is only necessary for an object with a lower collision probability to satisfy a lower visibility, so that unnecessary warnings that may be bothersome to the user can be suppressed.

[0067] In addition, because the effective collision probability is defined as an index that reflects both visibility and collision probability, it is possible to dynamically determine whether or not to issue a warning and adjust its intensity according to the visibility of objects with low to medium collision probability. This makes it possible to present a warning of appropriate intensity for objects that truly require attention or action while reducing annoyance.

[0068] (Appendix 2) The driving assistance device of Appendix 2 is the driving assistance device described in Appendix 1, further comprising a warning unit (80) that determines the content of a warning to be given to the target person in accordance with the calculated effective collision possibility and controls the warning operation by the warning device. .

[0069] According to the driving assistance device of Appendix 2, it is possible to control the warning operation for the target person according to the effective collision possibility.

[0070] (Appendix 3) The driving assistance device of Appendix 3 is a driving assistance device as described in Appendix 1 or 2, further comprising a vehicle information acquisition unit (90) that acquires at least one of vehicle speed, steering control information, and pedal control information as vehicle information from a control system mounted on the vehicle, and the collision likelihood calculation unit calculates the collision likelihood based on the extracted object information and the acquired vehicle information.

[0071] According to the driving assistance device of Appendix 3, by using the motion information of the vehicle equipped with the device in addition to the relative motion of surrounding objects as seen from the vehicle equipped with the device obtained from the outside-vehicle image, it becomes possible to calculate the motion of the object in real space, and the accuracy of calculating the possibility of a collision regarding approach and contact between objects is improved based on the motion prediction of the object and the motion prediction of the vehicle equipped with the device.

[0072] (Appendix 4) The driving assistance device of Supplementary Note 4 is a driving assistance device according to any one of Supplementary Notes 1 to 3, wherein the visibility calculation unit calculates a directional agreement between a direction of a target vector traveling from the vehicle to the target object and a direction of a gaze vector of the target object at a predetermined timing, calculates a weighted directional agreement by weighting the directional agreement at each time point within a certain time period in the past from the current time, and, if a change in state or movement of the target object occurs within the measurement time, calculates a corrected directional agreement by multiplying the calculated weighted directional agreement before the time when the change occurred by a coefficient, and calculates the visibility by calculating the calculated weighted directional agreement or the sum of the calculated corrected directional agreement.

[0073] According to the driving support device of appendix 4, by weighting the directional agreement degree at each time within a certain time period from the current time, the visibility is calculated based on the weighted directional agreement degree, so that it is possible to reflect in the visibility the attenuation over time of the memory of the object that occurs when the subject turns away from the object. Furthermore, by giving a weight that is higher the closer to the current time, it is possible to adjust the intensity of attenuation over time. Furthermore, by giving a coefficient according to the change in state or movement of the surrounding object, it is possible to reflect in the visibility the state in which the subject is unable to grasp the change in the behavior of the object after it has been seen once.

[0074] (Appendix 5) The driving assistance device of Supplementary Note 5 is a driving assistance device described in any one of Supplementary Notes 1 to 4, wherein the object information calculation unit calculates type information indicating a type of the detected object, and status information indicating a status or movement information indicating a movement.

[0075] According to the driving assistance device of Supplementary Note 5, type information indicating the type of the detected object is calculated, so that it is possible to calculate the visibility and the collision possibility using an index for each type of object.

[0076] (Appendix 6) The driving assistance device of Supplementary Note 6 is the driving assistance device described in Supplementary Note 5, wherein the collision likelihood calculation unit calculates the collision likelihood based on at least one of the calculated state information or the motion information in accordance with the calculated type information.

[0077] According to the driving assistance device of Appendix 6, when the object is a fixed indicator such as a traffic light, it is possible to calculate the collision possibility based on the illumination state of the LED and the distance to the fixed indicator, and when the object is a moving object such as a car, it is possible to calculate the collision possibility based on the motion information, making it possible to calculate the collision possibility for each type of object.

[0078] (Appendix 7) The road-to-vehicle driving assistance system of Supplementary Note 7 is a road-to-vehicle driving assistance system including a driving assistance device (1; 2) described in any one of Supplements 1 to 6, and a roadside system (SYS-R) capable of communicating with the driving assistance device, wherein the roadside system transmits information about the object or other objects other than the object that exist around the vehicle to the driving assistance device, and the object information calculation unit of the driving assistance device extracts the object information based on the outside-vehicle image and the transmitted information.

[0079] (Appendix 8) The driving assistance method of appendix 8 is a driving assistance method performed by a driving assistance device including a gaze information calculation unit (30), an object information calculation unit (40), a visibility calculation unit (50), a collision likelihood calculation unit (60), and an effective collision likelihood calculation unit (70), comprising: a step (ST3) in which the gaze information calculation unit calculates gaze information of a target from an interior image of the inside of the vehicle captured by an imaging device including at least one camera; a step (ST4) in which the object information calculation unit detects an object present in the periphery of the vehicle from an outside image of the outside of the vehicle captured by the imaging device, and extracts object information that is information of the detected object; The method includes a step (ST5) in which a visibility calculation unit calculates visibility indicating the degree to which the subject can see the detected object from the calculated line of sight information and the extracted object information, a step (ST6) in which the collision likelihood calculation unit calculates a collision likelihood indicating the degree of possibility of a collision between the detected object and the vehicle from the extracted object information, and a step (ST7) in which the effective collision likelihood calculation unit calculates an effective collision likelihood from the calculated visibility and the calculated collision likelihood, which decreases as the calculated visibility increases and increases as the calculated collision likelihood increases.

[0080] (Appendix 9) The driving assistance program of Appendix 9 causes a computer to execute the following steps: a step (ST3) of calculating gaze information of a subject from an in-vehicle image of the inside of the vehicle captured by an imaging device having at least one camera; a step (ST4) of detecting an object present around the vehicle from an outside image of the outside of the vehicle captured by the imaging device, and extracting object information which is information about the detected object; a step (ST5) of calculating a visibility indicating the degree to which the subject can see the detected object from the calculated gaze information and the extracted object information; a step (ST6) of calculating a collision probability indicating the likelihood that the detected object will collide with the vehicle from the extracted object information; and a step (ST7) of calculating an effective collision probability which decreases as the calculated visibility increases and increases as the calculated collision probability increases, from the calculated visibility and the calculated collision probability.

[0081] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]

[0082] The driving assistance device of the present disclosure can be used as a device for assisting in driving a vehicle. [Explanation of symbols]

[0083] 1 driving assistance device, 2 driving assistance device, 10 imaging device, 20 data input unit, 30 gaze information calculation unit, 40 object information calculation unit, 50 visibility calculation unit, 60 collision probability calculation unit, 70 effective collision probability calculation unit, 80 warning unit, 81 processor, 82 memory, 83 processing circuit, 90 vehicle information acquisition unit, 101 roadside device, 102 communication device, SYS-R roadside system, SYS-V1 driving assistance system, SYS-V2 driving assistance system.

Claims

1. a gaze information calculation unit that calculates gaze information of a target person from an in-vehicle image of the inside of the vehicle captured by an imaging device having at least one camera; an object information calculation unit that detects an object present around the vehicle from an outside image of the outside of the vehicle captured by the imaging device and extracts object information that is information about the detected object; a visibility calculation unit that calculates a visibility for each time period within a certain past time range from the current time, which indicates the degree to which the subject visually recognizes the object at each time period, from the calculated line of sight information and the extracted object information, corrects the calculated visibility by multiplying the calculated visibility by a correction coefficient corresponding to the time lapse from the current time to each past time period, re-corrects the corrected visibility by multiplying the calculated visibility by a correction coefficient corresponding to a change in state of the object, and calculates a visibility associated with the current time period based on the re-corrected visibility; a collision probability calculation unit that calculates a collision probability indicating a degree of possibility that the detected object will collide with the vehicle from the extracted object information; an effective collision probability calculation unit that calculates an effective collision probability based on the calculated visibility and the calculated collision probability, the effective collision probability decreasing with an increase in the calculated visibility and increasing with an increase in the calculated collision probability; A driving assistance device comprising:

2. a gaze information calculation unit that calculates gaze information of a target person from an in-vehicle image of the inside of the vehicle captured by an imaging device having at least one camera; an object information calculation unit that detects an object present around the vehicle from an outside image of the outside of the vehicle captured by the imaging device and extracts object information that is information about the detected object; a visibility calculation unit that calculates a visibility indicating a degree to which the subject visually recognizes the detected object from the calculated line of sight information and the extracted object information; a collision probability calculation unit that calculates a collision probability indicating a degree of possibility that the detected object will collide with the vehicle from the extracted object information; an effective collision probability calculation unit that calculates an effective collision probability based on the calculated visibility and the calculated collision probability, the effective collision probability decreasing with an increase in the calculated visibility and increasing with an increase in the calculated collision probability; Equipped with the visibility calculation unit calculates a degree of directional agreement between a direction of a target vector traveling from the vehicle to the target object and a direction of a line-of-sight vector of the target person at a predetermined timing; Calculate a weighted directional agreement degree by weighting the directional agreement degree for each time within a certain time period in the past from the current time; If a change in state or movement occurs with respect to the object within the measurement time, a corrected directional agreement is calculated by multiplying the calculated and weighted directional agreement before the time when the change occurred by a coefficient; Calculating the visibility by calculating the sum of the calculated and weighted directional agreement degree or the calculated corrected directional agreement degree. Driving assistance device.

3. A warning unit that determines the content of a warning to be given to the target person according to the calculated effective collision possibility and controls a warning operation by the warning device. A driving assistance device according to claim 1 or 2.

4. a vehicle information acquisition unit that acquires at least one of vehicle speed, steering control information, and pedal control information as vehicle information from a control system mounted on the vehicle; The collision likelihood calculation unit calculates the collision likelihood based on the extracted object information and the acquired vehicle information. A driving assistance device according to claim 1 or 2.

5. The object information calculation unit calculates type information indicating a type of the detected object, and state information indicating a state or movement information indicating a movement of the object. A driving assistance device according to claim 1 or 2.

6. the collision likelihood calculation unit calculates the collision likelihood based on at least one of the calculated state information or the calculated motion information in response to the calculated type information; A driving assistance device according to claim 5.

7. A driving assistance device according to claim 1 or 2; A road-to-vehicle driving assistance system including a roadside system capable of communicating with the driving assistance device, the roadside system transmits information about the object or another object different from the object that exists around the vehicle to the driving assistance device; The object information calculation unit of the driving assistance device extracts the object information based on the outside-of-vehicle image and the transmitted information. Road-to-vehicle driving assistance system.

8. A driving assistance method performed by a driving assistance device including a line-of-sight information calculation unit, an object information calculation unit, a visibility calculation unit, a collision likelihood calculation unit, and an effective collision likelihood calculation unit, A step in which the gaze information calculation unit calculates gaze information of a target person from an in-vehicle image of an interior of a vehicle captured by an imaging device having at least one camera; a step in which the object information calculation unit detects an object present around the vehicle from an outside image of the outside of the vehicle captured by the imaging device, and extracts object information that is information on the detected object; a step in which the visibility calculation unit calculates a visibility for each time period indicating a degree to which the subject visually recognizes the object at each time period within a certain time range from the current time period based on the calculated gaze information and the extracted object information, corrects the calculated visibility period by multiplying the calculated visibility period by a correction coefficient corresponding to the time lapse from the current time period to each past time period, re-corrects the corrected visibility period by multiplying the calculated visibility period by a correction coefficient corresponding to a change in state of the object, and calculates a visibility period associated with the current time period based on the re-corrected visibility period; a step of calculating a collision likelihood indicating a degree of likelihood that the detected object will collide with the vehicle from the extracted object information by the collision likelihood calculation unit; a step of the effective collision likelihood calculation unit calculating an effective collision likelihood from the calculated visibility and the calculated collision likelihood, the effective collision likelihood decreasing with an increase in the calculated visibility and increasing with an increase in the calculated collision likelihood; A driving assistance method comprising:

9. Calculating gaze information of a target person from an in-vehicle image of the inside of a vehicle captured by an image capturing device having at least one camera; A step of detecting an object present around the vehicle from an outside image of the outside of the vehicle captured by the imaging device, and extracting object information that is information about the detected object; a step of calculating a visibility for each time period within a certain time range from the current time based on the calculated line-of-sight information and the extracted object information, the visibility indicating the degree to which the subject views the object at each time period, correcting the calculated visibility by multiplying the visibility by a correction coefficient corresponding to the time lapse from the current time to each past time period, re-correcting the corrected visibility by multiplying the visibility by a correction coefficient corresponding to a change in state of the object, and calculating a visibility associated with the current time period based on the re-corrected visibility; calculating a collision probability indicating a degree of possibility that the detected object will collide with the vehicle from the extracted object information; calculating an effective collision likelihood from the calculated visibility and the calculated collision likelihood, the effective collision likelihood decreasing as the calculated visibility increases and increasing as the calculated collision likelihood increases; A driving assistance program that causes a computer to execute the above.

Images