Driver assistance device, road-to-vehicle driver assistance system, driver assistance procedure and driver assistance program

Abstract

A driver assistance device comprises the following: a gaze-line information processing unit (30) for calculating gaze-line information of a subject from a vehicle interior video of the interior of a vehicle, captured by an imaging device comprising at least one camera; a target object information processing unit (40) for detecting a target object located in the vicinity of the vehicle from a vehicle exterior video of the exterior of the vehicle, captured by the imaging device, and extracting target object information, which is information about the detected target object; a detectability processing unit (50) for calculating a detectability, which indicates the degree of visual recognition of the detected target object by the subject, from the calculated gaze-line information and the extracted target object information.a collision possibility calculation unit (60) for calculating a collision possibility, which indicates the extent of a collision possibility degree between the detected target object and the vehicle, from the extracted target object information, and an effective collision possibility calculation unit (70) for calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility.

Description

Technical area

[0001] The present invention relates to a driver assistance technology. State of the art

[0002] Patent Literature 1 discloses a driving assistance device comprising: a detectability calculation unit that calculates, based on the direction of a sight line of a driver of a vehicle and information regarding one or more target objects located in the surroundings of the vehicle, a first detectability that is a degree of detectability of the driver of the vehicle to each of the one or more target objects; a risk calculation unit that calculates a risk level indicating the possibility of contact of the vehicle with each of the one or more target objects based on a state of each of the one or more target objects and a state of the vehicle; and a display control unit that causes first warning information associated with a first target object to be displayed on a display unit.if the first detectability of the first target object among the one or more target objects is less than a first threshold, and causes the first warning information to be hidden if the first detectability is equal to or greater than the first threshold, wherein the display control unit causes, if the first warning information is not displayed on the display unit and the risk level of the first target object is equal to or greater than a second threshold, a second warning information for the first target object to be displayed on the display unit along with the first target object.

[0003] As described above, in the driving assistance device in Patent Literature 1, after determining whether or not the first detectability of the first target object is less than the first threshold and performing display control of the first warning information, the display control unit performs display control to display the second warning information when the risk level of the first target object is equal to or greater than the second threshold in a state where the first warning information is not displayed. BibliographyPatent literature

[0004] Patent literature 1: JP 2020- 166 542 A Summary of the inventionTechnical problem

[0005] With a configuration in which the risk level is additionally evaluated under the premise that the detectability is evaluated, as in the driving assistance device in Patent Literature 1, there is a problem that it is not possible to simultaneously evaluate the detectability and the risk level (collision possibility) and perform a warning.

[0006] The present invention has been conceived to solve such a problem, and it is the object of the present invention to provide a driving assistance technique capable of simultaneously evaluating detectability and collision possibility and performing a warning. Solution to the problem

[0007] One aspect of a driving assistance device according to an embodiment of the present invention includes: a sight line information calculation unit for calculating sight line information of a subject from a vehicle interior video of the interior of a vehicle acquired by an imaging device having at least one camera; a target object information calculation unit for detecting a target object located in the surroundings of the vehicle from a vehicle exterior video of the exterior of the vehicle acquired by the imaging device and extracting target object information that is information of the detected target object; a detectability calculation unit for calculating a detectability that indicates the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information;a collision possibility calculation unit for calculating a collision possibility indicating the degree of a collision possibility between the detected target object and the vehicle from the extracted target object information, and an effective collision possibility calculation unit for calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility. Advantageous effects of the invention

[0008] According to one aspect of the driving assistance device according to embodiments of the present invention, it is possible to simultaneously evaluate the detectability and the risk level and perform a warning. Short description of the drawings Fig. 1 is a block diagram illustrating a driving assistance device and a driving assistance system according to a first embodiment of the present invention. Fig. 2 is a flowchart for describing the operation for providing driving assistance according to the first embodiment of the present invention. Fig. 3 is a block diagram illustrating a driving assistance device and a driving assistance system according to a second embodiment of the present invention. Fig. 4 is a diagram illustrating a specific example for describing the concept of an effective collision possibility regarding the driving assistance device and the driving assistance system according to the first and second embodiments of the present invention. Fig. 5 is a diagram illustrating a configuration example of a roadside system including a roadside device according to a third embodiment of the present invention. Fig. 6A is a diagram illustrating a configuration example of hardware of the driving assistance device and the roadside device according to the first to third embodiments of the present invention. Fig. 6B is a diagram illustrating a configuration example of hardware of the driving assistance device and the roadside device according to the first to third embodiments of the present invention. Description of embodiments

[0009] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. Components designated by the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted. First embodiment configuration

[0010] A driver assistance device 1 and a driver assistance system SYS-V1 according to a first embodiment of the present invention are described with reference to Fig. 1 described. Fig. 1 is a block diagram illustrating the driving assistance device 1 and the driving assistance system SYS-V1 according to the first embodiment of the present invention. As shown in Fig. 1, the driving assistance system SYS-V1 includes an imaging device 10 that captures a vehicle exterior video including a visual target object and a vehicle interior video including a subject, and the driving assistance device 1.

[0011] As an example, the driving assistance device 1 includes: a data input unit 20 that receives images captured by the imaging device 10 as inputs, classifies the images into a vehicle interior video and a vehicle exterior video necessary for calculating sight line information and target object information, and outputs the classified images; a sight line information calculation unit 30 that calculates sight line information of a subject from the vehicle interior video; a target object information calculation unit 40 that detects a target object in the vicinity of the vehicle from the vehicle exterior video and extracts information of the target object; a detectability calculation unit 50 that calculates detectability taking time-series attenuation into account from the sight line information and the target object information;a collision possibility calculation unit 60 that calculates a collision possibility of the target object from the target object information; an effective collision possibility calculation unit 70 that calculates an effective collision possibility from the detectability and the collision possibility; and a warning unit 80 that determines and presents warning contents according to the effective collision possibility.

[0012] All functional units of the driving assistance device 1 can be implemented with a device mounted on the vehicle, or only some functional units of the driving assistance device 1 can be implemented with a device mounted on the vehicle, and the remaining functional units can be implemented with a device located remotely. Hereinafter, the vehicle on which the driving assistance device 1 is mounted is referred to as a device-mounted vehicle. Imaging device

[0013] The imaging device 10 has at least one camera, captures a vehicle interior video A1 including a plurality of frame images including a facial image of a subject in the vehicle, and a vehicle exterior video B1 including a plurality of frame images including a target object to be visually recognized by the subject, and outputs video data including the vehicle interior video A1 and the vehicle exterior video B1.

[0014] The vehicle interior video A1 and the vehicle exterior video B1 can be captured as individual videos using a single camera dedicated to each video, or they can be captured using a stereo camera dedicated to each video to measure an accurate three-dimensional position. Furthermore, the vehicle interior video A1 and the vehicle exterior video B1 can be collectively captured into the same video using a camera capable of capturing a wide range, such as an omnidirectional camera. The imaging device 10 is connected to the data input unit 20 of the driving assistance device 1. Data input unit

[0015] The data input unit 20 classifies video data including the vehicle interior video A1 and the vehicle exterior video B1 into the vehicle interior video A1 necessary for calculating the sight line information and the vehicle exterior video B1 necessary for calculating the target object information, and outputs the classified vehicle interior video A1 and vehicle exterior video B1. The data input unit 20 is connected to the sight line information calculation unit 30 and the target object information calculation unit 40, outputs the vehicle interior video A1 to the sight line information calculation unit 30, and outputs the vehicle exterior video B1 to the target object information calculation unit 40. Calculation unit for sight line information

[0016] The sight line information calculation unit 30 receives the vehicle interior video A1 as input, calculates a sight line vector of the subject from the received vehicle interior video A1, and outputs the calculated sight line vector as sight line information C1. As a method for calculating the sight line vector, a model-based sight line detection method embodied by using a corneal reflection image or an appearance-based method by machine learning may be used. The sight line information calculation unit 30 is connected to the recognizability calculation unit 50. Calculation unit for target object information

[0017] The target object information calculation unit 40 receives the vehicle exterior video B1 as input, detects a target object that needs to be visually recognized when performing appropriate driving from the received vehicle exterior video B1, and outputs target object information D1, which is information regarding the detected target object. Type information indicating the type of the target object can be used as the target object information D1, such as "stationary indicator" when the target object is a traffic signal, and "moving object" when the target object is an automobile or a pedestrian.

[0018] If the target object is a stationary indicator, the target object information D1 may further include position information indicating the position of the target object, size information indicating the detected size of the target object, and state information embodied by a light mode of a light-emitting diode (LED). On the other hand, if the target object is a moving object, the target object information D1 may further include position information indicating the position of the target object, size information indicating the detected size of the target object, and time-series change information (movement information) indicating the traveling direction and speed of the target object.

[0019] The target object information calculation unit 40 is connected to the detectability calculation unit 50 and the collision possibility calculation unit 60, and outputs the target object information D1 to both. Recognizability calculation unit

[0020] The detectability calculation unit 50 receives the subject's sight line information C1 and target object information D1 as inputs, calculates a detectability E1 for each target using the received sight line information C1 and target object information D1, and outputs the calculated detectability E1. The detectability E1 is an index indicating the degree to which the subject visually detects the detected target object. The detectability calculation unit 50 calculates the detectability E1, for example, as follows.

[0021] First, the detectability calculation unit 50 calculates a direction matching degree between vectors by using a sight line unit vector of the sight line vector and a target unit vector of the target vector from the vehicle toward the target object at any given time. The direction matching degree can be the degree of an angle formed by the sight line unit vector and the target unit vector, or the degree of an angle formed by two unit vectors in a displacement direction per unit time of those two unit vectors. The direction matching degree can be an index smoothed per a certain time.

[0022] Next, the detectability calculation unit 50 goes back a certain time from the present time toward the past and calculates the sum of the directional matching degrees, to which a weight is assigned according to each time. The weight may be higher toward the present time, or a non-linear weight may be assigned. In a case where a state change or a movement change of the target object is observed within a detectability measurement time, a correction coefficient is defined for each type of target object, the directional matching degree before the time when the change in the target object occurred is multiplied by the correction coefficient, and the sum of the corrected directional matching degrees within the measurement time is obtained, and the detectability is calculated.

[0023] In a case where the target object is a stationary indicator represented by a traffic signal, the correction coefficient can be specified by a constant multiple at the time the indicator content changes. If the target object is a moving object represented by an automobile, the difference between the movement at the start of the detectability measurement and the movement at the current time can be calculated, and the correction coefficient can be dynamically defined depending on the value of the difference. The detectability calculation unit 50 is connected to the effective collision possibility calculation unit 70. Collision possibility calculation unit

[0024] The collision possibility calculation unit 60 receives the target object information D1 as input, calculates a collision possibility F1 with respect to each target object using the received target object information D1, and outputs the calculated collision possibility F1. The collision possibility F1 is an index indicating the degree of possibility that the detected target object will collide with the device-mounted vehicle. When calculating the collision possibility F1, the calculation method can be changed depending on the type of target object indicated by the type information.

[0025] For example, in the case of a stationary indicator represented by a traffic signal, a time limit for a stop instruction position is measured from the position and movement information of the target object, and the collision probability F1 is defined as an index inversely proportional to the time limit. The time limit can be multiplied by a correction coefficient depending on the difference in instruction form from "Go" to "Stop." Examples of the instruction form of the target object include: the color display of a traffic signal, information about a road on a road information board (traffic congestion, accidents, weather, or the like), and the like.

[0026] Furthermore, as an example, the motion information is information regarding the speed, acceleration, and direction of travel of the target object. Even if the target object is a stationary object, such as a signal, the target object appears to be moving relatively when viewed by a camera mounted on the vehicle, and thus, motion information is present.

[0027] If the target object is a moving object, such as a car, the time limit until contact with the device-mounted vehicle is measured from the position and motion information of the target object, and a collision probability F1 is defined as an index inversely proportional to the time limit. Furthermore, in a case where the target object is a moving object, the kinetic energy can be calculated, and the collision probability F1 can be defined as an index proportional to the kinetic energy.

[0028] In this way, the collision possibility calculation unit 60 selects at least one of the instruction form or the movement information of the target object, depending on the type of the target object, and calculates the collision possibility F1. The collision possibility calculation unit 60 is connected to the effective collision possibility calculation unit 70. Calculation unit for effective collision possibilities

[0029] The effective collision possibility calculation unit 70 receives the detectability E1 and the collision possibility F1 as inputs, calculates an effective collision possibility G1 using the received detectability E1 and collision possibility F1, and outputs the calculated effective collision possibility G1. The effective collision possibility G1 is an index that decreases according to an increase in the detectability E1 and increases according to an increase in the collision possibility F1. The detectability E1, which is such an index, is expressed, for example, as Equation (1). Here, K is a constant. The effective collision possibility calculation unit 70 is connected to the warning unit 80. G1=K×F1 / (1+E1)

[0030] As stated in expression (1), the effective collision possibility G1 decreases when the collision possibility F1 remains constant and the detectability E1 increases. Conversely, the effective collision possibility G1 increases when the detectability E1 remains constant and the collision possibility F1 increases. Since the detectability E1 and the collision possibility F1 are contained in a single equation, they can be evaluated simultaneously.

[0031] With the configuration in which the risk level is additionally evaluated on the premise of evaluating detectability, as in Patent Literature 1, there is a problem that the detectability and the risk level (collision possibility) cannot be evaluated simultaneously, and a warning cannot be issued. On the other hand, with the configuration according to the first embodiment, which has the effective collision possibility calculation unit 70, it is possible to simultaneously evaluate the detectability E1 and the collision possibility F1 as the effective collision possibility G1. By using such an effective collision possibility G1, it is possible to issue a warning in which the detectability E1 and the collision possibility F1 are simultaneously evaluated.

[0032] By performing the warning in which the detectability E1 and the collision possibility F1 are evaluated at the same time, it is possible to eliminate a nuisance or disturbance to the user that may be caused by the configuration according to Patent Literature 1. That is, with the configuration of performing the display control when displaying the second warning information when the risk level of the first target object is equal to or greater than the second threshold, in a state where the first warning information is not displayed, after determining whether or not the first detectability of the first target object is less than the first threshold, and performing the display control of the first warning information, as in Patent Literature 1, the determination of whether or not to display the warning information (first warning information) is made depending on,whether or not the detectability (first detectability) is equal to or greater than the threshold (first threshold) when the risk level of the target object (first target object) is less than the threshold (second threshold), and consequently, there is a problem that the warning display occurs and the user of the facility feels disturbed when the detectability of the target object with a low risk level (ie, the target object whose risk level is less than the second threshold) is low (ie, the detectability is less than the first threshold).

[0033] On the other hand, with the configuration according to the first embodiment, which includes the effective collision possibility calculation unit 70, the relationship between the detectability E1 and the collision possibility F1 is evaluated at the same time as the effective collision possibility G1. Consequently, for a target object whose collision possibility F1 is lower, the detectability E1 becomes lower, satisfying the effective collision possibility threshold G1. Therefore, for a target object whose collision possibility F1 is lower, an unnecessary warning can be suppressed by satisfying the detectability E1 associated with it.

[0034] Furthermore, in a case where the collision probability is moderate and constant, it is difficult to determine whether the subject (driver) can actually be induced to pay attention or whether failing to do so will not cause a problem if only the collision probability is considered without considering detectability. In such a case, the difficulty of such a determination can be eliminated by considering not only the collision probability but also detectability.

[0035] Even with the same moderate collision probability, the following holds: If the subject is looking at the target object (or immediately in front of it), it is considered that the subject is sufficiently aware of the presence of the target object, and consequently, there is no need to issue an unnecessary warning, and the subject can be alert. Conversely, in a case where the subject is not looking at the target object at all, it is sufficient for a warning to be issued in such a way that advance attention is established (at least the presence of the target object is recognized) before the collision probability increases.

[0036] With the configuration according to the first embodiment, which includes the effective collision possibility calculation unit 70, both the detectability E1 and the collision possibility F1 are evaluated, thus eliminating the difficulty of such determination. As described above, the effective collision possibility indicator G1 is capable of expressing the difference between whether an action should be initiated by the vehicle, even with the same medium-sized collision possibility.

[0037] By determining the threshold value for the effective collision possibility G1, it is possible to initially determine whether or not to issue a warning. Furthermore, by changing the warning intensity depending on the strength of the effective collision possibility G1 to be equal to or greater than the threshold value used in the threshold determination, it is possible to implement a dynamic warning mode without finely specifying the threshold levels for different parameters, as in Patent Literature 1. warning unit

[0038] The warning unit 80 receives the effective collision possibility G1 as an input and outputs warning information H1, which is control information for controlling a warning operation by a warning device (not shown) based on the effective collision possibility G1. The warning information H1 is not output for a target for which the effective collision possibility G1 is lower than the threshold, and the warning information H1 is output when the effective collision possibility is equal to or greater than the threshold.

[0039] Depending on the severity of the actual collision possibility G1, the intensity and form of the warning can be changed. Furthermore, the form of the warning can be a visual presentation, an audio presentation, or another presentation method that appeals to the five human senses. In a case where the warning is a visual presentation, a display is used as the warning device, which is not shown.

[0040] In a case where the warning mode is an audio presentation form, a speaker (not shown) is used as the warning device. Furthermore, a suitable device, such as a vibration device (not shown), is used as the warning device, depending on the warning presentation method. Operation

[0041] Fig. 2 is a flowchart for describing the operation for providing driving assistance.

[0042] In step ST1, the imaging device 10 acquires the vehicle interior video A1 including the subject and the vehicle exterior video B1 including the visual target object.

[0043] In step ST2, the data input unit 20 receives the obtained videos as inputs, classifies the videos into the vehicle interior video A1 and the vehicle exterior video B1 necessary for calculating the sight line information and the target object information, and outputs the classified videos.

[0044] In step ST3, the sight line information calculation unit 30 calculates the sight line information C1 of the subject from the vehicle interior video A1.

[0045] In step ST4, the target object information calculation unit 40 detects a target object located in the surroundings of the vehicle from the vehicle exterior video, and extracts the target object information D1, which is information of the target object.

[0046] In step ST5, the detectability calculation unit 50 calculates the detectability E1 from the sight line information C1 and the target object information D1. The detectability E1 may be a detectability taking time-series attenuation into account.

[0047] In step ST6, the collision possibility calculation unit 60 calculates the collision possibility F1 of the target object from the target object information D1.

[0048] In step ST7, the effective collision possibility calculation unit 70 calculates the effective collision possibility G1 from the detectability E1 and the collision possibility F1.

[0049] In step ST8, the warning unit 80 determines the content of the warning depending on the effective collision possibility G1. Second embodiment configuration

[0050] A driver assistance device 2 and a driver assistance system SYS-V2 according to a second embodiment of the present invention are described with reference to Fig. 3 described. Fig. 3 is a block diagram illustrating the driving assistance device 2 and the driving assistance system SYS-V2 according to the second embodiment of the present invention. As shown in Fig. 3, the driving assistance system SYS-V2 includes an imaging device 10 that captures a vehicle exterior video including a visual target object and a vehicle interior video including a subject, and a driving assistance device 2.

[0051] As an example, the driving assistance device 2 includes: a data input unit 20 that receives images captured by the imaging device 10 as inputs, classifies the images into a vehicle interior video and a vehicle exterior video necessary for calculating sight line information and target object information, and outputs the classified images; a sight line information calculation unit 30 that calculates sight line information of a subject from the vehicle interior video; a target object information calculation unit 40 that detects a target object in the vicinity of the vehicle from the vehicle exterior video and extracts information of the target object; a detectability calculation unit 50 that calculates detectability taking time-series attenuation into account from the sight line information and the target object information;a collision possibility calculation unit 60 that calculates a collision possibility of the target object from the target object information; an effective collision possibility calculation unit 70 that calculates an effective collision possibility from the detectability and the collision possibility; a warning unit 80 that determines and presents warning contents according to the effective collision possibility; and a vehicle information acquisition unit 90 that acquires vehicle information from a control system mounted on a device-mounted vehicle.

[0052] As in the case of the first embodiment, all the functional units of the driving assistance device 2 may be implemented by a device mounted on the device-mounted vehicle, or only a part of the functional units of the driving assistance device 2 may be implemented by a device mounted on the vehicle, and the remaining functional units may be implemented by a device located remotely. Imaging device

[0053] The imaging device 10 has at least one camera, captures a vehicle interior video A1 including a plurality of frame images including a facial image of a subject in the vehicle, and a vehicle exterior video B1 including a plurality of frame images including a target object to be visually recognized by the subject, and outputs video data including the vehicle interior video A1 and the vehicle exterior video B1.

[0054] The vehicle interior video A1 and the vehicle exterior video B1 can be captured as individual videos using individual cameras, each related to the vehicle interior video A1 and the vehicle exterior video B1, or they can be captured using a stereo camera related to each of the vehicle interior video A1 and the vehicle exterior video B1 to measure an accurate three-dimensional position. Furthermore, the vehicle interior video A1 and the vehicle exterior video B1 can be captured together in the same video using a camera capable of capturing a wide range, such as an omnidirectional camera. The imaging device 10 is connected to the data input unit 20 of the driving assistance device 1. Data input unit

[0055] The data input unit 20 classifies video data including the vehicle interior video A1 and the vehicle exterior video B1 into the vehicle interior video A1 necessary for calculating the sight line information and the vehicle exterior video B1 necessary for calculating the target object information, and outputs the classified vehicle interior video A1 and vehicle exterior video B1.

[0056] The data input unit 20 is connected to the sight line information calculation unit 30 and the target object information calculation unit 40, outputs the vehicle interior video A1 to the sight line information calculation unit 30, and outputs the vehicle exterior video B1 to the target object information calculation unit 40. Vehicle information acquisition unit

[0057] The vehicle information acquisition unit 90 acquires vehicle information I1 of the device-mounted vehicle from a control system mounted on the device-mounted vehicle and outputs the acquired vehicle information I1. For example, the vehicle speed, steering control information, or pedal control information can be acquired as the vehicle information I1. The vehicle information acquisition unit 90 is connected to the target object information calculation unit 40. Calculation unit for sight line information

[0058] The sight line information calculation unit 30 receives the vehicle interior video A1 as input, calculates a sight line vector of the subject from the received vehicle interior video A1, and outputs the calculated sight line vector as sight line information C1. As a method for calculating the sight line vector, a model-based sight line detection method embodied by using a corneal reflection image or an appearance-based method by machine learning may be used. The sight line information calculation unit 30 is connected to the recognizability calculation unit 50. Calculation unit for target object information

[0059] The target object information calculation unit 40 receives the vehicle exterior video B1 and the vehicle information I1 as inputs, detects a target object that needs to be visually recognized to perform appropriate driving from the received vehicle exterior video B1 and the vehicle information I1, and outputs the target object information D1 and the vehicle information I1, which are information regarding the detected target object.

[0060] Type information indicating the type of the target object can be used as target object information D1, such as a "stationary indicator" when the target object is a traffic signal or traffic light, and a "moving object" when the target object is a car or a pedestrian. When the target object is a stationary indicator, the target object information D1 can further include position information containing the position of the target object, size information indicating the detected size of the target object, and state information embodied by a light mode of a light-emitting diode (LED).

[0061] On the other hand, when the target object is a moving object, the target object information D1 may further include position information indicating the position of the target object, size information indicating the detected size of the target object, and time series change information (movement information) indicating the traveling direction and speed of the target object.

[0062] The target object information calculation unit 40 is connected to the detectability calculation unit 50 and the collision possibility calculation unit 60, outputs the target object information D1 to the detectability calculation unit 50, and outputs the target object information D1 and the vehicle information I1 to the collision possibility calculation unit 60. Recognizability calculation unit

[0063] The detectability calculation unit 50 receives the subject's sight line information C1 and target object information D1 as inputs, calculates a detectability E1 for each target using the received sight line information C1 and target object information D1, and outputs the calculated detectability E1. The detectability E1 is an index indicating the degree to which the subject visually detects the detected target object. The detectability calculation unit 50 calculates the detectability E1, for example, as follows.

[0064] First, the detectability calculation unit 50 calculates the degree of directional alignment between the vectors by using the sight line unit vector of the sight line vector and the target unit vector of the target vector from the vehicle toward the target object at any given time. The degree of directional alignment may be the degree of an angle formed by the sight line unit vector and the target unit vector, or the degree of an angle formed by two unit vectors in a displacement direction per unit time of those two unit vectors. The degree of directional alignment may be an index smoothed per a certain time.

[0065] Next, the detectability calculation unit 50 goes back a certain time from the present time toward the past and calculates the sum of the directional matching degrees, to which a weight is assigned according to each time. The weight may be set higher toward the present time, or a non-linear weight may be set.

[0066] In a case where a state change or a movement change of the target object is observed within a detectability measurement time, a correction coefficient is defined for each type of target object, the directional matching degree before the time when the change of the target object occurred is multiplied by the correction coefficient, and the sum of the corrected directional matching degrees within the measurement time is obtained, and the detectability is calculated.

[0067] In a case where the target object is a stationary indicator represented by a traffic signal, the correction coefficient can be specified by a constant multiple at the time the instruction content changes. If the target object is a moving object represented by an automobile, the difference between the movement at the start of the detectability measurement and the movement at the current time can be calculated, and the correction coefficient can be dynamically defined depending on the value of the difference. The detectability calculation unit 50 is connected to the effective collision possibility calculation unit 70. Collision possibility calculation unit

[0068] The collision possibility calculation unit 60 receives the target object information D1 and the vehicle information I1 as inputs, calculates the collision possibility F1 for each target object using the received target object information D1 and vehicle information I1, and outputs the calculated collision possibility F1. The collision possibility F1 is an index indicating the degree of possibility that the detected target object will collide with the device-mounted vehicle.

[0069] When calculating the collision probability F1, the calculation method can be varied depending on the type of target object specified by the type information. For example, in the case of a stationary indicator represented by a traffic signal, a time limit for a stop instruction position is measured from the position and movement information of the target object, and the collision probability F1 is defined as an index inversely proportional to the time limit.

[0070] The time limit can be multiplied by a correction coefficient that depends on the difference in the instruction form from "drive" to "stop." Examples of the instruction form of the target object include: the color display of a traffic signal, information about a road via a road information board (traffic congestion, accidents, weather, or the like), and the like.

[0071] Furthermore, as an example, the motion information is information regarding the speed, acceleration, and direction of travel of the target object. Even if the target object is a stationary object, such as a signal, the target object appears to be moving relatively when viewed by a camera mounted on the vehicle, and thus, motion information is present.

[0072] If the target object is a moving object, such as a car, the time limit until contact with the device-mounted vehicle is measured from the position and motion information of the target object, and the collision probability F1 is defined as an index inversely proportional to the time limit. Furthermore, in a case where the target object is a moving object, the kinetic energy can be calculated, and the collision probability F1 can be defined as an index proportional to the kinetic energy.

[0073] At the time of calculating the motion information, the motion information of the vehicle information I1 and the relative motion information of the target object information as viewed from the device-mounted vehicle may be combined and replaced with motion information of a real space, or a collision possibility based on future motion predictions of the device-mounted vehicle and the target object may be calculated from the replaced motion information.

[0074] In this way, the collision possibility calculation unit 60 selects at least one of the instruction form or the movement information of the target object, depending on the type of the target object, and calculates the collision possibility F1. The collision possibility calculation unit 60 is connected to the effective collision possibility calculation unit 70. Calculation unit for effective collision possibilities

[0075] The effective collision possibility calculation unit 70 receives the detectability E1 and the collision possibility F1 as inputs, calculates the effective collision possibility G1 using the received detectability E1 and collision possibility F1, and outputs the calculated effective collision possibility G1. The effective collision possibility G1 is an index that decreases according to an increase in the detectability E1 and increases according to an increase in the collision possibility F1. The detectability E1, which is such an index, is expressed, for example, by the above equation (1). The effective collision possibility calculation unit 70 is connected to the warning unit 80.

[0076] As described according to the first embodiment, by using such an effective collision possibility G1, various effects can be achieved such that the detectability E1 and the collision possibility F1 can be evaluated simultaneously as the effective collision possibility G1. warning unit

[0077] The warning unit 80 receives the effective collision possibility G1 as an input and outputs warning information H1, which is control information for controlling a warning operation by a warning device (not shown) based on the effective collision possibility G1. The warning information H1 is not output for a target for which the effective collision possibility G1 is lower than the threshold, and the warning information H1 is output when the effective collision possibility is equal to or greater than the threshold.

[0078] Depending on the strength of the effective collision possibility G1, the intensity and form of the warning can be changed. Furthermore, the warning form can be a visual presentation, an audio presentation, or another presentation method that appeals to the five human senses. In a case where the warning form is a visual presentation, a display (not shown) is used as the warning device. In a case where the warning mode is an audio presentation, a loudspeaker (not shown) is used as the warning device. Furthermore, a suitable device, such as a vibration device (not shown), is used as the warning device, depending on the warning presentation method. Fig. Fig. 4 is a diagram illustrating a concept of the effective collision possibility according to the first embodiment and the second embodiment of the present invention by means of a specific example. As the driving environment described in Fig. 4, it is assumed that target objects O1 to O3 as moving objects and a facility-mounted vehicle S1 travel in the directions of the arrows shown on the target objects or the vehicles.

[0079] It is assumed that the possibility of contact with the device-mounted vehicle S1 in the area of ​​the area V increases depending on the motion state of the device-mounted vehicle S1 and the motion state of the target object. Furthermore, it is assumed that the subject's line of sight direction changes from the target object O3 to O2.

[0080] Since the traveling direction of the target object O1 is in the direction away from the area V, the collision probability is calculated as low. On the other hand, the detectability is also calculated as low, the effective collision probability based on expression (1) is calculated as low, and no warning is presented.

[0081] The target object O2 is traveling in the direction the vehicle is passing, and the collision probability is moderate. However, the direction of travel changes en route, and the time until contact is shortened, thus increasing the collision probability. The subject visually recognizes the target object O2 at the current time and maintains a state of high detectability.

[0082] When the target object O2 is traveling straight ahead, the effective collision probability is low to medium, but the collision probability increases as the target object O2 changes direction, and the effective collision probability increases. As the effective collision probability increases, the warning transitions in the direction in which it is displayed, and the presentation intensity gradually increases.

[0083] Target object O3 is traveling outside of area V and has a low collision probability. The subject averts their eyes after visually detecting target object O3, which has a low collision probability. The effective collision probability is calculated as low at the stage of visual detection by the subject, and the warning is not presented. However, the movement of target object O3 changes as the subject averts their eyes, and they continue traveling toward area V.

[0084] The detectability gradually decreases due to the passage of time while the eyes remain averted and during the change in the target object's motion. The collision probability also decreases due to the change in the direction of the target object toward area V, so the effective collision probability increases, and the warning presentation is initiated. When the target object O3's direction of travel is a direction different from area V, the detectability is attenuated, but the collision probability is also reduced, and the effective collision probability is maintained at a low level.

[0085] Assume that the target object O4 is a traffic signal, and there are several stages of instruction forms from "go" to "stop." When the instruction form of the traffic signal changes, the degree of directional alignment between the sight line vector and the target vector obtained at a time before that time decreases instantaneously, and the detectability decreases.

[0086] The collision probability is calculated based on the distance to the stopping position and the vehicle speed. However, in a case where the collision probability is low, the effective collision probability increases moderately, and it is possible to gain attention with a reserve. On the other hand, if the collision probability is medium, the effective collision probability increases to a high degree, and a strong warning is presented. Third embodiment

[0087] A road-to-vehicle driving assistance system according to a third embodiment includes the driving assistance device 1 or the driving assistance system SYS-V1 described in the first embodiment, or the driving assistance device 2 or the driving assistance system SYS-V2 described in the second embodiment, and a roadside system SYS-R capable of communicating with the driving assistance device 1 or 2. The roadside system SYS-R includes a roadside device 101 and a communication device 102.

[0088] The roadside device 101 acquires information of the device-mounted vehicle S1 and target objects in the vicinity of the device-mounted vehicle S1 using, for example, a roadside monitoring camera or a satellite, and transmits the acquired information to the target object information calculation unit 40 of the driving assistance device 1 or 2 via the communication device 102.

[0089] The detection of a target object in the vicinity of the device-mounted vehicle S1 can also be performed only by the roadside device 101, or can be performed by both the roadside device 101 and the vehicle-side device or system (driver assistance device 1 or driver assistance system SYS-V1 / driver assistance device 2 or driver assistance system SYS-V2).

[0090] In addition, the roadside device 101 and the vehicle-mounted device or system (driver assistance device 1 or driver assistance system SYS-V1 / driver assistance device 2 or driver assistance system SYS-V2) can detect different target objects. Furthermore, the roadside device 101 can receive at least one of the vehicle speed, steering control information, or pedal control information from the device-mounted vehicle S1, calculate the collision possibility between the vehicle and the target object, and transmit the possibility to the collision possibility calculation unit 60. Hardware configuration

[0091] Next, a configuration example of hardware of the driving assistance devices 1 and 2 and the roadside device 101 according to the first to third embodiments will be described with reference to FIG. Fig. 6. These devices can be implemented by the control circuit shown in Fig. 6A, ie, a processor 81 and a memory 82. An example of the processor 81 is a central processing unit (CPU, also referred to as a processing device, arithmetic device, microprocessor, microcomputer processor, or digital signal processor (DSP)) or a large-scale integration system (LSI).

[0092] The memory 82 is, for example, a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM) or an electrically erasable programmable read-only memory (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, a digital versatile disk (DVD), or the like.

[0093] The driver assistance device 1 or 2 or the roadside device 101 is implemented by the processor 81, which reads and executes a program stored in the memory 82. The program executed by the processor 81 may be a file in an installable format or an executable format, stored on a computer-readable storage medium, and provided as a computer program product. Furthermore, the program executed by the processor 81 may be provided to the driver assistance device via a network, such as the Internet.

[0094] As another example, the driving assistance devices 1 and 2 and the roadside device 101 according to the first to third embodiments may be implemented by a dedicated processing circuit 83 provided in Fig.6B. Processing circuitry 83 is dedicated hardware. Processing circuitry 83 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

[0095] It should be noted that some of the functions of the driving assistance device 1 or 2 or the roadside device 101 may be implemented in software or firmware, and the remaining parts may be implemented in dedicated hardware. Additional remark

[0096] Some aspects of the various embodiments described above are summarized as follows. Additional note 1

[0097] A driving assistance device according to Supplementary Note 1 comprises: a sight line information calculation unit (30) for calculating sight line information of a subject from a vehicle interior video of the interior of a vehicle acquired by an imaging device having at least one camera; a target object information calculation unit (40) for detecting a target object located in the surroundings of the vehicle from a vehicle exterior video of the exterior of the vehicle acquired by the imaging device and extracting target object information that is information of the detected target object; a recognizability calculation unit (50) for calculating recognizability that indicates the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information;a collision possibility calculation unit (60) for calculating a collision possibility indicating the degree of a collision possibility between the detected target object and the vehicle from the extracted target object information, and an effective collision possibility calculation unit (70) for calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility.

[0098] With the driving assistance device according to Supplementary Note 1, a determination is made using the actual collision possibility, which simultaneously evaluates both detectability and collision possibility. Therefore, by performing a warning based on the actual collision possibility, it is possible to perform a warning that simultaneously evaluates both detectability and collision possibility. Furthermore, by performing a threshold determination of the actual collision possibility, it is sufficient to meet a lower detectability for a target object with a lower collision possibility, and thus it is possible to suppress an unnecessary warning that might disturb or annoy the user.

[0099] Since the effective collision possibility is defined as an index that reflects both detectability and collision possibility, it is possible to dynamically determine the presence or absence of a warning and adjust its intensity depending on detectability, even for a target with a low to medium collision possibility. Consequently, it is possible to present a warning with an appropriate intensity for a target object that truly requires attention or handling, while mitigating interference. Additional note 2

[0100] A driving assistance device according to Supplementary Note 2 is the driving assistance device according to Supplementary Note 1, further comprising a warning unit (80) for determining the content of a warning to be performed for the subject according to the calculated effective collision possibility, and for controlling a warning operation by means of a warning device.

[0101] With the driving assistance device according to Additional Note 2, it is possible to control the warning process for the subject according to the effective collision possibility. Additional note 3

[0102] A driving assistance device according to Supplementary Note 3 is a driving assistance device according to Supplementary Note 1 or 2, further comprising a vehicle information acquisition unit (90) for acquiring at least one of a vehicle speed, length control information, or pedal control information as vehicle information from a control system mounted on the vehicle, wherein the collision possibility calculation unit calculates the collision possibility based on the extracted target object information and the acquired vehicle information.

[0103] With the driving assistance device according to Supplementary Note 3, the movement of the target object in real space can be calculated by using the movement information of the device-mounted vehicle in addition to the relative movement of the surrounding target object as viewed from the device-mounted vehicle obtained from the vehicle exterior video, and the calculation accuracy of the collision possibility regarding the approach and contact between objects is improved from the movement prediction of the target object and the movement prediction of the device-mounted vehicle. Additional note 4

[0104] A driving assistance device according to Supplementary Note 4 is a driving assistance device according to any one of Supplementary Notes 1 to 3, wherein the recognizability calculation unit calculates a direction coincidence degree between a direction of a target vector from the vehicle toward the target object and a direction of a sight line vector of the subject at a predetermined time, calculates a weighted direction coincidence degree by adding a weight to a direction coincidence degree at any time within a past specified time from a present time when a change in the state or movement of the target object occurs within a measurement time, calculates a corrected direction coincidence degree that is corrected by multiplying the calculated,corrected directional mismatch degree before a time of occurrence of the change is multiplied by a coefficient, and the detectability is calculated by calculating the sum of the calculated weighted directional mismatch degrees or the calculated corrected directional mismatch degree.

[0105] With the driving assistance device according to Supplementary Note 4, since the detectability is obtained based on the weighted direction matching degree by adding the weight to the direction matching degree at any time within the past specified time from the present time, it is possible to reflect a temporary attenuation of a memory of the target object, which occurs when the subject deviates his / her line of sight from the target object, on the detectability.

[0106] Furthermore, it is possible to adjust the intensity of the attenuation over time by assigning a weight that increases as it approaches the current time. By using a coefficient related to a change in state or a change in movement of a target object in the environment, it is also possible to reflect on the detectability a condition in which the subject cannot detect a change in the target object's behavior after visually detecting the target object once. Additional note 5

[0107] A driving assistance device according to Supplementary Note 5 is a driving assistance device according to any one of Supplementary Notes 1 to 4, wherein the target object information calculation unit calculates type information indicating the type of the detected target object and state information indicating the state, or movement information indicating movement.

[0108] With the driving assistance device according to Additional Note 5, the following applies: Since the type information indicating the type of the detected target object is calculated, it is possible to calculate the detectability and the collision possibility by the index for each type of target object. Additional note 6

[0109] A driving assistance device according to Supplementary Note 6 is a driving assistance device according to Supplementary Note 5, wherein the collision possibility calculation unit calculates the collision possibility based on at least one of the calculated state information or movement information according to the calculated type information.

[0110] With the driving assistance device according to Additional Note 6, in a case where the target object is a stationary indicator such as a traffic signal, it is possible to calculate the collision possibility according to the illumination state of the LED and the distance to the stationary indicator, and in a case of a moving object such as an automobile, it is possible to calculate the collision possibility according to the movement information, and it is possible to calculate each collision possibility even for target objects of different types. Additional note 7

[0111] A road-to-vehicle driving assistance system according to Supplementary Note 7 is a road-to-vehicle driving assistance system comprising a driving assistance device (1; 2) according to any one of Supplementary Notes 1 to 6 and a roadside system (SYS-R) capable of communicating with the driving assistance device, wherein the roadside system transmits to the driving assistance device information of the target object or another target object that is in the vicinity of the vehicle and is different from the target object, and the target object information calculation unit of the driving assistance device extracts the target object information based on the vehicle exterior video and the transmitted information. Additional note 8

[0112] A driving assistance method according to Supplementary Note 8 is a driving assistance method performed by a driving assistance device comprising: a sight line information calculation unit (30), a target object information calculation unit (40), a detectability calculation unit (50), a collision possibility calculation unit (60), and an effective collision possibility calculation unit (70), the driving assistance method comprising: a step (ST3) of calculating, by means of the sight line information calculation unit, sight line information of a subject from a vehicle interior video of the interior of a vehicle acquired by an imaging device having at least one camera, a step (ST4) of detecting, by means of the target object information calculation unit, a target object located in the surroundings of the vehicle,from a vehicle exterior video of the exterior of the vehicle captured by the imaging device, and extracting target object information that is information of the detected target object, a step (ST5) of calculating, by means of the detectability calculation unit, a detectability indicating the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information, a step (ST6) of calculating, by means of the collision possibility calculation unit, a collision possibility indicating the degree of collision possibility between the detected target object and the vehicle from the extracted target object information, and a step (ST7) of calculating, by means of the effective collision possibility calculation unit, an effective collision possibility,which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility., Additional note 9

[0113] A driving assistance program according to Supplementary Note 9 comprises: a step (ST3) of calculating sight line information of a subject from a vehicle interior video of the interior of a vehicle acquired by an imaging device having at least one camera; a step (ST4) of detecting a target object located in the vicinity of the vehicle from a vehicle exterior video of the exterior of the vehicle acquired by the imaging device and extracting target object information that is information of the detected target object; a step (ST5) of calculating detectability that indicates the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information; a step (ST6) of calculating a collision possibility;which indicates the degree of a collision possibility between the detected target object and the vehicle, from the extracted target object information, and a step (ST7) for calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility.

[0114] It should be noted that the embodiments may be combined, and each of the embodiments may be appropriately modified or omitted. Industrial applicability

[0115] The driving assistance device of the present invention can be used as a device that assists driving a vehicle. List of reference symbols 1 driver assistance device, 2 driver assistance system, 10 imaging device, 20 data input unit, 30 Calculation unit for sight line information, 40 Calculation unit for target object information, 50 Recognizability calculation unit, 60 Collision possibility calculation unit, 70 Calculation unit for effective collision possibilities, 80 warning unit, 81 processor, 82 storage, 83 processing circuit, 90 vehicle information acquisition unit, 101 street-side facility, 102 communication device, SYS-R roadside system, SYS-V1 driver assistance system, SYS-V2 driver assistance system. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-166 542 A

[0004]

Claims

[1] Driver assistance device comprising: a sight line information calculation unit for calculating sight line information of a subject from a vehicle interior video about the interior of a vehicle acquired by an imaging device having at least one camera; a target object information calculation unit for detecting a target object located in the surroundings of the vehicle from a vehicle exterior video about the exterior of the vehicle acquired by the imaging device, and extracting target object information that is information of the detected target object; a detectability calculation unit for calculating a detectability indicating the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information; a collision possibility calculation unit for calculating a collision possibility indicating the extent of a collision possibility degree between the detected target object and the vehicle from the extracted target object information; and an effective collision possibility calculation unit for calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility. [2] A driver assistance device according to claim 1, further comprising: a warning unit for determining a content of a warning to be performed for the subject according to the calculated effective collision possibility, and for controlling a warning operation by means of a warning device. [3] Driver assistance device according to claim 1 or 2, further comprising: a vehicle information acquisition unit for acquiring at least one of a vehicle speed, steering control information, or pedal control information as vehicle information from a control system mounted on the vehicle, wherein the collision possibility calculation unit calculates the collision possibility based on the extracted target object information and the detected vehicle information. [4] The driving assistance device according to any one of claims 1 to 3, wherein the recognizability calculation unit calculates a directional matching degree between a direction of a target vector from the vehicle toward the target object and a direction of a sight line vector of the subject at a predetermined time, calculates a weighted directional matching degree by adding a weight to a directional matching degree at any time within a past specified time from the present time, when a change in the state or movement of the target object occurs within a measurement time, calculates a corrected directional matching degree, which is corrected by multiplying the calculated weighted directional matching degree before the occurrence time of the change by a coefficient, and calculates detectability by calculating a sum of the calculated weighted directional mismatch degrees or the calculated corrected directional mismatch degree. [5] The driving assistance device according to any one of claims 1 to 4, wherein the target object information calculation unit calculates type information indicating the type of the detected target object and state information indicating the state or movement information indicating movement. [6] The driving assistance device according to claim 5, wherein the collision possibility calculation unit calculates the collision possibility based on at least one of the calculated state information or the movement information according to the calculated type information. [7] Road-to-vehicle driver assistance system comprising: the driver assistance device according to one of claims 1 to 6; and a roadside system capable of communicating with the driver assistance system, whereby the roadside system transmits information about the target object or another target object that is in the vicinity of the vehicle and is different from the target object to the driver assistance device, and the target object information calculation unit of the driving assistance device extracts the target object information based on the vehicle exterior video and the transmitted information. [8] A driving assistance method performed by a driving assistance device comprising a sight line information calculation unit, a target object information calculation unit, a detectability calculation unit, a collision possibility calculation unit, and an effective collision possibility calculation unit, the driving assistance method comprising: Calculating, by means of the sight line information calculation unit, sight line information of a subject from a vehicle interior video about the interior of a vehicle acquired by an imaging device having at least one camera; Detecting, by means of the target object information calculation unit, a target object located in the surroundings of the vehicle from a vehicle exterior video about the exterior of the vehicle acquired by the imaging device, and extracting target object information that is information of the detected target object; Calculating, by means of the detectability calculation unit, a detectability indicating the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information; Calculating, by means of the collision possibility calculation unit, a collision possibility indicating the extent of a collision possibility degree between the detected target object and the vehicle from the extracted target object information; and Calculating, by means of the effective collision possibility calculation unit, an effective collision possibility that decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility. [9] Driver assistance program for causing a computer to perform the following: Calculating line-of-sight information of a subject from a vehicle interior video about the interior of a vehicle acquired by an imaging device having at least one camera; Detecting a target object located in the surroundings of the vehicle from a vehicle exterior video about the exterior of the vehicle acquired by the imaging device, and extracting target object information that is information of the detected target object; Calculating a detectability indicating the degree of visual recognition of the detected target object by the subject from the calculated sight line information and the extracted target object information; Calculating a collision possibility indicating the degree of collision possibility between the detected target object and the vehicle from the extracted target object information; and Calculating an effective collision possibility, which decreases according to an increase in the calculated detectability and increases according to an increase in the calculated collision possibility, from the calculated detectability and the calculated collision possibility.

Images