Driving assistance device, driving assistance method, and program

Abstract

A driving assistance device according to an embodiment of the present invention comprises: a recognition unit that recognizes the surrounding situation of a vehicle; an inference unit that infers a line-of-sight direction of a driver of the vehicle; and a determination unit that determines, on the basis of the line-of-sight direction, whether or not the driver has visually confirmed traffic participants around the vehicle who have been recognized by the recognition unit, wherein the determination unit determines that the driver has visually confirmed a traffic participant who is present within a prescribed distance from another traffic participant who is determined to have been visually confirmed by the driver among the traffic participants.

Description

Technical Field

[0001] The present invention relates to a driving support device, a driving support method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. In order to achieve this, research and development related to driving support technology has focused on further improving traffic safety and convenience through research and development. In this regard, conventionally, depending on the distance or angle from the driver's line of sight direction to an object, the driver's visibility of a safety confirmation object has been used to evoke safety confirmation or control the driving state of the vehicle, or based on the degree of overlap between a plurality of determination points in the area of a target and the field of view, it has been determined whether the driver recognizes the target, or pedestrian information such as whether the driver belongs to a group of pedestrians formed by a plurality of pedestrians received from a pedestrian communication device has been used to present pedestrian attention information (see, for example, Patent Documents 1 to 3).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in driving support technology, when there are a plurality of traffic participants around a vehicle, it is a problem that the processing load and processing time may be required to individually determine whether the driver has visually recognized each traffic participant.

[0005] One of the objectives of this invention is to provide a driver assistance device, a driver assistance method, and a program that can more efficiently determine whether or not a driver has visually identified surrounding traffic participants, in order to solve the above-mentioned problems. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The driver assistance device, driver assistance method, and program according to this invention employ the following configuration. (1) A driving assistance device according to one aspect of the present invention comprises a recognition unit that recognizes the surrounding conditions of a vehicle, an estimation unit that estimates the direction of the driver's line of sight of the vehicle, and a determination unit that determines whether or not the driver has seen a traffic participant in the vicinity of the vehicle that has been recognized by the recognition unit based on the direction of sight, wherein the determination unit also determines that another traffic participant located within a predetermined distance from a traffic participant that has been determined to have been seen by the driver has also been seen by the driver.

[0007] (2) In the embodiment of (1) above, the predetermined distance is different in the direction extending in the line of sight of the driver of the vehicle and in the direction perpendicular to the line of sight.

[0008] (3) In the embodiment of (2) above, the predetermined distance is longer the closer it is to the direction of extension and shorter the closer it is to the orthogonal direction.

[0009] (4) In the embodiment of (1) above, the estimation unit estimates the risk value of the traffic participant, and the determination unit determines that it has not recognized the other traffic participant if the risk value of the other traffic participant estimated by the estimation unit is greater than or equal to a threshold when the other traffic participant is within the predetermined distance.

[0010] (5) In the embodiment of (4) above, the estimation unit estimates the risk value according to the amount of travel or attributes of the traffic participant.

[0011] (6) The embodiment of (1) above further includes a notification unit that notifies the driver if there is a traffic participant that the driver has not seen.

[0012] (7): A driving assistance method according to one aspect of the present invention is a driving assistance method in which a computer recognizes the surrounding conditions of a vehicle, estimates the direction of the driver's line of sight of the vehicle, determines whether the driver has seen the recognized traffic participants around the vehicle based on the direction of sight, and determines that the driver has also seen other traffic participants that are within a predetermined distance from the traffic participant that the driver has determined to have seen.

[0013] (8) A program according to one aspect of the present invention causes a computer to recognize the surrounding conditions of a vehicle, to estimate the direction of the driver's line of sight of the vehicle, to determine whether the driver has seen the recognized traffic participants around the vehicle based on the direction of sight, and to determine that the driver has also seen other traffic participants that are within a predetermined distance from the traffic participant that the driver has determined to have seen. [Effects of the Invention]

[0014] According to the embodiments described in (1) to (8) above, it is possible to more efficiently determine whether or not the driver has seen the surrounding traffic participants. [Brief explanation of the drawing]

[0015] [Figure 1] This is a diagram showing the configuration of a vehicle system 1 utilizing a driver assistance device according to the first embodiment. [Figure 2] This is a diagram illustrating the content of the visual determination in the first embodiment. [Figure 3] This is a diagram illustrating how a predetermined range is adjusted. [Figure 4] This flowchart shows an example of processing by the driver assistance device 100 of the first embodiment. [Figure 5] This is a diagram for explaining the content of visual recognition determination in the second embodiment. [Figure 6] This is a diagram for explaining the adjustment of the fourth predetermined distance with respect to the line-of-sight direction SD2. [Figure 7] This is a diagram for explaining the estimation of risk values. [Figure 8] This is a flowchart showing an example of the processing by the driving support device 100 of the second embodiment.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the driving support device, driving support method, and program of the present invention will be described with reference to the drawings. In the following, the case where the left-hand traffic regulations are applied will be described, but in the case where the right-hand traffic regulations are applied, the left and right should be read in reverse.

[0017] (First Embodiment) [Overall Configuration] FIG. 1 is a configuration diagram of a vehicle system 1 using a driving support device according to the first embodiment. The vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using the generated electric power from a generator connected to the internal combustion engine, or the discharge electric power of a secondary battery or a fuel cell.

[0018] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection And Ranging) 14, a sonar 15, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a driver monitor camera 50, a driver control unit 80, a driver assistance device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. Camera 10, radar device 12, LiDAR 14, and sonar 15 are examples of "detection devices DD". The detection devices DD may also include the object recognition device 16. HMI 30 and the HMI control unit 160, which will be described later, are examples of "notification units".

[0019] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the area in front, camera 10 is mounted on the top of the front windshield or behind the rearview mirror, etc. When imaging the area behind vehicle M, camera 10 is mounted on the top of the rear windshield, etc. When imaging the right or left side of vehicle M, camera 10 is mounted on the right or left side of the vehicle body or door mirror, etc. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may be a stereo camera.

[0020] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to determine at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0021] LIDAR14 irradiates light (or electromagnetic waves with a wavelength close to light) around vehicle M and measures the scattered light. LIDAR14 detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. LIDAR14 can be attached to any location on vehicle M. LIDAR14 detects the distance from vehicle M to the target by scanning in the lateral and vertical directions relative to the direction of vehicle M's movement.

[0022] The sonar 15 emits ultrasonic waves around the vehicle M and detects the distance or position of an object by detecting reflection or scattering from an object within a predetermined distance from the vehicle M. The sonar 15 is installed, for example, at the front and rear ends of the vehicle M, on the bumper, etc.

[0023] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the detection devices DD (camera 10, radar device 12, LIDAR 14, and sonar 15) to recognize the position, type, speed, etc. of an object. The object recognition device 16 outputs the recognition results to the driver assistance device 100. The object recognition device 16 may output the detection results of the detection devices DD directly to the driver assistance device 100. Alternatively, the functions of the object recognition device 16 may be incorporated into the driver assistance device 100, and the object recognition device 16 may be omitted from the vehicle system 1.

[0024] The communication device 20 communicates with other vehicles in the vicinity of vehicle M, the terminal device of the driver using vehicle M, or various server devices, for example, by utilizing networks such as cellular networks, Wi-Fi networks, Bluetooth®, DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0025] The HMI 30 presents various information to the vehicle occupants and accepts input operations from the occupants. The HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 may be, for example, a meter display unit in front of the driver's seat, a display device located in the center of the instrument panel, or a head-up display (HUD). The speaker 34 may be, for example, an audio output device located inside the vehicle M. In addition to the display unit 32 and speaker 34, the HMI 30 may also include a buzzer, touch panel, switches, keys, microphone, etc.

[0026] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting yaw rate (for example, the angular velocity of rotation around the vertical axis passing through the center of gravity of the vehicle M), and an orientation sensor for detecting the orientation of the vehicle M. The vehicle sensor 40 may also be provided with a position sensor for detecting the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver.

[0027] The driver monitoring camera 50 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The driver monitoring camera 50 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front (in a direction that captures the face). For example, the driver monitoring camera 50 can be mounted on the instrument panel of the vehicle M, the top of the front windshield, the rearview mirror, the steering wheel, etc. The driver monitoring camera 50 periodically and repeatedly captures images that include the driver.

[0028] The driver control elements 80 include, for example, a steering wheel, as well as an accelerator pedal, brake pedal, shift lever, and other controls. The driver control elements 80 are equipped with sensors that detect the amount of operation or whether or not an operation is performed, and the detection results are output to the driver assistance device 100, or to some or all of the driving force output device 200, brake device 210, and steering device 220. The steering wheel is an example of a "control element that accepts steering operations by the driver." The control element does not necessarily have to be ring-shaped and may take the form of an irregularly shaped steering wheel, a joystick, a button, etc. The driver control elements 80 also output to the driver assistance device 100 information such as the steering angle and steering torque when the driver of the vehicle M steers the steering wheel in a predetermined direction, and information indicating that the driver has detected that they are gripping the steering wheel.

[0029] The driver assistance device 100 is a device that assists the driver in driving the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 110, an estimation unit 120, a prediction unit 130, a determination unit 140, a driving control unit 150, an HMI control unit 160, and a storage unit 180. The recognition unit 110, estimation unit 120, prediction unit 130, determination unit 140, driving control unit 150, and HMI control unit 160 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as the HDD or flash memory of the driver assistance device 100 (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driver assistance device 100 when the storage medium (non-transient storage medium) is mounted on the drive device.

[0030] The storage unit 180 may be implemented by the above-mentioned various storage devices, or by EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 180 stores, for example, information, programs, and other various information necessary to perform various controls in the embodiment. The storage unit 180 may also include map information 182. The map information 182 is, for example, information in which the road shape is represented by links indicating roads in a predetermined section and nodes connected by links. The map information 182 may also include POI (Point of Interest) information depending on the location information, and may also include information about road shapes and road structures. The road shape includes, for example, branches and merges, tunnels (entrances and exits), curved roads (entrances and exits), curvature of roads or road markings (hereinafter referred to as "markings"), radius of curvature, number of lanes, width, gradient, etc. Information regarding road structures may include, for example, the type, location, orientation relative to the road's extension direction, size, shape, and color of the road structure. In the classification of road structures, for example, lane markings may be classified as one type, while lane markers, curbs, median strips, and walls (including fences, etc.) installed along the road's extension direction may each be classified as a different type. Map information 182 may be updated as needed by the communication device 20 communicating with other devices.

[0031] The recognition unit 110 recognizes the surrounding conditions of the vehicle M based on information input from some or all of the detection devices DD, or information input via the object recognition device 16. For example, the recognition unit 110 recognizes the position, size, shape, speed, acceleration, and other states of objects present around the vehicle M (within a first predetermined distance). The position of an object may be recognized as a position in an absolute coordinate system (vehicle coordinate system) with a representative point of the vehicle M (such as the center of gravity or the center of the drive axis) as the origin. The position of an object may be represented by a representative point such as the center of gravity, a corner, or the leading edge in the direction of travel, or by a represented region. The "state" of an object may include, for example, the acceleration or jerk of the object, or its "action state" (for example, whether or not it is changing lanes or attempting to change lanes), if the object is a moving object such as another vehicle.

[0032] Furthermore, the recognition unit 110 recognizes the attributes of an object by matching its characteristic information, such as its size, shape, color, and speed, with attribute patterns that have been pre-associated with the characteristic information. The attributes of an object may be, for example, its type, such as a vehicle, pedestrian, or utility pole, or it may be a type used to identify four-wheeled vehicles, two-wheeled vehicles, bicycles, electric scooters, etc., for each type of vehicle. In addition, when the recognition unit 110 recognizes a pedestrian, it may recognize whether the pedestrian is an adult, a child, or an elderly person, etc., as separate attributes based on its size (for example, height), shape, and amount of movement.

[0033] Furthermore, the recognition unit 110 may recognize objects that are traffic participants based on their attributes, etc. Traffic participants are not limited to, for example, pedestrians present on the road on which vehicle M is traveling, but also include other moving objects such as bicycles, motorcycles, and automobiles. In addition, if a predetermined number or more traffic participants with a specific attribute are present within a predetermined range, the recognition unit 110 may cluster the traffic participants with that specific attribute present within the predetermined range into a single group.

[0034] Furthermore, the recognition unit 110 recognizes the road conditions around the vehicle M based on information input from some or all of the detection devices DD, or information input via the object recognition device 16. Road conditions include the location of road markings, the location of road boundaries, and the condition of the area from the road markings to the road boundaries (e.g., roadside, shoulder). The recognition unit 110 may also recognize the distance between the recognized object and the road markings, or the distance between the object and the road boundary. The recognition unit 110 may also recognize the vehicle M's lane of travel and adjacent lanes.

[0035] Furthermore, the recognition unit 110 may refer to the map information 182 stored in the storage unit 180 based on the location information of the vehicle M obtained from the position sensor of the vehicle sensor 40, and recognize the road conditions (for example, the position of lane markings) on which the vehicle M is traveling from the map information 182. Alternatively, the recognition unit 110 may recognize the road conditions around the vehicle M by comparing the recognition results from the detection device DD, etc., with the recognition results obtained from the map information 182.

[0036] The estimation unit 120 detects the positional relationship between the driver's head and eyes, the combination of a reference point and a moving point in the eyes, etc., from the image of the driver monitor camera 50, for example, using a method such as template matching. The estimation unit 120 then estimates the orientation of the face based on the relative position of the eyes to the head. The estimation unit 120 also estimates the direction of the driver's gaze based on the position of the moving point relative to the reference point. For example, if the reference point is the inner corner of the eye, the moving point will be the iris. If the reference point is the corneal reflection area, the moving point will be the pupil. The estimation unit 120 may also estimate the point of fixation based on the direction of gaze. The estimation unit 120 may also estimate the driver's field of view range (a predetermined range around the direction of gaze) based on the direction of gaze. The field of view range is, for example, the range that extends in the direction of gaze with the position of the driver's eyes in the vehicle M (or the midpoint between both eyes), or the center of the driver's face or head as the vertex, and widens at a predetermined angle as it moves away from the driver.

[0037] The prediction unit 130 predicts the future behavior (e.g., position, speed, direction of movement) of the traffic participant recognized by the recognition unit 110 (after a first predetermined time has elapsed). For example, the prediction unit 130 may predict future behavior based on the amount of movement (e.g., speed) and direction of movement of the same traffic participant up to the present, or it may predict future behavior based on a predetermined behavior pattern according to the attributes of the traffic participant and the road shape. A behavior pattern is, for example, a pattern in which a person with the attribute of being a cyclist is likely to continue moving in the direction they have been moving so far, or a pattern in which a traffic participant approaching an intersection will slow down.

[0038] Furthermore, if the recognition unit 110 recognizes multiple traffic participants as a single group, the prediction unit 130 predicts future behavior on a group basis. In this case, the prediction unit 130 predicts the future position of the group based, for example, on the average speed and average position of the traffic participants included in the group. The prediction unit 130 may also give greater weight to the speed and position of traffic participants that are closer to vehicle M among the multiple traffic participants included in a single group. This allows for a higher priority to the speed and position of traffic participants closer to vehicle M when predicting the group's future speed and position, enabling more appropriate visual recognition and driver notification. The prediction unit 130 may also predict the future position and speed of vehicle M based on its position and speed VM.

[0039] The determination unit 140 determines whether the driver of vehicle M, as recognized by the recognition unit 110, has seen the surrounding traffic participants based on the driver's line of sight direction estimated by the estimation unit 120. Furthermore, if the recognition unit 110 recognizes multiple traffic participants as a single group, the determination unit 140 determines whether the traffic participants have been seen based on the group. Details of the processing performed by the determination unit 140 will be described later.

[0040] The driving control unit 150 automatically controls either the steering or acceleration / decision of the vehicle M, or both, based on the recognition results from the recognition unit 110 and the determination results from the determination unit 140, thereby executing driving control. In addition, if the driving control unit 150 receives an operation from the HMI 30 to perform at least one of the various driving assistance functions, it may perform the driving assistance function corresponding to the received operation. The driving assistance functions performed by the driving control unit 150 may include various driving controls such as ACC (Adaptive Cruise Control), LKAS (Lane Keeping Assistance System), LCA (Lane Change Assist), FCW (Forward Collision Warning), and CMBS (Collision Mitigation Braking System).

[0041] The HMI control unit 160 acquires information received by the HMI 30 and causes the HMI 30 to output predetermined information to notify the occupants of the vehicle M (driver, etc.). The predetermined information includes, for example, the recognition result by the recognition unit 110 and the determination result by the determination unit 140. The predetermined information may also include information related to the driving of the vehicle M, such as information about the status of the vehicle M and driving support information. Information about the status of the vehicle M includes, for example, information such as the speed of the vehicle M, engine speed, and shift position. Driving support information includes, for example, information to let the driver know that there are objects (traffic participants, etc.) around the vehicle M (warning information) and information to support steering and speed control operations by the driver to avoid contact between the vehicle M and objects. The predetermined information may also include information unrelated to the driving control of the vehicle M, such as television programs and other content (e.g., audio, video) received by the communication device 20, etc. The predetermined information may also include, for example, information about the current location and destination of the vehicle M and the remaining fuel level.

[0042] For example, the HMI control unit 160 may generate an image containing the predetermined information described above and display the generated image on the display unit 32 of the HMI 30, or it may generate sound (including alarm sounds, etc.) indicating the predetermined information and output the generated sound from the speaker 34 of the HMI 30.

[0043] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle M to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission mounted on the vehicle M, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 150 or information input from the accelerator pedal of the driving control device 80.

[0044] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 150 or from the brake pedal of the driving control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 150 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0045] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 150 or from the steering wheel of the driving control unit 80.

[0046] [Determination of the visibility of traffic participants in the first embodiment] Next, the visual recognition determination of traffic participants in the first embodiment will be explained using a diagram. Figure 2 is a diagram for explaining the content of the visual recognition determination in the first embodiment. In the example in Figure 2, it is assumed that the vehicle M driven by driver P1 is traveling along lane L1 at speed VM in the direction of extension (X-axis direction in the figure) (hereinafter, lane L1 may be referred to as "driving lane L1"). Driving lane L1 is demarcated by lane markings LL and RL. Also, in the example in Figure 2, it is assumed that there are traffic participants OB1 to OB7 around vehicle M. The attributes of traffic participants OB1 to OB5 are pedestrians, and the attributes of traffic participants OB6 to OB7 are bicycles. It is assumed that traffic participants OB1 to OB7 are moving in the direction of the arrows shown in Figure 2 at speeds VOB1 to VOB7, respectively.

[0047] In the example shown in Figure 2, the recognition unit 110 recognizes the position and speed VM of vehicle M relative to the driving lane L1. The recognition unit 110 also recognizes the positions and speeds VOB1 to VOB7 of traffic participants OB1 to OB7 that are in the vicinity of vehicle M (for example, in the direction of travel (forward) of vehicle M and within a first predetermined distance). The recognition unit 110 also recognizes the attributes of traffic participants OB1 to OB7.

[0048] Furthermore, the recognition unit 110 recognizes (clusters) each of the recognized traffic participants OB1 to OB7 as a single group if a predetermined number or more traffic participants with specific attributes exist within a predetermined range. Specific attributes include, for example, attributes of the same type. Alternatively, specific attributes may include attributes where the amount of movement during a predetermined time (second predetermined time) is small (speed is below a threshold) and there is a high probability of approaching vehicle M (e.g., pedestrian). In addition to the above conditions, specific attributes may also include, for example, that the error in the direction of travel of the traffic participants is below a threshold, or that the increase in the relative position between traffic participants during a predetermined time (third predetermined time) is less than a predetermined amount. The above-mentioned amount of movement, direction of travel, and increase in relative position may include future behavior prediction results predicted by the prediction unit 130. The predetermined number may be a fixed value (e.g., 2), or it may be set variably according to the total number of traffic participants recognized by the recognition unit 110, road conditions, the speed of vehicle M or traffic participants, the distance between vehicle M and traffic participants, etc.

[0049] In the example in Figure 2, the recognition unit 110 recognizes traffic participants OB1 to OB3, who are all pedestrians and located within a predetermined range, as a single group GR1. The recognition unit 110 also recognizes traffic participants who are all cyclists and located within a predetermined range as a single group GR2. In the example in Figure 2, traffic participants OB5 and OB6 are located within the predetermined range, but are not recognized as a single group because they have different attributes.

[0050] Furthermore, in the example in Figure 2, traffic participants OB4 and OB5 have the same attribute (pedestrian) and are within a predetermined range, but their directions of travel are opposite (the error in their directions of travel is greater than or equal to a threshold (first threshold)), and there is a high probability that their respective positions will be outside the predetermined range in the near future. For this reason, the recognition unit 110 may choose not to recognize them as a single group if, for example, the positions of traffic participants OB4 and OB5 in the future (first predetermined time) predicted by the prediction unit 130 will be outside the predetermined range. In the following explanation, traffic participants OB4 and OB5 will not be recognized as a single group by the recognition unit (each will be recognized as an individual traffic participant).

[0051] The determination unit 140 determines, for example, whether the driver P1 of the vehicle M has seen the traffic participants OB1 to OB7, based on the direction of the driver P1's gaze and the positions of the traffic participants OB1 to OB7 recognized by the recognition unit 110. For example, the determination unit 140 projects the direction of the driver P1's gaze SD1 onto the image plane of the image captured by the camera 10 (hereinafter referred to as the camera image), based on the position of the vehicle M, the position of the driver P1 inside the vehicle, the position of the camera 10 mounted on the vehicle M, etc., and determines that the driver P1 has seen the traffic participant if a traffic participant is present in the projected direction of gaze SD1. The presence of a traffic participant in the projected direction of gaze SD1 means, for example, that at least a part of the point of fixation or field of view on the image plane based on the projected direction of gaze SD1 is included in (overlaps with) the area of ​​the traffic participant on the image plane. The determination unit 140 also determines that the driver has not seen the traffic participant if there is no traffic participant in the projected direction of gaze SD1.

[0052] Furthermore, the determination unit 140 may determine that a traffic participant has been seen if the presence of a traffic participant in the line of sight SD1 is continuous for a predetermined time (fourth predetermined time) or longer. This can prevent the driver P1 from mistakenly determining that they have seen a traffic participant that they may have missed.

[0053] Furthermore, if the recognition unit 110 recognizes a predetermined number of traffic participants as a group, the determination unit 140 determines that driver P1 has seen all traffic participants in the group if at least some of the traffic participants in the group are present in the driver P1's line of sight direction SD1. Conversely, the determination unit 140 determines that driver P1 has not seen all traffic participants in the group if at least some of the traffic participants in the group are not present in the driver P1's line of sight direction SD1.

[0054] In the example in Figure 2, the driver P1 of vehicle M has a line of sight SD1 (point of gaze VP1) where traffic participant OB2 is located, out of the three traffic participants OB1 to OB3 included in group GR1. Therefore, the determination unit 140 determines that driver P1 has seen all of traffic participants OB1 to OB3 included in group GR1. Also, in the example in Figure 2, the determination unit 140 determines that driver P1 has not seen traffic participants OB4 to OB7.

[0055] The HMI control unit 160 notifies driver P1 by outputting information such as images and audio to the HMI 30 if there are any traffic participants OB4 to OB7 that are determined not to be visible to driver P1 and whose relative distance from vehicle M is less than a predetermined distance (second predetermined distance), or if there are traffic participants whose contact margin time (TTC) with vehicle M is less than a threshold. This information includes the presence of traffic participants around vehicle M, the direction in which the traffic participants are located relative to vehicle M, the attributes of the traffic participants, and information to encourage caution. The contact margin time (TTC) is a value calculated, for example, by dividing the relative speed by the relative distance between vehicle M and the traffic participant.

[0056] Furthermore, if the recognition unit 110 recognizes multiple traffic participants as a single group, the HMI control unit 160 may make the above-mentioned notification determination based on the group's position and speed. For example, in the example shown in Figure 2, the above-mentioned notification determination is performed based on the position (e.g., the average position of traffic participants OB6 and OB7) and speed (e.g., the average speed of speeds VOB6 and VOB7) of group GR2 to which traffic participants OB6 and OB7 belong.

[0057] In addition to (or instead of) the above notification, the HMI control unit 160 may also notify the vehicle M of information regarding driving operations to avoid contact with traffic participants. The HMI control unit 160 also terminates the notification if it determines that the driver P1 has seen the traffic participant after the notification. The determination unit 140 may also determine whether the driver will see the traffic participant in the near future based on the future position of the traffic participant (or group of people) predicted by the prediction unit 130 and the direction of the driver P1's line of sight. If it is determined that the driver will see the traffic participant in the near future, the HMI control unit 160 may control not to notify the traffic participant.

[0058] Furthermore, if there is a traffic participant whose relative distance to vehicle M is less than a second predetermined distance, or if there is a traffic participant whose contact margin time (TTC) with vehicle M is less than a threshold (second threshold), the driving control unit 150 may control one or both of the steering and / or speed of vehicle M to avoid contact between vehicle M and the traffic participant (or group). In this case, the driving control unit 150 may also perform driving control to avoid contact based on the prediction results of the future behavior of the traffic participant (or group) predicted by the prediction unit 130.

[0059] In this way, when a predetermined number of traffic participants with specific attributes are present within a predetermined range around vehicle M, they are recognized as a single group, and visibility is determined on a crowd-by-crowd basis. This eliminates the need to individually determine visibility for each traffic participant predicted to be within the field of view, resulting in more efficient visibility determination. Furthermore, the processing load for visibility determination is reduced, and processing time is shortened, allowing information about traffic participants that driver P1 may have overlooked to be notified to driver P1 at a more appropriate time.

[0060] The predetermined range described above in the first embodiment may be adjusted, for example, according to the distance from the vehicle M to a traffic participant with specific attributes (a traffic participant whose inclusion in a group is to be determined). Figure 3 is a diagram illustrating how the predetermined range is adjusted. In the example in Figure 3, the vehicle M and traffic participants OB1 to OB3 are shown on the lane L1 shown in Figure 2. The field of view widens at a predetermined angle as the distance from the driver P1 increases, so conversely, the field of view narrows as the distance to the gaze point VP1 decreases. Therefore, the recognition unit 110 narrows the range (predetermined range) recognized as a single group as the distance to the traffic participant decreases. Alternatively, the predetermined range recognized as a single group may be widened as the distance to the traffic participant increases.

[0061] For example, in the example in Figure 3, the distance D2 between vehicle M and the traffic participant OB2 closest to vehicle M, in the direction of extension of the driving lane L1, is shorter than the distance D1 shown in Figure 2. In other words, the distance from driver P1 to traffic participant OB2 is shorter than in the situation shown in Figure 2. Therefore, the recognition unit 110 reduces the predetermined area recognized as a single group, recognizes traffic participants OB2 and OB3 as a single group GR1#, and excludes traffic participant OB1 from group GR1#. This makes it possible to recognize multiple traffic participants as a group within a range corresponding to the field of view, and further suppresses misjudgments of visual recognition.

[0062] Furthermore, the recognition unit 110 may reduce the number of traffic participants included in a group the closer they are to the vehicle M, and may also choose not to recognize them as a group if the distance from the vehicle M to the traffic participants is shorter than a predetermined distance (third predetermined distance). As a result, the closer a group is to the vehicle M, the smaller the number of people or the less they are recognized as a group, and individual traffic participants are targeted for visual recognition. This improves the accuracy of visual recognition for traffic participants closer to the vehicle M, enabling appropriate driving assistance through notifications and the like.

[0063] [Processing flow] The following describes the sequence of processes performed by the driver assistance device 100 of the first embodiment using a flowchart. Figure 4 is a flowchart showing an example of the processes performed by the driver assistance device 100 of the first embodiment. In the following description, the processes performed by the driver assistance device 100 will be mainly described in focus on the visual judgment process in the first embodiment. The processes in this flowchart may be executed repeatedly, for example, at predetermined timings or cycles.

[0064] In the example in Figure 4, the recognition unit 110 recognizes the surrounding conditions of vehicle M (step S100). Next, the determination unit 140 determines whether or not there are traffic participants in the direction of travel of vehicle M (for example, forward) (step S110). If it is determined that there are traffic participants, the determination unit 140 determines whether or not there are a predetermined number or more traffic participants of a specific attribute within a predetermined range (step S130). If it is determined that there are a predetermined number or more traffic participants of a specific attribute within a predetermined range, the recognition unit 110 recognizes the predetermined number of traffic participants as a group (step S140). Also, if in the process of step S130 it is determined that there are not a predetermined number or more traffic participants of a specific attribute within a predetermined range, the recognition unit 110 recognizes the recognized traffic participants as individual traffic participants (step S150).

[0065] After processing in step S140 or S150, the estimation unit 120 estimates the direction of the driver's gaze in vehicle M (step S160). Next, the determination unit 140 determines whether or not the driver has seen a traffic participant based on the driver's gaze direction and the traffic participant recognized by the recognition unit 110 (step S170). In the process of step S170, if the recognition unit 110 recognizes a predetermined number (multiple) traffic participants as a group, the determination unit 140 determines whether or not at least some (one person) of the traffic participants included in the group has been seen. If it is determined that no one has been seen, the HMI control unit 160 generates an image or sound to notify the driver of the presence of the traffic participant and outputs it to the HMI 30 to notify the driver (step S180). This completes the processing of this flowchart.

[0066] Furthermore, if it is determined in step S110 that there are no traffic participants, or if it is determined in step S170 that traffic participants have been sighted, the process of this flowchart ends.

[0067] According to the first embodiment described above, the driver assistance device 100 includes a recognition unit 110 that recognizes the surrounding conditions of the vehicle M, an estimation unit 120 that estimates the direction of the driver's gaze of the vehicle M, and a determination unit 140 that determines whether the driver has seen the traffic participants around the vehicle M recognized by the recognition unit 110 based on the direction of the gaze. The recognition unit 110 recognizes the traffic participants within a predetermined range as a group if a predetermined number or more of traffic participants with specific attributes are within a predetermined range, and the determination unit 140 determines that the driver has seen the traffic participants included in the group if at least some of the traffic participants included in the group are in the direction of the driver's gaze, thereby enabling a more efficient determination of whether the driver has seen the surrounding traffic participants.

[0068] For example, according to the first embodiment, there is no need to perform visual recognition for each traffic participant, and furthermore, multiple traffic participants can be clustered as a group without communicating with the traffic participants. This reduces the processing load related to visual recognition and shortens the processing time. In addition, according to the first embodiment, the presence of traffic participants in the surrounding area can be notified to the vehicle driver more appropriately.

[0069] (Second embodiment) Next, a second embodiment of the driver assistance device 100 will be described. In the second embodiment, the same configuration as the vehicle system 1 (driver assistance device 100) shown in Figure 1 can be applied. Therefore, in the second embodiment, the same configuration as the vehicle system 1 shown in Figure 1 will be used, and the explanation will mainly focus on the differences from the first embodiment, with other explanations omitted.

[0070] [Determination of traffic participants' visual presence in the second embodiment] In the second embodiment, if the determination unit 140 determines that there is a traffic participant among the traffic participants recognized by the recognition unit 110 that the driver has seen, it also determines that another traffic participant located within a predetermined distance (fourth predetermined distance) from that traffic participant has been seen by the driver. Figure 5 is a diagram illustrating the content of the visibility determination in the second embodiment. In the example in Figure 5, traffic participants OB11 to OB13 are located in front of a vehicle M traveling at speed VM along the extension direction of lane L1, driven by driver P1. Traffic participants OB11 to OB13 are moving in the direction of the arrows shown in Figure 5, each at speeds VOB11 to VOB13.

[0071] As shown in the example in Figure 5, if a traffic participant OB13 is located in the line of sight SD2 of the driver P1 of vehicle M, the determination unit 140 determines that the traffic participant OB13 has been sighted. In this case, if another traffic participant is located within a fourth predetermined distance from the position of traffic participant OB13 (for example, the gaze point VP2 projected onto the screen plane of the camera image or the center position of the image area of ​​traffic participant OB13), the determination unit 140 determines that that other traffic participant has also been sighted. In the example in Figure 5, region AR1 is shown within a fourth predetermined distance from the position of traffic participant OB13. The determination unit 140 determines that other traffic participants OB11 to OB12 located within region AR1 have also been sighted. This eliminates the need to determine whether each traffic participant has been sighted, allowing for efficient sight determination, reducing processing load, and shortening processing time.

[0072] The fourth predetermined distance may be adjusted according to the driver P1's line of sight direction SD2, etc. Figure 6 is a diagram illustrating the adjustment of the fourth predetermined distance with respect to the line of sight direction SD2. For example, the fourth predetermined distance may differ in the direction extending in the driver P1's line of sight direction SD2 and in the direction perpendicular to the line of sight direction SD2. For example, objects near the extension direction of the line of sight direction SD2 (point of gaze VP2) are more visible, and the further away from the line of sight direction SD2 (the larger the field of view angle), the lower the visibility of the object. Therefore, the fourth predetermined distance is adjusted so that the direction from the reference position (for example, the position of the seen traffic participant OB13) is closer to the direction extending in the line of sight direction SD2, and shorter when it is closer to the direction perpendicular to the line of sight direction SD.

[0073] In the example shown in Figure 6, the fourth predetermined distance adjusted region AR1 is shown as an ellipse with major axis A1 and minor axis A2 centered on the position of traffic participant OB13 located in the line of sight direction SD2. However, it is not limited to this, and for example, the distance from the front to the rear of traffic participant OB3 or the distance from the left to right may be varied based on the road shape and the position and speed VOB13 of the spotted traffic participant OB13 on the road. Furthermore, the outer perimeter of region AR1 may be formed as a non-curved shape instead of a curved shape. This makes it possible to determine that other traffic participants located in positions where the driver P1 is expected to have a higher degree of visibility have been spotted together.

[0074] Furthermore, in the second embodiment, if the determination unit 140 determines that it has seen one of the multiple traffic participants, it also determines that it has seen the traffic participants in the vicinity (within the fourth predetermined distance). However, if the risk value of another traffic participant located within the fourth predetermined distance is above a threshold, it may determine that it has not seen that other traffic participant. The risk value is, for example, an index value indicating the likelihood of contact with vehicle M based on the amount of movement of the traffic participant and the behavior prediction results. In the second embodiment, the risk value is estimated by the estimation unit 120.

[0075] Figure 7 is a diagram illustrating the estimation of risk values. In the example in Figure 7, traffic participants OB11 to OB14 are located within the driving lane L1 of vehicle M, in the direction of travel (forward) of vehicle M. In addition, in the example in Figure 7, it is assumed that other traffic participants OB11, OB12, and OB14 are located within a fourth predetermined distance area AR1, based on traffic participant OB13, which is located in the driver P1's line of sight SD2.

[0076] In this case, the estimation unit 120 estimates the risk values ​​for each of the other traffic participants OB11, OB12, and OB14. For example, the estimation unit 120 estimates the risk value based on the amount of travel, such as the speed VOB of the other traffic participants. In this case, the estimation unit 120 increases the risk value as the amount of travel increases. The estimation unit 120 may also estimate the risk value by including not only the current speed (amount of travel) of the traffic participants but also the future amount of travel based on the behavior prediction results of the prediction unit 130.

[0077] Furthermore, the estimation unit 120 may estimate risk values ​​according to the attributes of traffic participants. In this case, the estimation unit 120 will set a higher risk value if the traffic participant is a child compared to if they are an adult. Also, the estimation unit 120 will set a higher risk value if the traffic participant is a cyclist compared to if they are a pedestrian. The estimation unit 120 may also set a higher risk value the closer the traffic participant is to the center of the vehicle M's lane L1. Furthermore, the estimation unit 120 may increase the risk value of traffic participants with a high probability of contact based on the predicted future position of the vehicle M predicted by the prediction unit 130 and the predicted future behavior of the traffic participants. The estimation unit 120 may also estimate the final risk value by combining some of the above-described conditions for setting risk values.

[0078] In the example in Figure 7, traffic participant OB14 is a child and is located near the center of the lane, so its risk value is above the threshold, while the risk values ​​of traffic participants OB11 and OB12 are below the threshold. In this case, in the example in Figure 7, if the determination unit 140 determines that driver P1 has seen traffic participant OB13, it also determines that traffic participants OB11 and OB12, which are located within a fourth predetermined distance (within area AR1) from traffic participant OB13, have also been seen, and that traffic participant OB14 has not been seen.

[0079] The HMI control unit 160 notifies the driver P1 by outputting information (image or sound) indicating the presence of a traffic participant to the HMI 30 if the relative distance between the traffic participant and the vehicle M, which the driver P1 has determined not to have seen, is within a predetermined distance (fifth predetermined distance), or if the contact margin time (TTC) for that traffic participant is less than a threshold (third threshold).

[0080] Furthermore, if there is a traffic participant whose relative distance to vehicle M is less than the fifth predetermined distance, or if there is a traffic participant whose contact margin time (TTC) with vehicle M is less than the third threshold, the driving control unit 150 may control either or both of the steering and / or speed of vehicle M to perform driving control to avoid contact between vehicle M and the traffic participant (or group).

[0081] [Processing flow] The following describes a series of processes performed by the driver assistance device 100 of the second embodiment using a flowchart. Figure 8 is a flowchart showing an example of the processes performed by the driver assistance device 100 of the second embodiment. In the following description, we will mainly focus on the visual judgment process in the second embodiment. The processes in this flowchart may be executed repeatedly, for example, at predetermined timings or cycles.

[0082] In the example in Figure 8, the recognition unit 110 recognizes the surrounding conditions of vehicle M (step S200). Next, the determination unit 140 determines whether or not there are traffic participants in the direction of travel of vehicle M (for example, forward) (step S210). If it is determined that there are traffic participants in the direction of travel of vehicle M, the estimation unit 120 estimates the direction of view of the driver of vehicle M (step S220). Next, the determination unit 140 determines whether or not the traffic participant has been seen based on the estimated direction of view of the driver and the position of the traffic participant (step S230). If it is determined that the traffic participant has been seen, the determination unit 140 determines whether or not there are other traffic participants within a predetermined distance (fourth predetermined distance) from the traffic participant (step S240). If it is determined that there are other traffic participants, it is determined that the driver has also recognized the other traffic participant (step S250).

[0083] Furthermore, if it is determined in step S230 that no traffic participants are visible, the HMI control unit 160 outputs information to the HMI 30 indicating the presence of traffic participants to notify the driver (step S260). This completes the processing of this flowchart. Also, if it is determined in step S210 that no traffic participants exist, or in step S240 that there are no other traffic participants within the fourth predetermined distance from the traffic participant, the processing of this flowchart also completes.

[0084] According to the second embodiment described above, the driver assistance device 100 includes a recognition unit 110 that recognizes the surrounding conditions of the vehicle M, an estimation unit 120 that estimates the direction of the driver's line of sight of the vehicle M, and a determination unit 140 that determines whether or not the driver has seen the traffic participants around the vehicle M recognized by the recognition unit 110 based on the direction of sight. The determination unit 140 determines whether or not the driver has seen other traffic participants that are within a fourth predetermined distance from a traffic participant that the driver has determined to have seen, thereby enabling a more efficient determination of whether or not the driver has seen the surrounding traffic participants.

[0085] For example, according to the second embodiment, by adjusting the predetermined distance mentioned above according to the driver's line of sight, it is possible to determine that another traffic participant is visible, who is located near the traffic participant that has been seen and is highly likely to have been seen. This improves the accuracy of the visibility determination process, as well as reducing the processing load and shortening the processing time. Furthermore, according to the second embodiment, the presence of traffic participants in the surrounding area can be notified to the vehicle driver more appropriately.

[0086] [Differentiation] Each of the first and second embodiments described above may be combined with some or all of the other embodiments. For example, the visibility determination in the first embodiment (first visibility determination) and the visibility determination in the second embodiment (second visibility determination) may be switched depending on the attributes of the traffic participants present in the surrounding area. Alternatively, the first and second visibility determinations may be switched depending on the speed of the traffic participants, the number of traffic participants recognized by the recognition unit 110, and the road conditions.

[0087] Furthermore, in the first and second embodiments described above, if a traffic participant is stopped or moving below a predetermined speed, there is a high probability that the traffic participant has already noticed the vehicle M and stopped or slowed down, making contact with the vehicle M less likely compared to when the traffic participant is moving above a predetermined speed. Therefore, the HMI control unit 160 may be configured to notify if there are traffic participants who are moving above a predetermined speed and are not visible to the driver. This allows for priority notification to traffic participants who are more likely to be involved in a collision.

[0088] In the second embodiment, the determination unit 140 may set a region AR1 within a predetermined distance as a sighted region, based on a traffic participant that the driver has determined to have seen. In this case, the determination unit 140 will determine that any traffic participants who enter region AR1 later have also been sighted until a predetermined time has elapsed since the region was set as a sighted region. Furthermore, if the traffic participant that serves as the basis for the sighted region moves, the position of region AR1 may also move to follow the traffic participant. This makes it possible to process not only individual traffic participants but also traffic participants who newly enter the sighted area (region AR1) as sighted. In addition, in the above example, after a predetermined time has elapsed since region AR1 was set as a sighted region, the setting as a sighted region may be canceled.

[0089] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Computers Recognize the surrounding conditions of the vehicle, The direction of the driver's gaze of the aforementioned vehicle is estimated, Based on the aforementioned line of sight, it is determined whether the driver has visually observed the traffic participants in the vicinity of the vehicle that were recognized. Among the aforementioned traffic participants, any other traffic participant located within a predetermined distance from a traffic participant that the driver determined to have seen will also be determined to have been seen by the driver. Driving assistance system.

[0090] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0091] 1. Vehicle System 10 Cameras 12 Radar equipment 14 LIDAR 15 Sonar 16 Object recognition device 20 Communication equipment 30 HMI 32 Display section 34 speakers 40 Vehicle Sensors 50 Driver Monitor Cameras 80. Driver control panel 100 Driving support devices 110 Recognition part 120 Estimation part 130 Prediction Section 140 Judgment section 150 Operation Control Unit 160 HMI Control Unit 180 Storage section M Vehicle

Claims

1. A recognition unit that recognizes the surrounding conditions of the vehicle, An estimation unit for estimating the direction of the driver's gaze on the vehicle, The system includes a determination unit that determines whether the driver has visually observed the traffic participants around the vehicle, as recognized by the recognition unit based on the line of sight direction, The determination unit, when it determines that there is a traffic participant in the vicinity of the vehicle that the driver has seen based on the direction of the driver's line of sight, determines that another traffic participant located within a predetermined distance from the traffic participant has been seen by the driver without performing the determination based on the direction of the driver's line of sight. Driving assistance system.

2. The predetermined distance is made to differ in the direction extending in the direction of the driver's line of sight of the vehicle and in the direction perpendicular to the direction of the line of sight. The driving support device according to claim 1.

3. The predetermined distance is longer the closer it is to the direction of extension, and shorter the closer it is to the orthogonal direction. The driving support device according to claim 2.

4. The estimation unit estimates the risk value of the traffic participant, The determination unit determines that it has not recognized the other traffic participant if, when the other traffic participant is present within the predetermined distance, the risk value of the other traffic participant estimated by the estimation unit is equal to or greater than a threshold. The driving support device according to claim 1.

5. The estimation unit estimates the risk value according to the amount of movement or attributes of the traffic participants. The driving support device according to claim 4.

6. The system further includes a notification unit that notifies the driver if there are traffic participants that the driver has not seen. The driving support device according to claim 1.

7. Computers Recognize the surrounding conditions of the vehicle, The direction of the driver's gaze of the aforementioned vehicle is estimated, Based on the aforementioned line of sight, it is determined whether the driver has visually observed the traffic participants in the vicinity of the vehicle that were recognized. If, among the traffic participants in the vicinity of the vehicle, there is a traffic participant that is determined to have been seen by the driver, then another traffic participant located within a predetermined distance from that traffic participant is determined to have been seen by the driver without performing the determination based on the direction of line of sight. Driving assistance methods.

8. On the computer, Allow the vehicle to recognize its surroundings. To estimate the direction of the driver's gaze on the aforementioned vehicle, Based on the aforementioned line of sight, the system determines whether the driver has visually observed the traffic participants in the vicinity of the recognized vehicle. If, among the traffic participants in the vicinity of the vehicle, there is a traffic participant that is determined to have been seen by the driver, another traffic participant located within a predetermined distance from that traffic participant will be determined to have been seen by the driver without performing the determination based on the direction of the line of sight. program.

Images