Driving assistance device, driving assistance method, and program

Abstract

To provide a driving support device, a driving support method, and a program capable of supporting driving by transmitting more appropriate information to an occupant according to a peripheral situation of a vehicle.SOLUTION: A driving assistance device according to an aspect of the present disclosure includes a peripheral situation recognizer configured to recognize a peripheral situation of a vehicle, a plurality of vibrators configured to give a stimulus to an occupant of the vehicle by vibration, and a vibration controller configured to vibrate at least one of the plurality of vibrators based on a relative position between an object recognized by the peripheral situation recognizer and the vehicle and a direction of the object with respect to the vehicle, in which at least one of the plurality of vibrators is installed at a position where vibration can be transmitted to a sole of the occupant of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a driving assistance method, and a program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have been gaining momentum. To achieve this, efforts are being focused on research and development to further improve traffic safety and convenience through research and development of driving assistance technologies. In this regard, an information presentation system has recently become known that includes a vehicle-to-vehicle information estimation unit that estimates vehicle-to-vehicle information including at least the vehicle-to-vehicle distance to another vehicle ahead in the vehicle's direction of travel, and a presentation unit that presents a predetermined tactile stimulus to the soles of the vehicle driver's feet based on the vehicle-to-vehicle information (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-131892 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with driving assistance technology, there is a possibility that targets may approach the vehicle not only from the front of the vehicle but also from other directions. However, conventional technology only targets the area in front of the vehicle in the direction of travel, which has led to the issue that it may not be possible to convey appropriate information to the occupants.

[0005] In order to solve the above-mentioned problems, one of the objects of the present application is to provide a driving assistance device, a driving assistance method, and a driving assistance program that can assist driving by transmitting more appropriate information to occupants according to the surrounding conditions of the vehicle, thereby contributing to the development of sustainable transportation systems. [Means for solving the problem]

[0006] A driving assistance device, a driving assistance method, and a program according to the present invention employ the following configuration. (1): A vehicle control device according to one embodiment of the present invention is a driving assistance device that includes a surrounding situation recognition unit that recognizes the surrounding situation of the vehicle, a plurality of vibration units that provide vibration stimulation to occupants of the vehicle, and a vibration control unit that vibrates at least one of the plurality of vibration units based on the relative position between a target recognized by the surrounding situation recognition unit and the vehicle and the direction of the target relative to the vehicle, and at least one of the plurality of vibration units is installed in a position that can transmit vibrations to the soles of the feet of occupants of the vehicle.

[0007] (2) In the above aspect (1), the vibration control unit determines the vibration mode of the plurality of vibration units in accordance with the relative position and the direction of the target.

[0008] (3): In the above aspect (1), a toe detection unit is further provided for detecting the position of the occupant's toes, and the vibration control unit determines the vibration mode of the multiple vibration units according to the toe position detected by the toe detection unit.

[0009] (4): In the above aspect (3), a notification unit is further provided that notifies the occupant of information by at least one of display and voice, the occupant including the driver of the vehicle, and the toe detection unit detects the driver's footwear, and further includes a notification control unit that notifies the driver via the notification unit if the driver's footwear is not suitable for driving.

[0010] (5): In the above aspect (1), a motion recognition unit is further provided that recognizes a predetermined motion made by the occupant's foot, and the vibration control unit controls the start or stop of vibration control for the vibration unit based on a predetermined tap motion or a predetermined gesture motion made by the occupant's foot recognized by the motion recognition unit.

[0011] (6): In the above aspect (4), the notification control unit notifies the occupant via the notification unit based on the degree of risk regarding contact or approach between the vehicle and the target, and the notification timing by the notification control unit and the vibration timing by the vibration control unit are controlled synchronized or in stages under specified conditions.

[0012] (7): In the above aspect (1), at least one of the plurality of vibration units is installed on a pedal operator of the vehicle, and the vibration control unit performs vibration control on the vibration unit installed on the pedal operator when the driving mode of the vehicle is a manual driving mode or when switching from an automatic driving mode to a manual driving mode.

[0013] (8): In the above aspect (1), at least one of the plurality of vibration units is installed on a pedal operator of the vehicle, and the vibration control unit performs vibration control on the vibration unit installed on the pedal operator when the foot of the driver of the vehicle comes into contact with the pedal operator.

[0014] (9): Another aspect of the present invention provides a vehicle control method, which is a driving assistance method in which a computer recognizes the surrounding conditions of a vehicle, and vibrates at least one of a plurality of vibration units that provide vibrational stimulation to an occupant of the vehicle based on the relative position of the vehicle and a target included in the recognized surrounding conditions, and the direction of the target relative to the vehicle, and at least one of the plurality of vibration units is installed in a position that can transmit vibrations to the soles of the feet of the occupant of the vehicle.

[0015] (10): Another aspect of the present invention is a program that causes a computer to recognize the surrounding conditions of a vehicle, and vibrates at least one of a plurality of vibration units that provide vibrational stimulation to an occupant of the vehicle based on the relative position of the vehicle and a target included in the recognized surrounding conditions, and the direction of the target relative to the vehicle, and at least one of the plurality of vibration units is installed in a position that can transmit vibrations to the soles of the feet of the occupant of the vehicle. [Effects of the Invention]

[0016] According to the above aspects (1) to (10), it is possible to provide the occupant with more appropriate information depending on the surrounding conditions of the vehicle, thereby assisting driving. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a configuration diagram of a vehicle M equipped with a driving assistance device according to an embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a first vibration control. [Figure 3] FIG. 10 is a diagram illustrating an example of a second vibration control. [Figure 4] 10A and 10B are diagrams for explaining specific examples of vibration modes in the second vibration control. [Figure 5] FIG. 10 is a diagram illustrating an example of a third vibration control. [Figure 6] FIG. 10 is a diagram for explaining the detected position of the toes. [Figure 7] FIG. 10 is a diagram illustrating an example of a fourth vibration control. [Figure 8] FIG. 10 is a diagram illustrating an example of a fifth vibration control. [Figure 9] FIG. 10 is a diagram illustrating an example of a sixth vibration control. [Figure 10] FIG. 12 is a diagram illustrating an example of a seventh vibration control. [Figure 11] 10 is a diagram showing a first installation example of a vibration section 36. FIG. [Figure 12] FIG. 10 is a diagram showing a second installation example of the vibration section 36. [Figure 13] FIG. 10 is a diagram illustrating an example of footwear detection. [Figure 14] 10 is a diagram for explaining recognition of the leg movements of an occupant by the movement recognition unit 116. FIG. [Figure 15] 3 is a flowchart showing an example of processing executed by the driving assistance device 100 of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of a driving assistance device, a driving assistance method, and a program according to the present invention will be described with reference to the drawings.

[0019] [Overall configuration] 1 is a configuration diagram of a vehicle M on which a driving assistance device according to an embodiment is installed. The vehicle M may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0020] The vehicle M is equipped with, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, a driver monitor camera 60, a toe detection unit 70, a driving operator 80, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiple communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The HMI 30 is an example of a "notification unit."

[0021] The camera 10 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location on the vehicle M. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically and repeatedly captures images of the surroundings of the vehicle M. The camera 10 may be a stereo camera.

[0022] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0023] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 may be attached to any location on the vehicle M.

[0024] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of an object (target). The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle M. Some or all of the camera 10, the radar device 12, the LIDAR 14, and the object recognition device 16 are examples of "external environment detection devices."

[0025] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, or with various server devices via a wireless base station, using a network such as a cellular network, a Wi-Fi network, Bluetooth (registered trademark), or DSRC (Dedicated Short Range Communication).

[0026] The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes, for example, a display unit 32, a speaker 34, and a vibration unit 36. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including videos) in the embodiment. The display unit 32 may be configured as a touch panel integrated with the input unit. The speaker 34 outputs a predetermined sound (for example, an alarm, etc.).

[0027] The vibration unit 36 ​​stimulates the occupant by vibration, for example, based on an instruction from the driving assistance device 100. There may be a plurality of vibration units 36, which are installed, for example, at positions where vibration can be applied to the seat or the soles of the occupants' feet. Positions where vibration can be applied to the soles of the occupants' feet include, for example, the floor (floor) in the vehicle cabin, pedal operators (accelerator pedal, brake pedal), footrest, etc. Furthermore, the vibration unit 36 ​​may be installed on the steering wheel 82 included in the driving operators 80, a seat belt in use, etc. Furthermore, the vibration units 36 may be arranged at predetermined intervals.

[0028] The vibrating unit 36 ​​may be, for example, a linear resonant actuator (LRA), which is a type of voice coil motor, but the means (actuator) is not limited to the above example as long as it can transmit a sensory stimulus to the driver through vibration. Therefore, an eccentric motor, a linear motor, a vibration speaker, or the like may also be used as the vibrating unit 36.

[0029] Furthermore, the HMI 30 may include a microphone, a buzzer, a touch panel, keys, etc. in addition to the display unit 32, the speaker 34, and the vibration unit 36. For example, the HMI 30 may include a changeover switch that switches the driving state (the content of driving control) of the vehicle M by operation by the driver of the vehicle M.

[0030] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the yaw rate (for example, the rotational angular velocity around a vertical axis passing through the center of gravity of the vehicle M), a lateral acceleration sensor (lateral G sensor) that detects the lateral acceleration (lateral G) of the vehicle M, a direction sensor that detects the orientation of the vehicle M, and a steering angle sensor that detects the steering angle of the vehicle M (which may be the angle of the steering wheels or the operating angle of the steering wheel). The vehicle sensor 40 may also be provided with a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50. The vehicle sensor 40 may also include a vibration sensor that detects vibrations obtained from the road on which the vehicle M is traveling.

[0031] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may share some or all of its components with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter referred to as a map route) from the position of the vehicle M identified by the GNSS receiver 51 (or an arbitrary input position) to a destination input by the occupant using the navigation HMI 52, with reference to the map information 54. The map information 54 is, for example, information representing road shapes using links indicating roads and nodes connected by the links. The map information 54 may include POI (Point Of Interest) information and the like. The map information 54 may also include, for example, lane center information or lane boundary information such as road dividing lines (hereinafter referred to as dividing lines) that divide lanes. The map information 54 may also include road information such as the radius of curvature (or curvature), gradient, and width of the road (or of each lane included in the road), traffic regulation information, address information (address and postal code), facility information, telephone number information, and the like. The map information 54 may be updated as needed by the communication device 20 communicating with another device. The map information 54 may also be stored in a storage unit within the driving assistance device 100.

[0032] The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the functions of a terminal device such as a smartphone or tablet device carried by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0033] The driver monitor camera 60 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The driver monitor camera 60 is attached to any location that can capture an image of the interior of the vehicle M, including the driver seated in the driver's seat. The camera image captured by the driver monitor camera 60 can be used to obtain the direction of the driver's face and the position of his or her hands, and can also obtain the positions of the seats in which passengers other than the driver are seated. The driver monitor camera 60 outputs the captured image to the driving assistance device 100.

[0034] The toe detector 70 detects, for example, the position of the occupant's toes (for example, the ankles or the part from the heel to the toe). A specific example of the toe detector 70 will be described later.

[0035] The driving operators 80 include, for example, a steering wheel 82, an accelerator pedal 84, a brake pedal 86, a turn signal switch, a shift lever, and other operators. The driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the driving assistance device 100 or some or all of the traveling drive force output device 200, the brake device 210, and the steering device 220. The steering wheel 82 is an example of a "steering operator." The accelerator pedal 84 and the brake pedal 86 are examples of "pedal operators."

[0036] For example, the steering wheel 82 is provided with a steering wheel sensor (SW sensor) 82A and a vibration unit 36 ​​that vibrates the part that the driver grips. The SW sensor 82A detects whether the driver is in contact with the steering wheel 82. The SW sensor 82A also detects the amount of operation of the steering wheel 82 (torque (also referred to as steer torque), steering amount, steering change rate) that changes in response to the driver's operation of the steering wheel 82 (hereinafter referred to as steering operation). The SW sensor 82A may also detect whether the driver is gripping the steering wheel 82. The steering wheel 82 does not necessarily have to be annular, and may be in the form of an irregularly shaped steering wheel, a joystick, buttons, or the like. In this case, the SW sensor 82A detects the amount of operation according to the respective form.

[0037] An accelerator pedal sensor (AP sensor) 84A is provided for accelerator pedal 84. AP sensor 84A detects whether the driver has his / her foot on accelerator pedal 84, whether the driver is operating accelerator pedal 84 (hereinafter referred to as accelerator operation) and the amount of operation of accelerator pedal 84 (amount of opening change, opening change rate) that changes depending on the operation.

[0038] A brake pedal sensor (BP sensor) 86A is provided on the brake pedal 86. The BP sensor 86A detects whether the driver has his / her foot on the brake pedal 86, whether the driver is operating the brake pedal 86 (hereinafter referred to as "brake operation"), and the amount of operation of the brake pedal 86 (amount of opening change, opening change rate) that changes depending on the operation. The accelerator operation and the brake operation are each an example of "speed operation."

[0039] The driving force output device 200 outputs a driving force (torque) to the driving wheels for driving the vehicle M. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components in accordance with information input from the driving assistance device 100 or information input from the driving operator 80.

[0040] Braking device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and an ECU. The ECU controls the electric motor according to information input from driving assistance device 100 or information input from driving operator 80, so that a brake torque corresponding to the braking operation is output to each wheel. Braking device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of brake pedal 86 included in driving operator 80 to the cylinder via a master cylinder. Note that braking device 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake device that controls an actuator according to information input from driving assistance device 100 to transmit hydraulic pressure from a master cylinder to the cylinder.

[0041] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to a rack and pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the driving operator 80.

[0042] [Driving assistance devices] The driving assistance device 100 includes, for example, a recognition unit 110, a determination unit 120, a control unit 130, and a storage unit 150. The recognition unit 110, the determination unit 120, and the control unit 130 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or an SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as the HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0043] For example, settings are made within the driving force output device 200, the braking device 210, and the steering device 220 so that instructions from the driving support device 100 to the driving force output device 200, the braking device 210, and the steering device 220 are executed with priority over detection results from the driving operator 80. Regarding braking, if the braking force based on the operation amount of the brake pedal 86 is greater than the instruction from the driving support device 100, the latter may be set to be executed with priority. Furthermore, communication priority in an in-vehicle local area network (LAN) may be used as a mechanism for executing instructions from the driving support device 100 with priority.

[0044] The storage unit 150 may be realized by the various storage devices described above, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 150 stores, for example, programs and various other information. The storage unit 150 may also store the map information 54 described above.

[0045] The recognition unit 110 includes, for example, a surrounding situation recognition unit 112, a footwear recognition unit 114, and a motion recognition unit 116. The surrounding situation recognition unit 112 recognizes the surrounding situation of the vehicle M based on information input from an external environment detection device. For example, the surrounding situation recognition unit 112 recognizes the position, speed, acceleration, and other status of targets (objects) present in the vicinity (for example, within a predetermined distance (first predetermined distance) from the vehicle M). Targets include, for example, traffic participants such as other vehicles, bicycles, and pedestrians, as well as road structures such as curbs, medians, and guardrails. The position of the target is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of the target may be represented by a representative point such as the center of gravity or a corner of the target, or may be represented by an area. The "state" of the target may include, if the target is a moving object, the acceleration or jerk of the target, or the "action state" (for example, whether the target is changing lanes or about to change lanes). In addition, the surrounding situation recognition unit 112 recognizes the relative position and relative speed of the target.

[0046] Furthermore, the surrounding situation recognition unit 112 recognizes, for example, the lane in which the vehicle M is traveling (driving lane). For example, the surrounding situation recognition unit 112 performs known analysis processing (for example, edge extraction, feature extraction, pattern matching processing, etc.) on an image captured by the camera 10 (hereinafter, referred to as a camera image), and recognizes the positions and patterns of marking lines around the vehicle M (for example, an arrangement of solid lines and dashed lines) from the analysis results. Marking lines are also an example of a landmark. The surrounding situation recognition unit 112 may also recognize the positions and patterns of marking lines around the vehicle M by referring to map information 54 based on the position information of the vehicle M. The surrounding situation recognition unit 112 may also recognize the driving lane using at least one of the positions and patterns of marking lines obtained from the camera image and the positions and patterns of marking lines obtained from the map information. The surrounding situation recognition unit 112 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only marking lines but also shoulders, curbs, medians, guardrails, etc. This recognition may take into account the position of the vehicle M obtained from the navigation device 50 and the processing results from the INS. The surrounding situation recognition unit 112 may also recognize adjacent lanes adjacent to the driving lane. The surrounding situation recognition unit 112 may also recognize the radius of curvature (or curvature), gradient, width, etc. of the driving lane (or road) from at least one of camera images and map information. The surrounding situation recognition unit 112 also recognizes obstacles, stop lines, red lights, toll booths, and other road phenomena from the recognition results of the surrounding situation. Obstacles include objects that obstruct travel in the driving lane and objects that need to be avoided (traffic participants, etc.). These objects are also included in the targets.

[0047] The surrounding situation recognition unit 112 may also recognize the position and attitude of the vehicle M with respect to the driving lane. For example, the surrounding situation recognition unit 112 may recognize the deviation of the reference point of the vehicle M from the center of the lane and the angle it forms with a line connecting the centers of the lanes in the traveling direction of the vehicle M as the relative position and attitude of the vehicle M with respect to the driving lane. Alternatively, the surrounding situation recognition unit 112 may recognize the position of the reference point of the vehicle M with respect to one of the side edges of the driving lane (a dividing line or a road boundary) as the relative position of the vehicle M with respect to the driving lane. The surrounding situation recognition unit 112 may also recognize the position and attitude of other vehicles traveling in the driving lane of the vehicle M, or recognize whether the other vehicles are located on the center side of the driving lane or on the dividing line side as viewed from the vehicle M.

[0048] The footwear recognition unit 114 recognizes the footwear worn by the driver of the vehicle M. For example, the footwear recognition unit 114 recognizes the type of footwear worn by the driver based on the detection results from the toe detection unit 70. The function of the footwear recognition unit 114 will be described in detail later.

[0049] The motion recognition unit 116 recognizes a motion made by the occupant's feet (for example, a tapping motion or a gesture motion) based on the detection result by the toe detection unit 70. The function of the motion recognition unit 116 will be described in detail later.

[0050] The determination unit 120 includes, for example, a risk determination unit 122 and a driving situation determination unit 124. Based on the recognition result by the surrounding situation recognition unit 112, the risk determination unit 122 determines whether or not there is a risk (possibility) of contact between the vehicle M and a target that exists around the vehicle M. Furthermore, if the risk determination unit 122 determines that there is a risk, it determines in which direction the risk is facing relative to the vehicle M (for example, forward, backward, to the right, to the left, etc.).

[0051] For example, the risk determination unit 122 calculates a first time to collision (TTC) until the vehicle M comes into contact with the target based on the relative distance and relative speed between the vehicle M and the target. The first time to collision TTC is calculated, for example, by dividing the relative distance by the relative speed. The risk determination unit 122 determines that there is a risk of the vehicle M coming into contact with the target if the calculated first time to collision TTC is within a predetermined time, and determines that there is no risk of contact if the first time to collision TTC is greater than the predetermined time. Alternatively, the risk determination unit 122 may determine whether there is a risk of the vehicle M coming within a predetermined distance of the target, instead of the risk of contact between the vehicle M and the target. In this case, the predetermined time may be set to a value greater than the time for making a contact determination, or the determination may be made based on the distance (relative distance) from the target. This not only notifies the driver of the vehicle M that there is a possibility of contact, but also notifies that a target is nearby (in other words, that a target is approaching), thereby further improving preventive safety. Furthermore, the risk determination unit 122 may determine the magnitude of the risk degree when it is determined that there is a possibility that the vehicle M and the target will come into contact with or approach each other. In this case, the risk degree increases as the first time to contact TTC (or the relative distance) decreases.

[0052] The driving situation determination unit 124 determines the driving situation of the vehicle M based on the recognition result by the surrounding situation recognition unit 112. For example, the driving situation determination unit 124 determines whether or not there is a possibility that the vehicle M will deviate from the lane dividing line that separates the driving lane based on the recognition result. For example, the driving situation determination unit 124 determines that there is a possibility that the vehicle M will deviate from the lane dividing line if there is a possibility that the reference position (e.g., an edge, center of gravity, center) of the vehicle M will go beyond (pass over) either of the left and right lane dividing lines that separate the driving lane recognized by the recognition unit 110 and deviate from the driving lane, and determines that there is no possibility that the vehicle M will deviate from the lane dividing line if there is no possibility that the vehicle M will deviate from the lane dividing line.

[0053] Furthermore, the driving situation determination unit 124 derives a predicted future route of the vehicle M from the speed and yaw rate of the vehicle M obtained from the vehicle sensor 40, and calculates a second time to line crossing (TTLC) (=d / VM) until the vehicle M reaches the lane marking based on the distance between the derived predicted route and the lane marking (deviation route length d) and the speed VM. If the second time to line crossing (TTLC) is less than a predetermined time, the driving situation determination unit 124 determines that there is a possibility that the vehicle M will deviate from the driving lane, and if the second time to line crossing (TTLC) is equal to or greater than the predetermined time, the driving situation determination unit 124 determines that there is no possibility of deviation. Furthermore, if it determines that there is a possibility that the vehicle M will deviate from the driving lane, the driving situation determination unit 124 may determine the magnitude of the possibility of deviation. In this case, the magnitude of the possibility of deviation increases as the second time to line crossing (TTLC) decreases.

[0054] In addition to the above-mentioned functions, the determination unit 120 may also determine whether the footwear recognized by the footwear recognition unit 114 is suitable for driving, or determine whether the foot movement of the occupant is a predetermined movement based on the recognition result by the movement recognition unit 116.

[0055] The control unit 130 controls various functions, devices, etc. of the vehicle M. The control unit 130 includes, for example, a vibration control unit 132, a notification control unit 134, and a driving control unit 136. The vibration control unit 132 controls the vibration of at least one of the multiple vibration units 36 based on, for example, the determination result by the risk determination unit 122 or the determination result by the driving situation determination unit 124. In this case, the vibration control unit 132 determines the vibration mode of the multiple vibration units depending on, for example, the relative position with respect to the target or the direction of the target. The vibration mode includes, for example, at least one of the position of the vibration unit to be vibrated, the magnitude of vibration (vibration intensity), and the vibration period (including, for example, frequency, pattern, etc.). The vibration control unit 132 may also control the vibration of the vibration unit 36 ​​based on the recognition result by the footwear recognition unit 114 or the recognition result by the action recognition unit 116.

[0056] The notification control unit 134 notifies the occupants of the vehicle M of predetermined information based on, for example, information obtained from the communication device 20, the HMI 30, the vehicle sensors 40, the driver monitor camera 60, etc., information detected by the SW sensor 82A, the AP sensor 84A, and the BP sensor 86A, the recognition result by the recognition unit 110, the determination result by the determination unit 120, etc. The predetermined information includes, for example, information related to the traveling of the vehicle M, such as information about the state of the vehicle M and information about driving control. The information about the state of the vehicle M includes, for example, the speed of the vehicle M, the engine speed, the shift position, etc. Furthermore, the information about driving control includes, for example, the type of driving control (driving state) currently being executed, the reason for activation of the driving control, the reason for activation of the vibration control, the status of the driving control, information indicating that the driving control has started or ended, etc. Furthermore, the information about driving control may include an alert to the driver (for example, a departure alert) and information prompting the driver to perform a predetermined driving operation or to pay attention. In addition, the specified information may include information regarding the current location and destination of vehicle M, the remaining fuel level, etc., and may also include information unrelated to the driving control of vehicle M, such as television programs, content (e.g., movies) stored on a storage medium such as a DVD, etc.

[0057] For example, the notification control unit 134 may generate an image including the predetermined information described above and display the generated image on the display unit 32 of the HMI 30, or may generate sound indicating the predetermined information and output the generated sound from the speaker 34 of the HMI 30. The sound is output, for example, when driving control is started or stopped, when a call is received, when the displayed image is switched, or when the vehicle M enters a predetermined state. Furthermore, the notification control unit 134 may perform control so that the occupant is notified in stages, either in synchronization with vibration control by the vibration control unit 132 or including vibration control.

[0058] The traveling control unit 136 executes driving control to control at least one of the speed and steering of the vehicle M based on the recognition result by the recognition unit 110. For example, when the traveling situation determination unit 124 determines that there is a possibility that the vehicle M will deviate from the traveling lane, the traveling control unit 136 controls at least the steering device 220 to execute control to prevent the vehicle M from deviating from the traveling lane. Furthermore, when the risk determination unit 122 determines that there is a possibility that the vehicle M will come into contact with a target, the traveling control unit 136 controls at least one of the brake device 210 or the traveling drive force output device 200 and the steering device 220 to execute driving control to avoid contact between the vehicle M and the target.

[0059] Furthermore, the driving control unit 136 may execute driving control such as ACC (Adaptive Cruise Control System) control that causes the vehicle M to travel at a constant speed in the driving lane at a preset speed (set vehicle speed) based on the recognition results by the surrounding situation recognition unit 112, etc., or instructions from the driver via the HMI 30, or ALC (Auto Lane Change) control that causes the vehicle M to change lanes by operating at least the steering of the vehicle M. The above-mentioned driving control includes fully automated driving, in which case driving control is executed in which the speed and steering of the vehicle M are controlled independently of the driver's operation.

[0060] [About vibration control] Next, details of vibration control by the vibration control unit 132 will be explained using several examples. Note that, although the following mainly focuses on vibration control for the driver seated in the driver's seat, similar vibration control may also be performed for passengers other than the driver who are seated in other seats of the vehicle M. The position at which the passenger is seated can be obtained, for example, by performing a known person recognition process on the camera image captured by the driver monitor camera 60.

[0061] <First vibration control> FIG. 2 is a diagram illustrating an example of the first vibration control. The example in FIG. 2 shows a schematic diagram of the vicinity of the driver's seat as viewed from above. In FIG. 2, the X-axis direction indicates the front direction of the vehicle M, and the Y-axis direction indicates the lateral direction of the vehicle M. The example in FIG. 2 schematically illustrates a floor portion (floor portion) FL (an example of an area capable of transmitting vibrations to the soles of the driver's feet), a seat portion ST on which the driver sits, and the positions of the driver's left foot LF and right foot RF (position of the toes) placed on the floor portion FL. The seat portion ST has a seat cushion portion ST1 and a seat back portion ST2. In the example in FIG. 2, a vibration unit 36-1 is installed in the center of the front of the floor portion FL, and vibration units 36-2 and 36-3 are installed on the left and right sides of the seat back portion ST2. For example, when a driving control is executed to control the speed of the vehicle M without relying on the operation of the occupant, both of the driver's feet may be placed on the floor portion FL as shown in FIG. 2.

[0062] In the configuration of FIG. 2, for example, if the risk determination unit 122 determines that there is a risk of contact with a target ahead of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-1. As a result, vibrations from the vibration unit 36-1 are transmitted to the soles of the driver's left foot LF and right foot RF placed on the floor unit FL. The driver receives stimulation from the vibrations through the soles of his / her feet (particularly the toe sides), and is therefore able to recognize that a risk has occurred ahead of the vehicle M. Furthermore, in the first vibration control, if the risk determination unit 122 determines that there is a risk of contact with a target behind the vehicle M, the vibration control unit 132 vibrates the vibration units 36-2 and 36-3. As a result, the driver feels the vibrations from his / her back, which is in contact with the seat back surface ST2, and is therefore able to recognize that a risk has occurred behind the vehicle M. This makes it possible to more accurately notify the occupant of the surrounding situation. The vibration control unit 132 may vibrate only the vibration unit 36-2 when it is determined that there is a risk on the left rear side of the vehicle M, and may vibrate only the vibration unit 36-3 when it is determined that there is a risk on the right rear side. This allows the driver to more accurately know that there is a risk on either the left or right rear side.

[0063] Furthermore, the vibration control unit 132 may change the vibration mode, such as the magnitude and vibration period, according to the risk level determined by the risk determination unit 122. In this case, for example, the vibration control unit 132 increases the vibration or shortens the vibration period as the risk level increases. Furthermore, the vibration control unit 132 may change the vibration mode according to the direction in which the risk exists. In this case, for example, the vibration control unit 132 increases the vibration or shortens the vibration period as the direction in which the risk exists is closer to the traveling direction of the vehicle M. In this way, by changing the vibration mode of the vibration unit according to the surrounding situation, the driver can be made to intuitively grasp the risk. For example, the strength of the vibration can allow the driver to intuitively grasp the direction in which the risk is approaching or the events moving away from the risk.

[0064] In the first vibration control, the vibration control unit 132 also performs control to vibrate the vibration unit closer to the side of departure when the driving situation determination unit 124 determines that the vehicle M will deviate from the driving lane (demarcation line), instead of (or in addition to) the determination result by the risk determination unit 122. Furthermore, the vibration control unit 132 may change the vibration mode of the vibration unit 36 ​​depending on the degree of possibility of departure. This allows the driver to more appropriately understand the driving situation. <Second vibration control> Fig. 3 is a diagram showing an example of the second vibration control. The example of Fig. 3 differs from the example of Fig. 2 in that the vibration units 36-4 and 36-5 are installed on the left and right sides of the front of the floor unit FL, rather than in the center of the front, and the seat back surface ST2 does not have the vibration unit 36. In the second vibration control, the vibration control unit 132 controls the vibration modes of the vibration units 36-4 and 36-5 according to the direction of the risk determined by the risk determination unit 122.

[0065] Fig. 4 is a diagram for explaining a specific example of a vibration mode in the second vibration control. In the example of Fig. 4, a vehicle M is traveling toward a T-junction, and a pedestrian OB1 (an example of a target object) is approaching the vehicle M from the left side as viewed from the vehicle M at the T-junction. Here, the vehicle M is traveling at a speed VM, and the pedestrian OB1 is moving at a speed Vob1.

[0066] In this case, the risk determination unit 122 determines whether there is a risk (possibility) of contact between the vehicle M and the pedestrian OB1 based on the relative position and relative speed (speed VM-speed Vob1) of the pedestrian OB1 with respect to the vehicle M. If it is determined that there is a risk of contact, the vibration control unit 132 vibrates only the vibration unit 36-4 that is closest to the direction in which the pedestrian OB1 is located as viewed from the vehicle M. As a result, when the driver places both feet near the center of the floor unit FL, the left foot LF receives stronger vibration, allowing the driver to more accurately grasp that a target (obstacle) is approaching from the left front (that a risk has occurred in the left front). Furthermore, instead of vibrating only the vibration unit 36-4, the vibration control unit 132 may vibrate both the vibration unit 36-4 and the vibration unit 36-5. In this case, the vibration control unit 132 controls the vibration mode so that the vibration of the vibration unit 36-4 in the direction in which the risk is occurring is stronger (and / or the vibration period is shorter) than the vibration of the other vibration units 36-5. As a result, similar to the first vibration control, the driver can be made to more accurately recognize the presence of a risk and the direction in which the risk exists.

[0067] Furthermore, in the second vibration control, the vibration control unit 132 may also perform control to vibrate the vibration unit closer to the side of departure when it is determined that the vehicle M will deviate from the lane, instead of (or in addition to) the determination result by the risk determination unit 122. Furthermore, the vibration control unit 132 may change the vibration mode of the vibration unit 36 ​​depending on the degree of risk and the magnitude of the possibility of departure. This allows the driver to more appropriately understand the driving situation.

[0068] <Third vibration control> 5 is a diagram showing an example of the third vibration control. The vehicle M of this embodiment is capable of not only manual driving but also automatic driving. Therefore, the driver's feet may be positioned forward (forward), backward (rearward), leftward (left side), or rightward (right side) from the center position of the floor portion FL, or the toes of each foot may be offset left and right, front and rear, or only one foot may be placed on the floor by crossing the legs. Therefore, in the third vibration control, the vibration modes of the multiple vibration units 36 are controlled according to the position of the feet on the floor portion FL, etc.

[0069] In the example of FIG. 5, similarly to FIG. 4, vibration units 36-4 and 36-5 are installed on the left and right sides of the front of floor unit FL. Note that seat unit ST is not shown in the example of FIG. 5. In the example of FIG. 5, the position of the driver's feet is located in a position lower right (rear right side) in the figure rather than the center of floor unit FL. In such a situation, if the risk determination unit 122 determines that there is a risk of contact with an object, and / or if the driving situation determination unit 124 determines that there is a possibility that vehicle M will deviate from the driving lane (demarcation line), vibration unit 36-4, which is located farther from the feet (left foot LF, right foot RF) of vibration units 36-4 and 36-5, is vibrated with stronger (larger) vibration than vibration unit 36-5. This ensures that vibration is transmitted to both feet.

[0070] The vibration control unit 132 may change the magnitude and cycle of the vibration depending on the distance from the feet (left foot LF, right foot RF), or may change the magnitude and cycle of the vibration depending on the degree of risk or the likelihood of deviation. Furthermore, when only one foot is placed on the vibrating floor part FL, the vibration control unit 132 may increase the vibration or shorten the vibration cycle compared to when both feet are placed. This allows the vibration to be reliably transmitted to the sole even when only one foot is placed.

[0071] [Foot tip detection section] Here, a method for detecting the toes of the driver (and other passengers) in the toe detection unit 70 will be described with reference to the drawings. FIG. 6 is a diagram for explaining the detection position of the toes. In the example of FIG. 6(A), a camera is used as the toe detection unit 70-1. This camera is installed in a position where it can capture an image of an area including the position where the driver's feet will be placed (for example, above the floor part FL), and captures images at a predetermined period. The captured camera images are subjected to object recognition and the like using known image analysis processing to detect the positions of the toes (left foot LF, right foot RF).

[0072] In the example of Figure 6(B), a plurality of pressure sensors (pressure sensors) are arranged on the floor portion FL as the toe detection unit 70-2. The pressure sensors are arranged in a grid pattern on the floor portion (upper surface), and the position of the toes is detected using the coordinate points where the sensors detect a pressure equal to or greater than a predetermined amount. The pressure sensors may be installed on the floor mat or within the floor portion FL.

[0073] In the example of FIG. 6(C), a vibrometer that measures the magnitude of vibration is used as the toe detection unit 70-3. The vibrometer is installed, for example, at the installation position of the vibration unit 36. The toe detection unit 70-3 detects the magnitude of the vibration caused by the vibration unit 36 ​​as feedback and estimates the foot position based on the detection result. For example, when the vibration unit 36 ​​vibrates the floor unit FL and the position of the feet (left foot LF, right foot RF) is close to the vibration unit 36, the weight of the feet is applied to the floor unit FL, making it difficult for the floor unit FL to vibrate. In other words, the closer the feet are to the vibration unit 36, the weaker the vibration measured by the toe detection unit 70-3 (vibration meter), and so the toe detection unit 70-3 detects the position of the feet based on the measured vibration.

[0074] In the example of Fig. 6(D), a camera is used as the toe detection unit 70-4. In the example of Fig. 6(D), an image including the driver's knee KN is captured, and the position of the toes is estimated from the position of the knee KN included in the analysis result of the captured camera image. This makes it possible to estimate the position of the feet from the position of the knee KN even in a situation where the toes cannot be captured (for example, a situation where the driver is wearing a long skirt or long pants and the toes cannot be captured by the camera).

[0075] The toe detection unit 70 may detect the toes using more than one of the above-mentioned detection methods using the toe detection units 70-1 to 70-4. The toe detection unit 70 may also use a radar device instead of (or in addition to) the toe detection units 70-1 to 70-4. In this case, the radar device emits radio waves such as millimeter waves to the floor unit FL and detects the radio waves (reflected waves) reflected by objects such as feet to detect the position (distance and direction) of the feet. The toe detection unit 70 may also use the above-mentioned method to detect that the toes are placed on the pedal operators. In this case, it is possible to detect that the toes are placed on the pedal operators by capturing an image including the pedal operators with a camera or by installing a pressure sensor on the pedal operators.

[0076] <The fourth vibration control> FIG. 7 is a diagram showing an example of the fourth vibration control. In the fourth vibration control, as shown in FIG. 7, nine vibration units 36-7 to 36-15 are arranged in a grid pattern (3 in the front-rear direction × 3 in the left-right direction) on the floor unit FL. The vibration units 36-7 to 36-15 may be spaced equally apart, or may be spaced at different distances in the left-right direction and the front-rear direction. The number and layout of the vibration units are not limited to the example shown in FIG. 6. For example, as shown in FIG. 7(A), when the toes of both feet (left foot LF, right foot RF) are placed slightly forward (closer to the front) of the center of the floor unit FL, and the risk determination unit 122 determines that there is a possibility of contact with a target (obstacle) located to the left and front of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-7 located to the left and front of the foot position. 7(A) , when an object determined to be at risk of contact is present in front of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-8, and when the object is present in the front right of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-9. When it is determined that there is a risk of contact with a target on the left side of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-10, and when it is determined that there is a risk of contact on the right side of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-12. When it is determined that there is a risk of contact with the left rear of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-13, when it is determined that there is a risk of contact with the rear of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-14, and when it is determined that there is a risk of contact with the right rear of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-15.

[0077] Also, as shown in Figure 7(B), when both feet are located at the right rear of the floor portion FL and the risk determination unit 122 determines that there is a risk of contact with an object (obstacle) at the left front of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-11 located at the closest position to the left front of the feet.

[0078] Here, it is assumed that vibrations are transmitted to the feet of both the driver sitting in the driver's seat and the passenger sitting in the passenger seat provided next to the driver's seat, with the feet of the driver sitting in the driver's seat in the position shown in FIG. 7(A) and the feet of the passenger sitting in the passenger seat in the position shown in FIG. 7(B). In this situation, if the risk determination unit 122 determines that there is a risk of contact with an object on the left front side of the vehicle M, the vibration control unit 132 vibrates the vibration unit 36-7 on the floor portion FL of the driver's seat and the vibration unit 36-11 on the floor portion FL of the passenger seat. This makes it possible to vibrate the vibration units in accordance with the foot positions of each of the multiple passengers in the vehicle M, allowing each passenger to more accurately grasp the surrounding situation.

[0079] <5th vibration control> FIG. 8 is a diagram illustrating an example of the fifth vibration control. In the fifth vibration control, nine vibration units 36-7 to 36-15 (three in the front-rear direction and three in the left-right direction) are installed in the floor area FL, as in FIG. 7. The fifth vibration control illustrates a situation in which a vehicle M is traveling at a speed VM in a lane L1 defined by left and right lane markings RS1 and RS2, and the traveling situation determination unit 124 determines that the vehicle M may deviate from the lane L1. In this case, the vibration control unit 132 vibrates the vibration unit 36-10 located to the left of the driver's feet (left foot LF and right foot RF) because the vehicle M may deviate to the left side of the lane. The vibration control unit 132 may also vibrate the left vibration units 36-7, 36-10, and 36-13 to help the driver pay attention to the left side of the vehicle M. In this case, the vibration units may vibrate synchronously or sequentially at a fixed interval. This makes it easier for the driver to understand that there is a possibility that the vehicle M may deviate from the lane marking RS1 extending to the left side of the vehicle M.

[0080] The vibration control unit 132 may control the vibration mode depending on the degree of the possibility of deviation as described above. In this case, the vibration control unit 132 increases the vibration or shortens the vibration period depending on the degree of the possibility of deviation.

[0081] <6th vibration control> 9 is a diagram showing an example of the sixth vibration control. The example of FIG. 9 shows vehicle M traveling at speed VM on lane L1, as well as another vehicle m1 approaching vehicle M from the right rear of vehicle M at speed Vm1. The other vehicle m1 is an example of an obstacle. In this case, it is assumed that the risk determination unit 122 determines that there is a possibility of contact between vehicle M and vehicle m1 based on the relative positions and relative speeds of vehicle M and vehicle m1. In this case, the vibration control unit 132 vibrates the vibration unit 36-15 located to the right rear of the positions of the left and right feet LF and RF.

[0082] The vibration control unit 132 may perform control to increase the vibration as the degree of risk increases and the possibility of contact increases. This allows the occupant to be notified of the degree of possibility of contact more accurately. The vibration control unit 132 may also change the vibration mode depending on the type of object that may be contacted. In this case, the vibration control unit 132 changes the vibration mode depending on whether the object is another vehicle or a pedestrian. The vibration control unit 132 may also vibrate in different modes depending on whether it is determined that there is a risk of contact with the object or whether it is determined that there is a possibility of deviation from a lane (marking line). This allows the driver (occupant) to understand the surrounding situation in more detail by using different vibration modes.

[0083] <7th vibration control> FIG. 10 is a diagram showing an example of the seventh vibration control. In the seventh vibration control, when vibration units are provided on each of the pedal operators (accelerator pedal 84, brake pedal 86) and the footrest, vibration control is performed on these vibration units 36 as well. In the example of FIG. 10, a footrest FR is provided on the upper left side of the floor FL, and a vibration unit 36-7 is provided there. In addition, vibration units 36-8 to 36-15 are provided in a grid pattern in other areas of the floor FL. Furthermore, in the example of FIG. 10, a vibration unit 36-16 is provided on the accelerator pedal 84, and a vibration unit 36-17 is provided on the brake pedal 86. Note that multiple vibration units may be provided on each of the footrest FR, accelerator pedal 84, and brake pedal 86. In the example of FIG. 10, the driver's left foot LF is placed in the center of the floor FL, and the right foot RF is placed on the accelerator pedal 84. The toe detection unit 70 detects the positions of the driver's left and right toes. In an embodiment, the AP sensor 84A described above may detect a foot placed on the accelerator pedal 84, and the BP sensor 86A may detect a foot placed on the brake pedal 86.

[0084] In the seventh vibration control, when the risk determination unit 122 determines that there is a risk of contact with a target and / or when the driving situation determination unit 124 determines that there is a possibility that the vehicle M will deviate from the driving lane (demarcation line), and one foot is placed on the pedal operator, the vibration unit closest to each foot is vibrated. In the example of Fig. 10, the vibration control unit 132 vibrates the vibration units 36-11 and 36-16.

[0085] As shown in the seventh vibration control, by incorporating a vibration unit into the pedal operator as well, even if the driver's foot moves from the floor to the pedal operator, a tactile stimulus can be given to the driver through the pedal, ensuring that the driver can perceive the vibration notification.

[0086] Furthermore, when a foot is placed on the floor portion FL, the vibration control portion 132 may vibrate the vibration portion in a direction where there is a risk of the foot position or a direction where there is a possibility of departure from the lane, thereby notifying the driver of information regarding the direction.

[0087] Note that, based on the control state of the driving control unit 136, when the driving mode of the vehicle M is manual driving or when the driving mode is switched from the automatic driving mode to the manual driving mode, the vibration control unit 132 may perform control to vibrate the vibration units 36-16, 36-17 provided on the pedal operators based on the determination result by the determination unit 120. This allows the vibration units provided on the pedals to vibrate in a state where the driver is more likely to operate the pedals.

[0088] In the seventh vibration control, when a foot is placed on the footrest portion FR, the vibration control portion 132 vibrates the vibration portion 36-7 provided on the footrest portion FR based on the determination result by the determination portion 120.

[0089] In the embodiment, each of the first to sixth vibration controls described above may be combined with at least a part of the other vibration controls.

[0090] [Installation example of vibrating unit 36] Next, an example of the installation of the vibration unit 36 ​​in the embodiment will be described with reference to the drawings. FIG. 11 is a diagram showing a first installation example of the vibration unit 36. The example in FIG. 11 is a schematic diagram showing the positional relationship between the floor portion FL and the seat portion ST around the driver's seat, as viewed from the side. In the example in FIG. 11, the vibration unit 36 ​​is provided on the seat back surface portion ST2, and multiple vibration units 36 are arranged at predetermined intervals in a floor mat FM laid on the floor portion FL. The floor mat FM may be provided with the pressure-sensitive sensor (toe detection unit 70-2) described above. By arranging the vibration unit 36 ​​in the floor mat FM, it is possible to transmit vibrations with higher resolution to the feet of the driver D. Furthermore, since there is no need to newly install the vibration unit 36 ​​in the floor portion FL, costs and the burden of installation work can be reduced.

[0091] The vibrating unit 36 ​​may be installed within the floor portion FL after taking into consideration the material of the floor mat FM and ensuring sufficient vibration propagation. Figure 12 is a diagram showing a second installation example of the vibrating unit 36. In the second installation example, multiple vibrating units 36 are arranged within the floor portion FL, as compared to the first installation example. This allows the vibrating unit in the appropriate position to vibrate even if the position of the floor mat FM is shifted.

[0092] In the first and second installation examples, a typical floor mat FM is made of a material with vibration-damping and sound-absorbing properties to reduce vibrations (road noise) when the vehicle M is running. Therefore, when installing the vibrating part 36 on the floor part FL, it is necessary to devise a method for transmitting the vibration so that the vibration is not attenuated. For example, when installing the vibrating part 36 on the floor mat FM as in the first installation example, the laminated structure of the floor mat FM is such that the first layer (lower layer) facing the floor part FL is a vibration-damping and sound-absorbing layer, the second layer (middle layer) contains the vibrating part 36, and the third layer (upper layer) is a surface layer. This facilitates the transmission of vibrations from the vibrating part 36 to the soles of the feet. In the second installation example, a material with vibration-damping and sound-absorbing properties is provided on the underside (ground side) of the floor part FL, the vibrating part is installed on top of that, and the floor mat FM is configured without a surface layer or vibration-damping and sound-absorbing layer, thereby facilitating the transmission of vibrations from the vibrating part 36 to the soles of the feet.

[0093] [Notification Control] In the embodiment, when the vibration control unit 132 vibrates the vibration unit 36, the notification control unit 134 may generate an image or sound indicating the reason for the vibration and output it from the HMI 30. This allows the occupant to more accurately understand the reason for the vibration to the feet.

[0094] Furthermore, the notification control unit 134 may synchronize and control under predetermined conditions the timing of notification of information by display on the HMI 30 or audio output and the timing of vibration of the vibration unit 36 ​​by the vibration control unit 132, or may control them in stages. For example, when the risk level determined by the risk determination unit 122 is less than a first threshold, the vibration control unit 132 controls vibration, and when the risk level is equal to or greater than the first threshold, the vibration control unit 132 controls vibration and the notification control unit 134 outputs an alarm (an alarm display and audio output). In this way, by providing notifications in which image display and audio output are linked according to the risk level, the driver can be made to take action according to the risk.

[0095] The notification control unit 134 may also change the color or brightness of the image or the sound depending on the level of risk. In this case, the notification control unit 134 increases the notification intensity (e.g., brightness, volume) linearly or stepwise as the risk level increases from low to high. Furthermore, vibration and display output (audio output) are always performed simultaneously regardless of the level of risk. In this way, by combining multiple sensory systems such as touch, vision, and hearing to provide notifications, a multimodal effect can be achieved, further improving intuitiveness.

[0096] The above-described control combining the notification control of the notification control unit 134 and the vibration control of the vibration control unit 132 may also be applied when the driving situation determination unit 124 determines that the vehicle M will deviate from its lane. In this case, the notification control and the vibration control are synchronized or stepwise control is performed depending on the degree of the possibility of deviation, etc.

[0097] [Footwear Notice] In addition, if the notification control unit 134 determines that the driver is wearing shoes that are not suitable for driving based on the type of footwear recognized by the footwear recognition unit 114, it may notify the driver of the information by at least one of display and audio via the HMI 30.

[0098] Fig. 13 shows an example of footwear detection. Fig. 13(A) shows an example in which the toe detection unit 70-1 is a camera, and Fig. 13(B) shows an example in which the toe detection unit 70-2 is a pressure-sensitive sensor arranged in a grid pattern on the floor part FL.

[0099] For example, the footwear recognition unit 114 recognizes the type of footwear based on the analysis results of the camera image of the toe detection unit 70-1 shown in Fig. 13(A). The type of footwear that has a higher match can be detected by, for example, pattern matching the feature information such as the shape, size, and color of the object obtained from the analysis results with the feature information associated with a predetermined type of footwear.

[0100] In addition, the footwear recognition unit 114 estimates the type of footwear from the sensor output range detected by the pressure-sensitive sensors arranged in a grid pattern in the toe detection unit 70-2, as shown in Fig. 12(b). Since vibrations from the vibration unit 36 ​​are not easily transmitted depending on the footwear, if the driver's footwear is not suitable for driving or is not easily transmitted by vibrations (e.g., high heels, platform shoes, etc.), the notification control unit 134 notifies the driver of a disclaimer, notifies that the vibration notification function is disabled, or prompts the driver to change into appropriate footwear by displaying an image or outputting audio via the display unit 32, speaker 34, etc. This makes it possible to raise the driver's awareness of safe driving.

[0101] [Activation or Stop of Vibration Control by the Action Recognition Unit 116] Next, the start or stop control of vibration control by the action recognition unit 116 will be specifically described. Fig. 14 is a diagram for explaining the recognition of the foot action of an occupant by the action recognition unit 116. In the example of Fig. 14, a pressure sensor is installed as a toe detection unit 70-2 on a floor mat FM laid on a floor portion FL. When the action recognition unit 116 detects, based on the detection result of the pressure sensor, that the occupant has tapped the floor mat FM with their feet (toes or heels) a predetermined number of times (for example, twice) within a predetermined time, it recognizes that a predetermined action has been performed.

[0102] In addition, when the vehicle has a toe detection unit 70-1 using a camera, the action recognition unit 116 analyzes the camera image to detect the movement of the occupant's toes, and recognizes that a specified action has been performed when, for example, the action of shaking the toes from side to side two or more times is detected.

[0103] When the motion recognition unit 116 recognizes that the occupant has performed a predetermined motion, the vibration control unit 132 controls the activation or deactivation of vibration control for the occupant (vibration control for the vibration unit installed on the floor unit FL under the seat where the occupant is seated). For example, if vibration control is activated, the vibration control is terminated, and if vibration control is terminated, the vibration control is activated.

[0104] This allows the occupant to intuitively stop or resume vibration control at will when the occupant finds the vibration notification annoying due to the surrounding environment, the occupant's state of mind, etc. Note that switching between starting and stopping vibration control may be controlled by operating a switch unit provided in the HMI 30 instead of the recognition result of the occupant's foot movement.

[0105] [Variations] In the embodiment, when the vehicle M is traveling on a road with vibrations (rough road), the vibration control unit 132 may stop vibration control. In this case, the vehicle M may refer to the map information 54 based on the position information of the vehicle M to determine whether the road on which the vehicle is traveling is a rough road such as a mountain road, or may determine whether the road is a rough road based on the detection result of the vibration sensor included in the vehicle sensor 40. This prevents the vibrations (road noise) when the vehicle M is traveling from mixing with the vibrations caused by the vibration unit 36, thereby preventing the occupants from feeling uncomfortable.

[0106] In addition, in the embodiment, the vibration control unit 132 may register the foot position for each occupant in advance, or may statistically acquire the foot positions, and determine the position of the vibration unit 36 ​​to vibrate among the multiple vibration units 36 corresponding to the occupant based on the registered foot positions. This reduces the processing load for detecting the foot positions. Furthermore, the vibration control unit 132 may register the vibration mode for each occupant in advance, and control the vibration mode of the vibration unit 36 ​​for each occupant based on the registered information. This allows the occupant to be notified by the vibration preferred by the occupant.

[0107] [Processing flow] Next, an example of processing executed by the driving assistance device 100 in the embodiment will be described using a flowchart. Note that the following example will mainly describe vibration control processing among the processing executed by the driving assistance device 100. The following processing may be repeatedly executed at a predetermined cycle or timing.

[0108] FIG. 15 is a flowchart showing an example of processing executed by the driving assistance device 100 according to the embodiment. In the example of FIG. 15, the surrounding situation recognition unit 112 recognizes the surrounding situation of the vehicle M (step S100). Next, the surrounding situation recognition unit 112 recognizes a target (e.g., an obstacle) present in the vicinity (step S110). Next, the risk determination unit 122 determines whether there is a risk of the vehicle M coming into contact with or approaching the target based on the relative position and relative speed between the vehicle M and the target (step S120). If it is determined that there is a risk of contact or approaching the target, the risk determination unit 122 detects a risk position and a risk amount where a risk is occurring for the vehicle M (step S130). Note that in the processing of step S130, instead of (or in addition to) the risk position and risk amount, a risk direction where a risk is occurring may be detected. Next, the toe detection unit 70 detects the driver's feet (step S140). Next, the vibration control unit 132 determines the vibration mode of the vibration unit 36 ​​(for example, the position of the vibration unit 36 ​​to be vibrated, the vibration magnitude, the vibration period, etc.) according to the risk position, the risk amount, the risk level, etc. (step S150). The target vibration unit 36 ​​is vibrated based on the determined vibration mode. This ends the processing of this flowchart. Also, if it is determined in the processing of step S120 that there is no risk of the vehicle M coming into contact with or approaching the target, this flowchart ends.

[0109] In the processing of steps S120 to S130 shown in FIG. 15, instead of (or in addition to) the above-described processing, the driving situation determination unit 124 may determine whether or not the vehicle M will deviate from the lane based on the distance between the vehicle M and the lane marking, the speed VM, the direction of movement, etc. (step S120), and if it is determined that the vehicle M will deviate, detect the direction of departure (step S130).

[0110] As described above, the driving assistance device of the embodiment includes a surrounding situation recognition unit 112 that recognizes the surrounding situation of the vehicle M, a plurality of vibration units 36 that provide vibration stimulation to the occupants of the vehicle M, and a vibration control unit 132 that vibrates at least one of the plurality of vibration units 36 based on the relative position between a target recognized by the surrounding situation recognition unit 112 and the vehicle M and the direction of the target relative to the vehicle M, and at least one of the plurality of vibration units 36 is installed in a position that can transmit vibrations to the soles of the feet of the occupants of the vehicle M, thereby enabling the transmission of more appropriate information to the occupants and assisting them in driving. This can therefore contribute to the development of sustainable transportation systems.

[0111] For example, according to the embodiment, by vibrating the floor portion, it is possible to realize a target notification that allows a seated occupant to intuitively understand the target position. For example, by directly transmitting a tactile stimulus by vibration to the driver's feet, etc., the position and moving direction of an obstacle can be intuitively recognized, which allows for quick pre-action to avoid a collision in a potentially dangerous situation (for example, shortening the search time spent on recognition due to improved intuition), and also reduces the burden on the driver because the notification by tactile stimulus allows for danger recognition without looking at the display unit.

[0112] Furthermore, according to the embodiment, the vibration unit is appropriately selected and the vibration magnitude (vibration intensity) and vibration period are controlled according to the driving conditions such as the risk level and the risk arrival direction (more specifically, the distance to the target object and the first time to contact TTC, etc.), so that the occupant can grasp the risk more intuitively. Furthermore, according to the embodiment, the position of the toes on the floor is recognized by a camera, and the coordinate position is determined from the output signal of a pressure sensor laid out in a grid pattern to identify the position, and the vibration unit is controlled according to the identified toe position, so that more appropriate vibration transmission can be realized.

[0113] Furthermore, according to the embodiment, by encouraging a driver who is wearing shoes that are not suitable for driving and that do not transmit vibrations well to change their shoes or by notifying that the function is disabled, it is possible to raise the driver's awareness of safe driving and ensure the reliability of the function. Furthermore, according to the embodiment, the on / off of vibration control is controlled in response to tapping or gesture movements on the floor, thereby improving convenience.

[0114] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Recognizes the vehicle's surroundings, vibrating at least one vibration unit among a plurality of vibration units that provide a vibration stimulus to an occupant of the vehicle based on a relative position between the vehicle and a target included in the recognized surrounding situation and a direction of the target relative to the vehicle; At least one of the plurality of vibration units is installed at a position capable of transmitting vibration to the soles of the feet of an occupant of the vehicle. Driving assistance device.

[0115] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0116] 10...camera, 12...radar device, 14...LIDAR, 16...object recognition device, 20...communication device, 30...HMI, 32...display unit, 34...speaker, 36...vibration unit, 38...foot switch unit, 40...vehicle sensor, 50...navigation device, 60...driver monitor camera, 70...toe detection unit, 80...driving operator, 82...steering wheel, 84...accelerator pedal, 86...brake pedal, 100...driving assistance device, 110...recognition unit, 112...surrounding situation recognition unit, 114...footwear recognition unit, 116...action recognition unit, 120...determination unit, 122...risk determination unit, 124...driving situation determination unit, 130...control unit, 132...vibration control unit, 134...notification control unit, 136...driving control unit, 150...memory unit, 200...driving drive force output device, 210...brake device, 220...steering device, M...vehicle

Claims

1. a surrounding situation recognition unit that recognizes the surrounding situation of the vehicle; a plurality of vibration units that provide vibration stimulation to occupants of the vehicle; a vibration control unit that vibrates at least one of the plurality of vibration units based on a relative position between the target and the vehicle recognized by the surrounding situation recognition unit and a direction of the target with respect to the vehicle, At least one of the plurality of vibration units is installed at a position capable of transmitting vibration to the soles of the feet of an occupant of the vehicle. Driving assistance device.

2. the vibration control unit determines a vibration mode of the plurality of vibration units in accordance with the relative position and the direction of the target. The driving assistance device according to claim 1 .

3. a toe detection unit for detecting the position of the toes of the occupant; the vibration control unit determines a vibration mode of the plurality of vibration units in accordance with the toe positions detected by the toe detection unit. The driving assistance device according to claim 1 .

4. a notification unit that notifies the occupant of information by at least one of a display and a sound, The occupants include a driver of the vehicle. The toe detection unit detects footwear of the driver, A notification control unit is further provided that notifies the driver via the notification unit when the driver's footwear is not suitable for driving. The driving assistance device according to claim 3 .

5. The vehicle further includes a motion recognition unit that recognizes a predetermined motion of the occupant's foot, the vibration control unit controls activation or deactivation of vibration control for the vibration unit in response to a predetermined tapping motion or a predetermined gesture motion by the occupant's foot recognized by the motion recognition unit. The driving assistance device according to claim 1 .

6. the notification control unit notifies the occupant via the notification unit based on the degree of contact or risk of contact between the vehicle and the target; The notification timing by the notification control unit and the vibration timing by the vibration control unit are controlled synchronously or in stages under predetermined conditions. The driving assistance device according to claim 4.

7. At least one of the plurality of vibration units is installed on a pedal operator of the vehicle, the vibration control unit controls vibration of the vibration unit installed on the pedal operator when the driving mode of the vehicle is a manual driving mode or when the driving mode is switched from an automatic driving mode to a manual driving mode. The driving assistance device according to claim 1 .

8. At least one of the plurality of vibration units is installed on a pedal operator of the vehicle, the vibration control unit performs vibration control on the vibration unit installed on the pedal operator when a foot of a driver of the vehicle comes into contact with the pedal operator. The driving assistance device according to claim 1 .

9. The computer Recognizes the vehicle's surroundings, vibrating at least one vibration unit among a plurality of vibration units that provide a vibration stimulus to an occupant of the vehicle based on a relative position between the vehicle and a target included in the recognized surrounding situation and a direction of the target relative to the vehicle; At least one of the plurality of vibration units is installed at a position capable of transmitting vibration to the soles of the feet of an occupant of the vehicle. Driving assistance methods.

10. On the computer, Recognize the vehicle's surroundings, vibrating at least one vibration unit among a plurality of vibration units that provide a vibration stimulus to an occupant of the vehicle based on a relative position between the vehicle and a target included in the recognized surrounding situation and a direction of the target relative to the vehicle; At least one of the plurality of vibration units is installed at a position capable of transmitting vibration to the soles of the feet of an occupant of the vehicle. program.

Images