Driving assistance device, driving assistance method, and storage medium

Abstract

The present application provides a driving assistance device, a driving assistance method, and a storage medium capable of assisting driving by transmitting more appropriate information to an occupant in accordance with the situation in the vicinity of a vehicle. A driving assistance device according to an embodiment is provided with: a peripheral situation recognition unit that recognizes a peripheral situation of a vehicle; a plurality of vibration units that apply a stimulus based on vibration to an occupant of the vehicle; and a vibration control unit that vibrates at least one of the plurality of vibration units on the basis of the relative position of the object target recognized by the surrounding situation recognition unit and the vehicle and the direction of the object target with respect to the vehicle. At least one of the plurality of vibration parts is provided at a position at which vibration can be transmitted to the foot sole of an occupant of the vehicle.

Description

Technical Field

[0001] This invention relates to a driving support device, a driving support method, and a storage medium. Background Technology

[0002] In recent years, efforts to provide sustainable transportation systems that also take into account vulnerable individuals among traffic participants have become increasingly active. Research and development related to driver assistance technologies are being undertaken to further improve traffic safety and convenience in order to achieve this. In this context, an information prompting system has been known in recent years, comprising: a vehicle information estimation unit that estimates vehicle information including at least the vehicle distance to other vehicles located ahead of the vehicle in the direction of travel; and a prompting unit that, based on the aforementioned vehicle information, provides a prescribed tactile stimulus to the driver's feet (e.g., Japanese Patent Application Publication No. 2020-131892). Summary of the Invention

[0003] Furthermore, in driver assistance technology, objects may approach the vehicle not only from the front but also from other directions. However, in previous technologies, only the area in front of the vehicle in the direction of travel was considered an object, which sometimes prevented the appropriate information from being conveyed to the occupants.

[0004] To address the aforementioned issues, one of the objectives of this application is to provide a driving support device, driving support method, and storage medium capable of delivering more appropriate information to occupants based on the vehicle's surrounding conditions to support driving. Furthermore, this will contribute to the development of sustainable transportation systems.

[0005] The driving support device, driving support method and storage medium involved in this invention adopt the following structure.

[0006] (1): One aspect of the vehicle control device of the present invention is a driving support device, wherein the driving support device includes: a surrounding condition recognition unit that recognizes the surrounding condition of the vehicle; a plurality of vibration units that apply vibration-based stimulation to the occupants of the vehicle; and a vibration control unit that vibrates at least one of the plurality of vibration units based on the relative position of an object target to the vehicle and the direction of the object target relative to the vehicle as recognized by the surrounding condition recognition unit, wherein at least one of the plurality of vibration units is disposed at a position capable of transmitting vibration to the feet of the occupants of the vehicle.

[0007] (2): Based on the above (1) scheme, the vibration control unit determines the vibration mode of the plurality of vibration units according to the relative position and the direction of the object target.

[0008] (3): Based on the above (1) scheme, the driving support device also includes a foot detection unit for detecting the position of the front of the occupant's feet, and the vibration control unit determines the vibration mode of the plurality of vibration units based on the position of the front of the feet detected by the foot detection unit.

[0009] (4): Based on the above (3) scheme, the driving support device further includes a notification unit that notifies the occupant of information via at least one of display and sound, the occupant including the driver of the vehicle, the foot detection unit detecting the driver's shoes, and the driving support device further includes a notification control unit that notifies the driver via the notification unit if the driver's shoes are unsuitable for driving.

[0010] (5): Based on the above (1) scheme, the driving support device also has an action recognition unit that recognizes a predetermined action performed by the occupant's feet, and the vibration control unit controls the start or stop of vibration control performed on the vibration unit by the predetermined tapping action or predetermined posture action performed by the occupant's feet recognized by the action recognition unit.

[0011] (6): Based on the above (4) scheme, the notification control unit notifies the occupant based on the risk level of contact or proximity between the vehicle and the target object. The timing of the notification by the notification control unit and the timing of the vibration by the vibration control unit are synchronized or controlled in stages under specified conditions.

[0012] (7): Based on the above (1) scheme, at least one of the plurality of vibration parts is provided on the pedal operating member of the vehicle, and the vibration control unit performs vibration control on the vibration part provided on the pedal operating member when the driving mode of the vehicle is manual driving mode or when switching from automatic driving mode to manual driving mode.

[0013] (8): Based on the above (1) scheme, at least one of the plurality of vibration parts is provided in the pedal operating member of the vehicle, and the vibration control unit performs vibration control on the vibration part provided in the pedal operating member when the driver's foot of the vehicle comes into contact with the pedal operating member.

[0014] (9): Another aspect of the vehicle control method of the present invention is a driving support method, wherein the driving support method causes a computer to perform the following processing: identify the surrounding conditions of the vehicle; based on the relative position of an object target included in the identified surrounding conditions to the vehicle and the direction of the object target relative to the vehicle, cause at least one of a plurality of vibrating parts to vibrate to apply vibration-based stimulation to the occupants of the vehicle; and at least one of the plurality of vibrating parts is disposed at a position capable of transmitting vibration to the feet of the occupants of the vehicle.

[0015] (10): Another aspect of the present invention relates to a storage medium storing a program, wherein the program causes a computer to perform the following processing: identifying the surrounding conditions of a vehicle; based on the relative position of an object target included in the identified surrounding conditions to the vehicle and the direction of the object target relative to the vehicle, causing at least one of a plurality of vibrating parts to vibrate to apply a vibration-based stimulus to the occupants of the vehicle; and at least one of the plurality of vibrating parts is disposed at a position capable of transmitting vibration to the feet of the occupants of the vehicle.

[0016] According to the schemes (1) to (10) above, more appropriate information can be transmitted to the occupants based on the surrounding conditions of the vehicle to support driving. Attached Figure Description

[0017] Figure 1 This is a structural diagram of a vehicle equipped with a driving support device according to the implementation method.

[0018] Figure 2 This is a diagram illustrating an example of the first vibration control method.

[0019] Figure 3 This is a diagram illustrating an example of the second vibration control method.

[0020] Figure 4 This is a diagram illustrating a specific example of the vibration mode in the second vibration control.

[0021] Figure 5 This is a diagram illustrating an example of third vibration control.

[0022] Figure 6 This is a diagram used to illustrate the detection position at the front of the foot.

[0023] Figure 7 This is a diagram illustrating an example of the fourth vibration control method.

[0024] Figure 8 This is a diagram illustrating an example of the fifth vibration control method.

[0025] Figure 9 This is a diagram illustrating an example of the sixth vibration control method.

[0026] Figure 10 This is a diagram illustrating an example of the seventh vibration control.

[0027] Figure 11 This is a diagram showing a first example of the installation of the vibrating part.

[0028] Figure 12 This is a diagram showing a second example of the vibrating section configuration.

[0029] Figure 13 This is an example of shoe inspection.

[0030] Figure 14 This diagram illustrates how the motion recognition unit recognizes the movements of an occupant's feet.

[0031] Figure 15 This is a flowchart illustrating an example of the processing performed by the driving support device in the embodiment. Detailed Implementation

[0032] Hereinafter, embodiments of the driving support device, driving support method, and storage medium of the present invention will be described with reference to the accompanying drawings.

[0033] [Overall Structure]

[0034] Figure 1 This is a structural diagram of a vehicle M equipped with the driving support device according to the implementation method. Vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a secondary battery or fuel cell.

[0035] The vehicle M is equipped with, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) device 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, a driver monitoring camera 60, a foot detection unit 70, driving controls 80, a driving support device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. Figure 1 The structure shown is just one example; a part of the structure can be omitted, or other structures can be added. HMI30 is an example of a "notification department".

[0036] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Camera 10 can be mounted anywhere on the vehicle M. When shooting forward, camera 10 can be mounted on the upper part of the windshield, behind the rearview mirror inside the vehicle, etc. Camera 10 can periodically and repeatedly take pictures of the surroundings of the vehicle M. Camera 10 can also be a stereo camera.

[0037] Radar device 12 radiates millimeter-wave and other radio waves around the vehicle M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and orientation) of the objects. Radar device 12 can be installed at any part of the vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0038] The LIDAR14 illuminates the periphery of vehicle M with light (or electromagnetic waves with wavelengths close to light) and measures the scattered light. The LIDAR14 determines the distance to the object based on the time from the emission of light to the reception of light. The illuminating light can be, for example, a pulsed laser. The LIDAR14 can be mounted at any location on vehicle M.

[0039] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the cameras 10, radar device 12, and LIDAR 14 to identify the position, type, speed, etc., of an object (object target). The object recognition device 16 outputs the recognition results to the driving support device 100. The object recognition device 16 can also directly output the detection results from the cameras 10, radar device 12, and LIDAR 14 to the driving support device 100. Alternatively, the object recognition device 16 can be omitted from the vehicle M. Some or all of the cameras 10, radar device 12, LIDAR 14, and object recognition device 16 are examples of "external detection devices".

[0040] The communication device 20 uses networks such as cellular networks, Wi-Fi networks, Bluetooth (registered trademark), and DSRC (Dedicated Short Range Communication) to communicate with other vehicles in the vicinity of vehicle M, or communicates with various server devices via wireless base stations.

[0041] The HMI 30 provides various information to the occupants of vehicle M and accepts input operations performed by the occupants. The HMI 30 includes, for example, a display unit 32, a speaker 34, and a vibration unit 36. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may also be integrated with the input unit as a touch panel. The speaker 34 outputs a prescribed sound (e.g., an alarm).

[0042] The vibration unit 36 ​​applies vibration-based stimulation to the occupant, for example, based on instructions from the driver support device 100. Multiple vibration units 36 are available, for example, located in the seat or at positions capable of applying vibration to the occupant's feet. Positions capable of applying vibration to the occupant's feet include, for example, the floor (base plate) within the vehicle interior, pedal controls (accelerator pedal, brake pedal), and footrests. The vibration unit 36 ​​may also be located in the steering wheel 82 included in the driver control unit 80, or in the seatbelt during use. The vibration units 36 may also be arranged at predetermined intervals.

[0043] The vibration unit 36 ​​may use, for example, an LRA (Linear Resonant Actuator), which is a type of voice coil motor, but it is not limited to the above-mentioned example as long as it can transmit vibration-based sensory stimuli to the driver. Therefore, an eccentric motor, a linear motor, a vibrating speaker, etc., may also be used as the vibration unit 36.

[0044] In addition to the display unit 32, speaker 34, and vibration unit 36, the HMI 30 may also include a microphone, buzzer, touch panel, buttons, etc. For example, the HMI 30 may include a switch that allows the driver of vehicle M to switch the driving state (driving control content) of vehicle M through operation.

[0045] Vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting yaw rate (e.g., the rotational angular velocity about a vertical axis passing through the center of gravity of vehicle M), a lateral acceleration sensor (lateral G sensor) for detecting the lateral acceleration (lateral G) of vehicle M, an orientation sensor for detecting the orientation of vehicle M, and a steering angle sensor for detecting the steering angle of vehicle M (which can be the angle of the steering wheel or the operating angle of the steering wheel). Vehicle sensor 40 may also be equipped with a position sensor for detecting the position of vehicle M. The position sensor may be, for example, a sensor that obtains position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be a sensor that obtains position information using a GNSS (Global Navigation Satellite System) receiver 51 of navigation device 50. Vehicle sensor 40 may also include a vibration sensor for detecting vibrations from the road surface traveled by vehicle M.

[0046] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented using INS (Inertial Navigation System) output from the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, buttons, etc. The navigation HMI 52 can also be partially or entirely shared with the aforementioned HMI 30. The route determination unit 53, for example, refers to the map information 54 to determine the route (hereinafter referred to as the map path) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52. The map information 54, for example, represents road shape information by indicating road segments and nodes connecting the road segments. The map information 54 may also include POI (Point of Interest) information, etc. Map information 54 may include, for example, information about the center of a lane, or lane boundary information such as road dividing lines (hereinafter referred to as dividing lines). Map information 54 may also include road information such as the radius of curvature (or curvature) of the road (or each lane contained in the road), slope, and width, as well as traffic restriction information, address information (address and postal code), facility information, and telephone number information. Map information 54 can be updated at any time by communicating with other devices through communication device 20. Map information 54 can be stored in the storage unit within the driver support device 100.

[0047] The navigation device 50 can also provide route guidance using the navigation HMI 52 based on the path on the map. The navigation device 50 can also be implemented using the functions of a terminal device such as a smartphone or tablet held by the occupant. The navigation device 50 can also send its current location and destination to the navigation server via the communication device 20 and obtain the path equivalent to the path on the map from the navigation server.

[0048] The driver monitoring camera 60 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS sensor. The driver monitoring camera 60 is installed in any part of the vehicle interior capable of capturing images of the driver, including the driver seat in the vehicle M. Using the images captured by the driver monitoring camera 60, it is possible to obtain the driver's facial orientation, hand position, and further, the positions of the seats occupied by other occupants. The driver monitoring camera 60 outputs the captured images to the driver support device 100.

[0049] The forefoot detection unit 70, for example, detects the position of the forefoot of the occupant (e.g., the part forward from the ankle or heel). Specific examples of the forefoot detection unit 70 are described later.

[0050] The driving controls 80 include, for example, a steering wheel 82, an accelerator pedal 84, a brake pedal 86, a steering indicator switch, a gear shift lever, and other controls. Sensors are installed on the driving controls 80 to detect the amount of operation or the presence or absence of operation, and the detection results are output to some or all of the following: the driving support device 100, the driving force output device 200, the braking device 210, and the steering device 220. The steering wheel 82 is an example of a "steering control". The accelerator pedal 84 and the brake pedal 86 are examples of "pedal controls".

[0051] For example, the steering wheel 82 is equipped with a steering wheel sensor (SW sensor) 82A and a vibration unit 36 ​​that vibrates the part held by the driver. The SW sensor 82A detects whether the driver is in contact with the steering wheel 82. The SW sensor 82A detects the amount of steering wheel 82 operation (torque (also called steering torque), steering amount, steering change rate) that changes according to the driver's operation of the steering wheel 82 (hereinafter referred to as steering operation). The SW sensor 82A can also detect whether the driver is holding the steering wheel 82. The steering wheel 82 does not necessarily have to be ring-shaped; it can also be an irregularly shaped steering wheel, a lever, a button, etc. In this case, the SW sensor 82A detects the amount of operation corresponding to each shape.

[0052] The accelerator pedal 84 is equipped with an accelerator pedal sensor (AP sensor) 84A. The AP sensor 84A detects whether the driver is placing his foot on the accelerator pedal 84, whether the driver's operation of the accelerator pedal 84 is activated or deactivated, and the amount of operation of the accelerator pedal 84 (opening change amount, opening change rate) that changes according to the operation.

[0053] A brake pedal sensor (BP sensor) 86A is provided on the brake pedal 86. The BP sensor 86A detects whether the driver is placing his foot on the brake pedal 86, the activation or deactivation of the driver's operation of the brake pedal 86 (hereinafter referred to as braking operation), and the amount of brake pedal 86 operation (opening change amount, opening change rate) that changes according to the operation. Accelerator operation and braking operation are examples of "speed operation".

[0054] The driving force output device 200 outputs driving force (torque) for the vehicle M to drive to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the above structure according to information input from the driving support device 100 or from the driving operation device 80.

[0055] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and an ECU. The ECU controls the electric motor according to information input from the driving support device 100 or from the driving operation unit 80, so that braking torque corresponding to the braking operation is output to each wheel. The braking device 210 may have a backup mechanism for transmitting hydraulic pressure generated by the operation of the brake pedal 86 included in the driving operation unit 80 via the master hydraulic cylinder to the hydraulic cylinder. The braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder by controlling the actuator according to information input from the driving support device 100.

[0056] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving support device 100 or from the driving control unit 80.

[0057] [Driver Support Device]

[0058] The driving support device 100 includes, for example, an identification unit 110, a determination unit 120, a control unit 130, and a storage unit 150. The identification unit 110, determination unit 120, and control unit 130 are implemented, for example, by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented using hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System on Chip), or through the coordinated use of software and hardware. The program can be pre-stored in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium) of the driving support device 100, or stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving support device 100 by mounting the storage medium (a non-transitory storage medium) to a drive unit.

[0059] For example, settings are incorporated within the driving force output device 200, braking device 210, and steering device 220 to prioritize the execution of instructions from the driving support device 100 to the driving force output device 200, braking device 210, and steering device 220 over detection results from the driving operation unit 80. Regarding braking, it can also be configured to prioritize the braking force obtained based on the amount of brake pedal operation compared to the instructions from the driving support device 100. As a mechanism for prioritizing the execution of instructions from the driving support device 100, communication priority within the vehicle's LAN (Local Area Network) can also be used.

[0060] The storage unit 150 can also be implemented using various storage devices described above, or SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 150 stores, for example, programs and other various information. The storage unit 150 can also store the map information 54 described above.

[0061] The recognition unit 110 includes, for example, a surrounding condition recognition unit 112, a shoe recognition unit 114, and a motion recognition unit 116. The surrounding condition recognition unit 112 recognizes the surrounding conditions of the vehicle M based on information input from external detection devices. For example, the surrounding condition recognition unit 112 recognizes the position, speed, acceleration, and other states of objects (objects) existing in the surrounding area (e.g., within a predetermined distance (first predetermined distance) from the vehicle M). Objects may include other vehicles, bicycles, pedestrians, and other traffic participants, curbs, medians, guardrails, and other road structures. The position of the object is recognized, for example, as its position on absolute coordinates with a representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, and is used for control. The position of the object may be represented by a representative point such as the object's center of gravity or corners, or by a region. The "state" of the object may also include the object's acceleration, jerk, or "action state" (e.g., whether it is changing lanes or about to change lanes) when the object is a moving body. The surrounding situation recognition unit 112 recognizes the relative position and relative speed of the target object.

[0062] The surrounding condition recognition unit 112 identifies, for example, the lane in which vehicle M is traveling (driving lane). For instance, the surrounding condition recognition unit 112 performs known analytical processing (e.g., edge extraction, feature extraction, pattern matching, etc.) on an image captured by camera 10 (hereinafter referred to as a camera image), and identifies the position and pattern (e.g., arrangement of solid and dashed lines) of the dividing lines surrounding vehicle M based on the analytical results. Dividing lines are also an example of object targets. The surrounding condition recognition unit 112 can also identify the position and pattern of the dividing lines surrounding vehicle M by referring to map information 54 based on the position information of vehicle M. The surrounding condition recognition unit 112 can also identify the driving lane using at least one of the position and pattern of the dividing lines obtained from the camera image and the position and pattern of the dividing lines obtained from the map information. The surrounding condition recognition unit 112 is not limited to identifying dividing lines; it can also identify driving lanes by identifying driving road boundaries (road boundaries) including shoulders, curbs, median strips, guardrails, etc. In this identification, the position of vehicle M obtained from navigation device 50 and the processing results from INS may also be taken into consideration. The surrounding condition recognition unit 112 can also recognize adjacent lanes to the driving lane. The surrounding condition recognition unit 112 can also recognize the radius of curvature (or curvature), slope, width, etc., of the driving lane (or road) based on at least one of camera images and map information. Based on the surrounding condition recognition results, the surrounding condition recognition unit 112 identifies obstacles, temporary stop lines, red lights, toll booths, and other road phenomena. Obstacles include objects that impede travel in the driving lane and objects that need to be avoided (traffic participants, etc.). These objects are also included in object targets.

[0063] The surrounding condition recognition unit 112 can also recognize the position and posture of vehicle M relative to the driving lane. For example, the surrounding condition recognition unit 112 can recognize the deviation of vehicle M's reference point from the center of the lane, and the angle formed by vehicle M's direction of travel relative to the line connecting the centers of the lanes, as the relative position and posture of vehicle M relative to the driving lane. Alternatively, the surrounding condition recognition unit 112 can recognize the position of vehicle M's reference point relative to any side end (dividing line or road boundary) of the driving lane, as the relative position of vehicle M relative to the driving lane. The surrounding condition recognition unit 112 can also recognize the position and posture of other vehicles traveling in vehicle M's driving lane, or recognize whether other vehicles are located on the center side of the driving lane or on the dividing line side when viewed from vehicle M.

[0064] The shoe recognition unit 114 identifies the shoes worn by the driver of vehicle M. For example, the shoe recognition unit 114 identifies the type of shoes worn by the driver based on the detection results detected by the forefoot detection unit 70. Details regarding the function of the shoe recognition unit 114 will be described later.

[0065] The motion recognition unit 116 identifies the actions performed by the occupant's feet (e.g., tapping actions, postural actions) based on the detection results detected by the foot front detection unit 70. Details regarding the function of the motion recognition unit 116 will be described later.

[0066] The determination unit 120 includes, for example, a risk determination unit 122 and a driving condition determination unit 124. The risk determination unit 122 determines, based on the identification results obtained by the surrounding condition recognition unit 112, whether there is a risk (possibility) of the vehicle M coming into contact with an object located in the vicinity of the vehicle M. If a risk is determined to exist, the risk determination unit 122 determines the direction of the risk relative to the vehicle M (e.g., front, rear, right, left, etc.).

[0067] For example, the risk assessment unit 122 calculates a first time to collision (TTC) before the vehicle M comes into contact with the target object based on the relative distance and relative speed between the vehicle M and the target object. The first time to collision (TTC) is calculated, for example, by dividing the relative distance by the relative speed. Furthermore, if the calculated first time to collision (TTC) is within a predetermined time, the risk assessment unit 122 determines that there is a risk of the vehicle M coming into contact with the target object; if the first time to collision (TTC) is greater than the predetermined time, it determines that there is no risk of contact. The risk assessment unit 122 can also determine whether there is a risk that the vehicle M will approach the target object to within a predetermined distance, instead of the risk of contact. In this case, the predetermined time can be set to a value greater than the time for determining contact, or the distance (relative distance) between the vehicle M and the target object can be used as a reference. Therefore, not only can the possibility of contact between the driver of the vehicle M and the target object be conveyed, but also the presence of a target object nearby (in other words, the presence of an approaching target), thus further improving safety. The risk assessment unit 122 can also determine the degree of risk if it determines that there is a possibility that the vehicle M may come into contact with or approach the target object. In this case, the smaller the first margin of error (TTC) (or relative distance), the greater the degree of risk.

[0068] The driving condition determination unit 124 determines the driving condition of vehicle M based on the recognition result identified by the surrounding condition recognition unit 112. For example, the driving condition determination unit 124 determines whether there is a possibility that vehicle M may leave the driving lane based on the recognition result. For example, if there is a possibility that vehicle M may leave the driving lane at a reference position (e.g., end, center of gravity, center) by crossing any of the left and right dividing lines of the driving lane identified by the recognition unit 110 (passing on the dividing line) and thus exceeding the driving lane, the driving condition determination unit 124 determines that there is a possibility that vehicle M may leave the driving lane; if there is no possibility that vehicle M may leave the driving lane, the driving condition determination unit 124 determines that there is no possibility that vehicle M may leave the driving lane.

[0069] The driving condition determination unit 124 derives the predicted future path of vehicle M based on the speed and yaw rate of vehicle M obtained from vehicle sensor 40, and calculates the second margin time TTLC (Time to Line Crossing) (=d / VM) before vehicle M reaches the dividing line based on the distance between the derived predicted path and the dividing line (departure path length d) and the speed VM. Then, if the second margin time TTLC is less than a predetermined time, the driving condition determination unit 124 determines that there is a possibility that vehicle M may leave the driving lane; if the second margin time TTLC is greater than or equal to the predetermined time, it determines that there is no possibility of departure. The driving condition determination unit 124 may also determine the magnitude of the probability of departure if it determines that there is a possibility that vehicle M may leave the driving lane. In this case, the smaller the second margin time TTLC, the greater the probability of departure.

[0070] In addition to the functions described above, the determination unit 120 can also determine whether the shoes identified by the shoe recognition unit 114 are suitable for driving, or determine whether the action performed by the occupant's feet is a prescribed action based on the recognition result identified by the action recognition unit 116.

[0071] The control unit 130 controls various functions and devices of the vehicle M. The control unit 130 includes, for example, a vibration control unit 132, a notification control unit 134, and a driving control unit 136. The vibration control unit 132 controls the vibration of at least one of a plurality of vibration units 36 based on the determination results made by the risk determination unit 122 and the driving condition determination unit 124. In this case, the vibration control unit 132 determines the vibration pattern of the plurality of vibration units based, for example, on the relative position to the object target and the direction of the object target. The vibration pattern includes, for example, at least one of the following: the position of the vibrating vibration unit, the magnitude of the vibration (vibration intensity), and the vibration period (e.g., including frequency, mode, etc.). The vibration control unit 132 may also control the vibration of the vibration unit 36 ​​based on the recognition results recognized by the shoe recognition unit 114 and the recognition results recognized by the motion recognition unit 116.

[0072] The notification control unit 134 notifies the occupants of vehicle M of prescribed information, such as information obtained from communication device 20, HMI 30, vehicle sensor 40, driver monitoring camera 60, etc.; information detected by SW sensor 82A, AP sensor 84A, BP sensor 86A; identification results identified by identification unit 110; and determination results determined by determination unit 120. The prescribed information includes, for example, information related to the state of vehicle M and information related to driving control, which are relevant to the movement of vehicle M. Information related to the state of vehicle M includes, for example, the speed of vehicle M, engine speed, and gear position. Information related to driving control includes, for example, the type of driving control (driving state) being executed, the reason for the driving control operation, the reason for the vibration control operation, the status of the driving control, and information indicating that driving control has started or ended. Information related to driving control may include warnings to the driver (e.g., exit warning), reminders for prescribed driving operations, and attention-awakening information. The specified information may include the current location of vehicle M, destination, information related to remaining fuel, etc., or information unrelated to the driving control of vehicle M, such as television programs, entries stored on storage media such as DVDs (e.g., movies).

[0073] For example, the notification control unit 134 can generate an image including the aforementioned information and display the generated image on the display unit 32 of the HMI 30. It can also generate an audio signal representing the specified information and output the audio signal from the speaker 34 of the HMI 30. The timing of the audio output may include, for example, the start or stop of driving control, receiving information, switching the displayed image, or when the vehicle M reaches a specified state. The notification control unit 134 can also control the system to notify the occupants in stages, either synchronously with or including vibration control performed by the vibration control unit 132.

[0074] The driving control unit 136 performs driving control on at least one of the vehicle M's speed and steering based on the recognition result identified by the recognition unit 110. For example, if the driving condition determination unit 124 determines that there is a possibility that the vehicle M may leave the driving lane, the driving control unit 136 performs control to at least control the steering device 220 to prevent the vehicle M from leaving the driving lane. If the risk determination unit 122 determines that there is a possibility that the vehicle M may come into contact with an object, the driving control unit 136 performs driving control to control at least one of the braking device 210 or the driving drive force output device 200 and the steering device 220 to prevent the vehicle M from coming into contact with an object.

[0075] The driving control unit 136 can also perform driving controls such as ACC (Adaptive Cruise Control System) control, which keeps the vehicle M at a preset speed (set speed) within the driving lane, and ALC (Auto Lane Change) control, which changes lanes by at least manipulating the steering of the vehicle M, based on the recognition results identified by the surrounding situation recognition unit 112, the driver's instructions from the HMI 30, etc. The aforementioned driving controls include fully automated driving, in which driving controls that regulate the speed and steering of the vehicle M are performed without relying on the driver's operation.

[0076] [Regarding vibration control]

[0077] Next, details regarding the vibration control performed by the vibration control unit 132 will be explained with several examples. The following explanation will primarily focus on vibration control performed on the driver seat, but the same vibration control can also be performed on other occupants in other seats of the vehicle M. The occupant's seat location can be determined, for example, by performing known person recognition processing on camera images captured by the driver monitoring camera 60.

[0078] <First Vibration Control>

[0079] Figure 2 This is a diagram illustrating an example of the first vibration control method. In Figure 2 In the example, a schematic diagram is shown when viewing the area near the driver's seat from above. Figure 2 In the diagram, the X-axis represents the frontal direction of vehicle M, and the Y-axis represents the lateral direction of vehicle M. Figure 2 In the example, the floor section FL (an example of an area capable of transmitting vibrations to the driver's feet) is schematically shown, along with the driver's seat section ST, the driver's left foot LF, and the position of the driver's right foot RF (the position of the front of the foot) placed on the floor section FL. The seat section ST has a seat surface ST1 and a seat back section ST2. Figure 2 In the example, a vibration unit 36-1 is provided in the center of the front of the floor panel FL, and vibration units 36-2 and 36-3 are also provided on the left and right sides of the seat back panel ST2. For example, when driving control of vehicle M, which controls the speed of vehicle M independently of the occupant's operation, is executed, the driver's feet may become... Figure 2 It is positioned on the base plate FL as shown.

[0080] exist Figure 2In this structure, for example, if the risk assessment unit 122 determines that there is a risk of contact with an object in front of the vehicle M, the vibration control unit 132 causes the vibration unit 36-1 to vibrate. This transmits the vibration from the vibration unit 36-1 to the soles of the driver's left foot LF and right foot RF, which are positioned on the floor FL. The driver receives vibration-based stimulation from the soles of their feet (especially the front side), thus enabling them to perceive a risk in front of the vehicle M. In the first vibration control, if the risk assessment unit 122 determines that there is a risk of contact with an object behind the vehicle M, the vibration control unit 132 causes the vibration units 36-2 and 36-3 to vibrate. This allows the driver to feel the vibration from their back, which is in contact with the seat back ST2, thus enabling them to perceive a risk behind them. This allows for more accurate notification of the surrounding situation to the occupants. If a risk is determined to exist to the left rear of vehicle M, the vibration control unit 132 can vibrate only the vibration unit 36-2; if a risk is determined to exist to the right rear, the vibration control unit 132 can vibrate only the vibration unit 36-3. This allows the driver to more accurately determine which side (left or right) of the rear is at risk.

[0081] The vibration control unit 132 can also change the vibration pattern, such as the magnitude and period, based on the level of risk determined by the risk assessment unit 122. For example, the greater the risk level, the more the vibration control unit 132 increases the vibration or shortens the period. The vibration control unit 132 can also change the vibration pattern based on the direction of the risk. For example, the closer the direction of the risk is to the direction of travel of the vehicle M, the more the vibration control unit 132 increases the vibration or shortens the period. By changing the vibration pattern of the vibration unit according to the surrounding conditions, the driver can more intuitively grasp the risk. For example, the driver can intuitively grasp the direction of the risk and the phenomenon of moving away from the risk by observing the intensity of the vibration.

[0082] In the first vibration control, the vibration control unit 132, instead of relying on the judgment result determined by the risk assessment unit 122 (or based thereon), also controls the vibration of the vibrating part closer to the side of the vehicle deviating from the driving lane (marking line) if the driving condition assessment unit 124 determines that the vehicle M has deviated from the driving lane. The vibration control unit 132 can also change the vibration mode of the vibrating part 36 according to the likelihood of deviating. As a result, the driver can more appropriately grasp the driving condition.

[0083] <Second Vibration Control>

[0084] Figure 3 This is a diagram illustrating an example of the second vibration control method. Figure 3 Examples and Figure 2The difference from the previous example is that the vibration unit 36-4 and vibration unit 36-5 are respectively provided on the left and right sides of the front of the base plate FL, instead of in the center. Also, the seat back part ST2 does not have a vibration unit 36. In the second vibration control, the vibration control unit 132 controls the vibration mode of each of the vibration units 36-4 and 36-5 according to the direction of the risk determined by the risk determination unit 122.

[0085] Figure 4 This is a diagram illustrating a specific example of the vibration mode in the second vibration control. In Figure 4 In the example, a vehicle M traveling towards the T-junction and a pedestrian OB1 (an example of an object target) approaching the vehicle M from the left when viewed from the vehicle M within the T-junction. Here, the vehicle M is traveling at speed VM, and the pedestrian OB1 is moving at speed Vob1.

[0086] In this case, the risk assessment unit 122 determines whether there is a risk (possibility) of contact between the vehicle M and the pedestrian OB1 based on the relative position and relative speed (speed VM - speed Vob1) of the pedestrian OB1 relative to the vehicle M. If a contact risk is determined to exist, the vibration control unit 132 only vibrates the vibration unit 36-4 that is closer to the direction in which the pedestrian OB1 is located when viewed from the vehicle M. As a result, when the driver places both feet near the center of the floor section FL, the left foot LF receives a strong vibration, thus enabling a more accurate assessment of the object (obstacle) approaching from the left front (a risk has occurred from the left front). The vibration control unit 132 may also vibrate both vibration units 36-4 and 36-5 instead of only vibrating vibration unit 36-4. In this case, the vibration control unit 132 controls the vibration pattern so that the vibration of the vibration unit 36-4, which is closer to the direction in which the risk has occurred, is greater (and / or has a shorter vibration period) than the vibration of the other vibration units 36-5. Therefore, similar to the first vibration control, it can more accurately enable the driver to grasp the existence of a risk and the direction in which the risk exists.

[0087] In the second vibration control, the vibration control unit 132 can also replace the judgment result made by the risk assessment unit 122 (or based on it) and, if it is determined that the vehicle M has left the lane, control the vibration of the vibration unit closer to the side that has left the lane. The vibration control unit 132 can also change the vibration mode of the vibration unit 36 ​​according to the degree of risk and the likelihood of leaving the lane. As a result, the driver can more appropriately grasp the driving situation.

[0088] <Third Vibration Control>

[0089] Figure 5This diagram illustrates an example of the third vibration control. The vehicle M in this embodiment can be driven both manually and automatically. Therefore, the driver's foot may be positioned forward, backward, left, or right of the center of the floor plate FL, or offset to the left, right, forward, or backward of each foot, or with feet overlapping and only one foot resting on the floor plate. Therefore, in the third vibration control, the vibration patterns of the multiple vibration units 36 are controlled based on the position of the feet on the floor plate FL.

[0090] exist Figure 5 In the example, with Figure 4 Similarly, vibration units 36-4 and 36-5 are respectively provided on the left and right sides in front of the base plate FL. Figure 5 In the example, the illustration of the seat section ST is omitted. Figure 5 In the example, the driver's feet are positioned on the bottom plate FL, further to the lower right (rear right) of the center in the diagram. In such a scenario, if the risk assessment unit 122 determines that there is a risk of contact with an object, and / or the driving condition assessment unit 124 determines that there is a possibility that the vehicle M may leave the driving lane (dividing line), the vibration unit 36-4, located further from the feet (left foot LF, right foot RF) among the vibration units 36-4 and 36-5, vibrates with a stronger (larger) vibration compared to the vibration unit 36-5. This ensures reliable transmission of vibration to both feet.

[0091] The vibration control unit 132 can adjust the vibration magnitude and period based on the distance from the feet (left foot LF, right foot RF), and can also adjust the vibration magnitude and period based on the level of risk and the likelihood of disengagement. Furthermore, the vibration control unit 132 can increase the vibration or shorten the vibration period when only one foot is placed on the vibrating base plate FL, compared to when both feet are placed on the vibrating base plate FL. Therefore, even with only one foot, vibration can be reliably transmitted to the sole of the foot.

[0092] [Forefoot Detection Section]

[0093] Here, the accompanying drawings are used to illustrate the method by which the foot detection unit 70 detects the front of the driver's (and other passengers') feet. Figure 6 This is a diagram used to illustrate the detection position at the front of the foot. Figure 6 In example (A), a camera is used as the foot front detection unit 70-1. The camera is positioned to capture images of an area including the location where the driver's foot is placed (e.g., above the floor panel FL) and captures images at a predetermined period. The captured camera images are processed by known image analysis techniques to perform object recognition and other operations to detect the position of the foot (left foot LF, right foot RF).

[0094] exist Figure 6 In example (B), multiple pressure sensors are arranged in the sole plate FL as the foot forefoot detection unit 70-2. The pressure sensors are arranged in a grid pattern on the upper surface of the sole plate, and the position of the forefoot is detected by using the points where the sensors detect pressure exceeding a specified amount as coordinate points. The pressure sensors can be installed on the sole plate blanket or inside the sole plate FL.

[0095] exist Figure 6 In example (C), a vibration meter is used as the foot forefoot detection unit 70-3 to measure the magnitude of vibration. The vibration meter is, for example, installed at the location where the vibration unit 36 ​​is installed. The foot forefoot detection unit 70-3 detects the magnitude of vibration from the vibration unit 36 ​​as feedback and infers the foot position based on the detection result. For example, if the vibration unit 36 ​​causes the sole plate FL to vibrate and the foot (left foot LF, right foot RF) is close to the vibration unit 36, a foot-based load is applied to the sole plate FL, making it difficult for the sole plate FL to vibrate. That is, the closer the foot is to the vibration unit 36, the weaker the vibration measured by the foot forefoot detection unit 70-3 (vibration meter), and therefore the foot forefoot detection unit 70-3 detects the foot position based on the measured vibration.

[0096] exist Figure 6 In example (D), a camera is used as the forefoot detection unit 70-4. Figure 6 In example (D), an image including the driver's knee (KN) is captured, and the position of the forefoot is inferred based on the position of the knee (KN) contained in the analysis results of the captured camera image. Thus, for example, even in situations where the forefoot cannot be captured (e.g., the driver is wearing a long skirt or trousers and the forefoot cannot be captured by the camera), the position of the foot can be inferred based on the position of the knee (KN).

[0097] The forefoot detection unit 70 can also detect the forefoot using one of the methods described above, which are employed by the forefoot detection units 70-1 to 70-4. The forefoot detection unit 70 can also replace the forefoot detection units 70-1 to 70-4 and (or based thereon) use a radar device. In this case, the radar device emits millimeter-wave or other radio waves towards the base plate FL and detects the radio waves (reflected waves) reflected by objects such as the foot to detect the foot's position (distance and orientation). The forefoot detection unit 70 can also use the methods described above to detect when the forefoot is positioned on the pedal operating member. In this case, by using a camera to capture an image that also includes the pedal operating member, or by providing a pressure-sensitive sensor on the pedal operating member, it is possible to detect when the forefoot is positioned on the pedal operating member.

[0098] <Fourth Vibration Control>

[0099] Figure 7 This diagram illustrates an example of the fourth vibration control method. In the fourth vibration control, such as... Figure 7 As shown, nine vibration units 36-7 to 36-15 are arranged in a grid pattern (3 in the front-to-back direction × 3 in the left-to-right direction) on the base plate FL. The intervals between the vibration units 36-7 to 36-15 can be equal or different in the left-to-right and front-to-back directions. Furthermore, the number and arrangement are not limited to... Figure 6 Examples. For example, in... Figure 7 As shown in (A), if the front of both feet (left foot LF, right foot RF) is placed slightly forward of the center of the base plate FL, and the risk assessment unit 122 determines that there is a possibility of contact with an object (obstacle) located to the left front of the vehicle M, the vibration control unit 132 causes the vibration unit 36-7 located to vibrate at a position slightly forward to the left of the feet. Figure 7 In the case of the foot position shown in (A), if an object that poses a risk of contact is located in front of vehicle M, vibration unit 36-8 vibrates; if an object that poses a risk of contact is located to the right front, vibration unit 36-9 vibrates. If a risk of contact with an object is determined to exist on the left side of vehicle M, vibration control unit 132 causes vibration unit 36-10 to vibrate; if a risk of contact is determined to exist on the right side of vehicle M, vibration control unit 132 causes vibration unit 36-12 to vibrate. If a risk of contact is determined to exist to the left rear of vehicle M, vibration control unit 132 causes vibration unit 36-13 to vibrate; if a risk of contact is determined to exist ...

[0100] like Figure 7 As shown in (B), when both feet are located to the right rear of the base plate FL and the risk assessment unit 122 determines that there is a risk of contact with an object (obstacle) to the left front of the vehicle M, the vibration control unit 132 causes the vibration unit 36-11 located at the position closest to the left front of the feet to vibrate.

[0101] Here, we assume that vibrations are transmitted to the feet of both the driver, seated in the driver's seat, and the passenger, seated in the front passenger seat located next to the driver's seat. The driver's feet are positioned... Figure 7 As shown in (A), the feet of the passenger seat occupant are in the position indicated. Figure 7The position of (B). In this situation, if the risk assessment unit 122 determines that there is a risk of contact with an object target at the left front of the vehicle M, the vibration control unit 132 causes the vibrating unit 36-7 to vibrate at the floor panel FL of the driver's seat and the vibrating unit 36-11 to vibrate at the floor panel FL of the passenger seat. As a result, the vibrating unit can vibrate in accordance with the position of the feet of each of the multiple occupants in the vehicle M, allowing each occupant to more accurately grasp the surrounding situation.

[0102] <Fifth Vibration Control>

[0103] Figure 8 This diagram illustrates an example of the fifth vibration control method. In the fifth vibration control method, [the following is a description of the method, which is not directly related to the previous sentence]: Figure 7 Similarly, nine vibration units 36-7 to 36-15 (three in the front-to-back direction and three in the left-to-right direction) are provided in the area of ​​the floor section FL. In the fifth vibration control, a case is shown where the driving condition determination unit 124 determines that there is a possibility that the vehicle M may leave lane L1 while traveling at speed VM in lane L1 divided by left and right dividing lines RS1 and RS2. In this case, the vibration control unit 132, because there is a possibility that the vehicle M may leave the lane to the left, causes the vibration unit 36-10, located to the left of the driver's feet (left foot LF, right foot RF), to vibrate. The vibration control unit 132 may also cause the left-side vibration units 36-7, 36-10, and 36-13 to vibrate to make it easier for the driver to notice the left side of the vehicle M. In this case, the vibration units can vibrate synchronously or sequentially at a certain period. This allows the driver to easily perceive the possibility that the vehicle M may leave the dividing line RS1 extending to the left.

[0104] The vibration control unit 132 can also control the vibration pattern based on the likelihood of detachment as described above. In this case, the vibration control unit 132 increases the vibration or shortens the vibration period based on the likelihood of detachment.

[0105] <Sixth Vibration Control>

[0106] Figure 9 This diagram illustrates an example of the sixth vibration control method. Figure 9 In the example, besides vehicle M traveling at speed VM in lane L1, another vehicle m1 is shown approaching vehicle M from its right rear at speed Vm1. This other vehicle m1 is an example of an obstacle. In this case, it is assumed that the risk assessment unit 122 determines, based on the relative positions and relative speeds of vehicle M and the other vehicle m1, that there is a possibility of contact between vehicle M and vehicle M. In this case, the vibration control unit 132 causes the vibration unit 36-15, located to the right rear of the left and right feet LF and RF, to vibrate.

[0107] Alternatively, the greater the risk and the higher the probability of contact, the greater the vibration control unit 132 will control the vibration to increase. This allows for more accurate notification to the occupant of the likelihood of contact. The vibration control unit 132 can also change the vibration pattern based on the type of object being contacted. In this case, the vibration control unit 132 changes the vibration pattern when the object is another vehicle and when it is a pedestrian. The vibration control unit 132 can also induce different vibration patterns when it determines there is a risk of contact with an object and when it determines there is a possibility of leaving the lane (marking). This allows the driver (occupant) to have a more detailed understanding of the surrounding situation through different vibration patterns.

[0108] <Seventh Vibration Control>

[0109] Figure 10 This diagram illustrates an example of the seventh vibration control. In this seventh vibration control, vibration control is performed including these vibration units 36, provided that are also provided in the pedal operating components (accelerator pedal 84, brake pedal 86) and the footrest area. Figure 10 In the example, a footrest section FR is provided on the upper left side of the base plate section FL, where a vibration section 36-7 is installed. Vibration sections 36-8 to 36-15 are arranged in a grid pattern in the remaining areas of the base plate section FL. Furthermore, in... Figure 10 In the example, a vibrating part 36-16 is provided on the accelerator pedal 84, and a vibrating part 36-17 is provided on the brake pedal 86. Alternatively, multiple vibrating parts may be provided on the footrest FR, the accelerator pedal 84, and the brake pedal 86 respectively. Figure 10 In this example, the driver's left foot LF is placed in the center of the base plate FL, and the right foot RF is placed on the accelerator pedal 84. The foot forefoot detection unit 70 detects the position of the driver's left and right feet forefoot. In this embodiment, the AP sensor 84A can detect the foot placed on the accelerator pedal 84, and the BP sensor 86A can detect the foot placed on the brake pedal 86.

[0110] In the seventh vibration control, when the risk assessment unit 122 determines that there is a risk of contact with an object target, and / or the driving condition assessment unit 124 determines that there is a possibility that the vehicle M may leave the driving lane (dividing line), and one party's foot is placed on the pedal operating member, the vibration unit closest to each foot is vibrated. Figure 10 In the example, the vibration control unit 132 causes the vibration units 36-11 and 36-16 to vibrate.

[0111] By incorporating the vibration unit into the pedal operating component as shown in the seventh vibration control, tactile stimulation can be applied to the driver via the pedal even when the driver's foot moves from the floor to the pedal operating component, ensuring that the vibration notification is reliably detected.

[0112] The vibration control unit 132 can also vibrate in directions where there is a risk relative to the position of the foot or a possibility of deviating from the lane, even when a foot is placed on the floor FL. This allows the driver to be notified of direction-related information.

[0113] The vibration control unit 132 can also control the vibration of the vibration units 36-16 and 36-17 installed on the pedal operating member based on the determination result determined by the determination unit 120 when the driving mode of the vehicle M is manual driving or when switching from automatic driving mode to manual driving mode, based on the control state of the driving control unit 136. Therefore, the vibration units installed on the pedal can be vibrated when the driver is highly likely to operate the pedal.

[0114] It should be noted that in the seventh vibration control, when a foot is placed on the footrest FR, the vibration control unit 132 causes the vibration unit 36-7 provided on the footrest FR to vibrate based on the determination result determined by the determination unit 120.

[0115] In the implementation, the first to sixth vibration controls described above can each be combined with at least a portion of other vibration controls.

[0116] [Example of the installation of the vibrating part 36]

[0117] Next, an example of the arrangement of the vibration unit 36 ​​in the embodiment will be described using the accompanying drawings. Figure 11 This is a diagram showing a first example of the installation of the vibrating part 36. Figure 11 In the example, this is a schematic diagram showing the positional relationship between the floor panel FL and the seat ST around the driver's seat when viewed laterally. Figure 11 In this example, a vibration unit 36 ​​is provided on the seat back ST2, and multiple vibration units 36 are arranged at predetermined intervals within the floor blanket FM laid on the floor FL. Alternatively, the aforementioned pressure sensor (foot forward detection unit 70-2) can be provided within the floor blanket FM. By arranging the vibration units 36 within the floor blanket FM, vibrations with higher resolution can be transmitted to the driver D's foot. Since there is no need to install new vibration units 36 within the floor FL, cost reduction and a reduction in installation workload are achieved.

[0118] The vibration unit 36 ​​can also be installed within the base plate FL, taking into account the material of the base plate blanket FM and ensuring sufficient vibration propagation. Figure 12This diagram shows a second arrangement example of the vibrating section 36. In this second arrangement example, compared to the first arrangement example, multiple vibrating sections 36 are arranged inside the base plate portion FL. Therefore, even if the position of the base plate blanket FM is offset, the vibrating section at the appropriate position can vibrate.

[0119] In the first and second installation examples, the general floor blanket FM is made of a material with damping and sound-absorbing properties to reduce vibration (road noise) when the vehicle M is in motion. Therefore, when the vibration element 36 is provided in the floor portion FL, it is necessary to study the vibration transmission method to prevent vibration attenuation. Therefore, for example, in the case where the vibration element 36 is provided in the floor blanket FM as in the first installation example, in the layered structure of the floor blanket FM, the first layer (lower layer) facing the floor portion FL is set as a damping and sound-absorbing layer, the vibration element 36 is wrapped in the second layer (middle layer), and the third layer (upper layer) forms the skin layer. As a result, vibration can be easily transmitted from the vibration element 36 to the sole of the foot. In the second installation example, a material with damping and sound-absorbing properties is provided on the lower side (ground side) of the floor portion FL, and the vibration element is provided on it. The floor blanket FM is set as the skin layer or a structure without a damping and sound-absorbing layer is adopted, thereby making it easy to transmit vibration from the vibration element 36 to the sole of the foot.

[0120] [Notification Control]

[0121] In one embodiment, the notification control unit 134 can also generate an image and sound indicating the cause of the vibration when the vibration unit 36 ​​is vibrated by the vibration control unit 132, and output them from the HMI 30. This allows the occupant to more accurately understand the cause of the vibration affecting their feet.

[0122] The notification control unit 134 can synchronously control the timing of notifications based on information displayed and output to the HMI 30, and the vibration control unit 132 can control the vibration of the vibration unit 36, either under specified conditions or in stages. For example, if the risk level determined by the risk assessment unit 122 is less than a first threshold, vibration control based on the vibration control unit 132 is performed; if the risk level is above the first threshold, vibration control based on the vibration control unit 132 and alarm output (alarm display and sound output) based on the notification control unit 134 are performed. In this way, by linking image display and sound output with notifications based on the risk level, the driver can take actions corresponding to the risk.

[0123] The notification control unit 134 can also change the image's color, brightness, or sound based on the level of risk. In this case, the notification control unit 134 linearly or gradually increases the notification intensity (e.g., brightness, volume) as the risk level increases from low to high. Furthermore, vibration and display output (sound output) are always performed simultaneously regardless of the level of risk. By combining multiple sensory systems such as touch, vision, and hearing for notification in this way, a multimodal effect can be achieved, further enhancing intuitiveness.

[0124] The control combining the notification control of the notification control unit 134 and the vibration control of the vibration control unit 132 can also be applied to situations where the driving condition determination unit 124 determines that the vehicle M has left the lane. In this case, notification control, vibration control, or phased control are performed depending on the likelihood of the vehicle leaving the lane.

[0125] [Notification regarding shoes]

[0126] The notification control unit 134 may also notify the driver via the HMI30 through at least one of a display and a sound if it determines that the driver is wearing shoes that are unsuitable for driving based on the type of shoes identified by the shoe recognition unit 114.

[0127] Figure 13 This is an example of shoe inspection. Figure 13 (A) indicates that the foot-front detection unit 70-1 is an example of a camera. Figure 13 (B) indicates that the foot front detection section 70-2 is an example of a pressure sensor arranged in a grid pattern on the base plate section FL.

[0128] For example, the shoe recognition unit 114 is based on Figure 13 The type of shoe is identified by analyzing the camera image of the foot detection unit 70-1 shown in (A). Regarding the type of shoe, for example, by pattern matching processing between the feature information such as the shape, size, and color of the object obtained from the analysis result and the feature information corresponding to the predetermined type of shoe, it is possible to detect shoe types with higher consistency.

[0129] Shoe Identification Department 114 Figure 13As shown in (B), the type of shoe is determined based on the output range of the pressure-sensitive sensors detected by the grid-like arrangement of the foot detection unit 70-2. Since the vibration of the vibration unit 36 ​​may be difficult to transmit depending on the type of shoe, the notification control unit 134, when the driver's shoes are unsuitable for driving or have difficulty transmitting vibrations (e.g., high heels, platform shoes), displays images and outputs sound via the display unit 32 and speaker 34, thereby notifying the driver of a disclaimer, announcing that the vibration-based notification function is invalid, or urging them to change to more appropriate shoes. This instills a sense of safe driving awareness in the driver.

[0130] [Starting or stopping vibration control based on motion recognition unit 116]

[0131] Next, the start or stop control of vibration control based on the motion recognition unit 116 will be explained in detail. Figure 14 This diagram illustrates how the motion recognition unit 116 recognizes the movements of an occupant's feet. Figure 14 In one example, a pressure-sensitive sensor is installed on the floor blanket FM laid on the floor section FL as a foot detection unit 70-2. The motion recognition unit 116 recognizes that a occupant has performed a prescribed action when it detects that the occupant has tapped the floor blanket FM with their foot (forefoot, heel) a prescribed number of times (e.g., twice) within a prescribed time based on the detection results of the pressure-sensitive sensor.

[0132] In the case of a foot detection unit 70-1 that uses a camera, the motion recognition unit 116 analyzes the camera image to detect the movement of the occupant's feet. For example, if a movement that causes the feet to swing left and right more than twice is detected, the prescribed movement is identified.

[0133] When the motion recognition unit 116 detects that an occupant has performed a prescribed action, the vibration control unit 132 controls the start or stop of the vibration control for that occupant (vibration control for a vibration unit installed under the floor panel FL where the occupant is seated). For example, if the vibration control is started, the vibration control is stopped; if the vibration control is stopped, the vibration control is started again.

[0134] Therefore, when passengers become annoyed by vibration-based notifications due to factors such as the surrounding environment or their mood, they can arbitrarily stop vibration control or restart it through intuitive operation by the passenger. Switching between starting and stopping vibration control can also be controlled via a switch located in the HMI30, replacing the recognition of passenger foot movements.

[0135] [Variation Example]

[0136] In this embodiment, when vehicle M is traveling on a road with vibration (a poor road), vibration control unit 132 can also stop vibration control. In this case, vehicle M can determine whether the road it is traveling on is a mountain road or a road with poor elevation based on the vehicle M's position information and map information 54, or it can determine whether it is a poor road based on the detection results of the vibration sensor included in vehicle sensor 40. As a result, the vibration of vehicle M during travel (road noise) and the vibration based on vibration unit 36 ​​will not mix, and the discomfort caused to passengers can be suppressed.

[0137] In this embodiment, the vibration control unit 132 may also pre-register the foot placement position for each occupant or statistically obtain the foot position, and determine the position of the vibrating part 36 among the multiple vibration parts 36 corresponding to the occupant based on that position. This reduces the processing load for detecting foot position. The vibration control unit 132 may also pre-register the vibration pattern for each occupant and control the vibration pattern of the vibration part 36 according to the registered information for each occupant. This allows the occupant to be notified of vibrations according to their preferences.

[0138] [Processing Flow]

[0139] Next, a flowchart will be used to illustrate an example of the processing performed by the driving support device 100 in the embodiment. In the following example, the processing performed by the driving support device 100 will be mainly described with a focus on vibration control processing. The following processing can be repeatedly performed at predetermined cycles or times.

[0140] Figure 15 This is a flowchart illustrating an example of the processing performed by the driving support device 100 of the embodiment. Figure 15In the example, the surrounding situation recognition unit 112 recognizes the surrounding situation of the vehicle M (step S100). Next, the surrounding situation recognition unit 112 recognizes objects (e.g., obstacles) existing in the surrounding area (step S110). Next, the risk determination unit 122 determines whether there is a risk of the vehicle M contacting or approaching the object based on the relative position and relative speed between the vehicle M and the object (step S120). If a risk of contact or approach is determined to exist, the risk determination unit 122 detects the risk location and risk level relative to the vehicle M (step S130). In the processing of step S130, the risk direction of the risk may be detected instead of the risk location and risk level (or based on this). Next, the foot detection unit 70 detects the driver's foot (step S140). Next, the vibration control unit 132 determines the vibration pattern of the vibrating part 36 (e.g., the position of the vibrating part 36, the magnitude of the vibration, the vibration period, etc.) corresponding to the risk location, risk level, risk intensity, etc. (step S150). The vibrating part 36 of the object is vibrated based on the determined vibration pattern. Therefore, the processing of this flowchart ends. If it is determined in step S120 that there is no risk of vehicle M contacting or approaching the target object, this flowchart ends.

[0141] exist Figure 15 In the processing of steps S120 to S130 shown, the above processing can be substituted (or based on this) by determining whether the vehicle has left the lane based on the distance between the vehicle M and the dividing line, the speed VM, the direction of movement, etc. in the driving condition determination unit 124 (step S120), and if it is determined that the vehicle has left the lane, the direction of the departure is detected (step S130).

[0142] As explained above, the driving support device of this embodiment includes: a surrounding condition recognition unit 112 that recognizes the surrounding conditions of the vehicle M; a plurality of vibration units 36 that apply vibration-based stimulation to the occupants of the vehicle M; and a vibration control unit 132 that vibrates at least one of the plurality of vibration units 36 based on the relative position of an object target to the vehicle M and the direction of the object target relative to the vehicle M, as recognized by the surrounding condition recognition unit 112. At least one of the plurality of vibration units 36 is disposed at a position that can transmit vibration to the feet of the occupants of the vehicle M, thereby enabling more appropriate information to be transmitted to the occupants to support driving. Therefore, it can contribute to the development of a sustainable transportation system.

[0143] For example, according to the implementation method, by vibrating the base plate, it is possible to provide seated occupants with object target notifications that allow for easy and intuitive understanding of the object's location. For example, by directly transmitting tactile stimulation with vibration to the driver's feet, the location and direction of movement of obstacles can be intuitively identified and grasped, thereby enabling rapid pre-emptive actions to avoid collisions in potentially dangerous scenarios (e.g., reducing search time due to enhanced intuitiveness). Furthermore, since hazard awareness can be achieved even without viewing the display unit through notification based on tactile stimulation, the workload on the driver can be reduced.

[0144] According to the implementation method, for example, the vibration unit can be appropriately selected based on driving conditions such as the degree of risk and the direction of the risk (more specifically, the distance to the target object, the first time to travel (TTC), etc.), or the magnitude (intensity) and period of the vibration can be controlled, thus enabling the occupant to more intuitively grasp the risk. According to the implementation method, the position can be determined by using a camera to identify the location of the forefoot on the floor, or by calculating the coordinate position based on the output signal of a pressure-sensitive sensor arranged in a grid pattern, and then controlling the vibration unit based on the determined forefoot position, thus achieving more appropriate vibration transmission.

[0145] According to the embodiment, by urging drivers wearing shoes with a construction that makes it difficult to transmit vibrations and is unsuitable for driving to change their shoes, or by notifying them that the function is ineffective, the reliability of the driver's sense of safe driving and the functionality can be ensured. According to the embodiment, the vibration control is controlled based on tapping or postural movements on the base plate, thus improving convenience.

[0146] The implementation methods described above can be performed as follows.

[0147] A driving assistance device comprising:

[0148] Storage medium, which stores computer-readable instructions; and

[0149] The processor connected to the storage medium,

[0150] The processor performs the following processing by executing computer-readable instructions:

[0151] Identify the vehicle's surroundings; and

[0152] Based on the relative position of the identified objects in the surrounding environment to the vehicle, and the orientation of the objects relative to the vehicle, at least one of a plurality of vibrating parts that applies vibration-based stimuli to the occupants of the vehicle is vibrated.

[0153] At least one of the plurality of vibrating elements is disposed at a position capable of transmitting vibrations to the feet of the occupants of the vehicle.

[0154] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A driving support device, wherein, The driving support device includes: The surrounding environment recognition unit identifies the vehicle's surrounding environment; Multiple vibrating parts that apply vibration-based stimulation to the occupants of the vehicle; and The vibration control unit vibrates at least one of the plurality of vibration units based on the relative position of the object target to the vehicle and the orientation of the object target relative to the vehicle, as identified by the surrounding condition recognition unit. At least one of the plurality of vibrating elements is disposed at a position capable of transmitting vibrations to the feet of the occupants of the vehicle.

2. The driving support device according to claim 1, wherein, The vibration control unit determines the vibration mode of the plurality of vibration units based on the relative position and the direction of the target object.

3. The driving support device according to claim 1, wherein, The driving support device also includes a foot detection unit for detecting the position of the occupant's feet. The vibration control unit determines the vibration mode of the plurality of vibration units based on the position of the forefoot detected by the forefoot detection unit.

4. The driving support device according to claim 3, wherein, The driving support device also includes a notification unit that notifies the occupants of information via at least one of display and sound. The occupants include the driver of the vehicle. The foot detection unit detects the driver's shoes. The driving support device also includes a notification control unit, which notifies the driver if the driver's shoes are unsuitable for driving.

5. The driving support device according to claim 1, wherein, The driving support device also includes a motion recognition unit that recognizes prescribed movements performed by the occupant's feet. The vibration control unit controls the start or stop of vibration control for the vibration unit by recognizing a prescribed tapping action or a prescribed posture action performed by the occupant's feet by the action recognition unit.

6. The driving support device according to claim 4, wherein, The notification control unit notifies the occupants via a notification unit based on the level of risk associated with contact or proximity between the vehicle and the target object. The timing of notifications issued by the notification control unit and the timing of vibrations issued by the vibration control unit are synchronized under specified conditions or controlled in stages.

7. The driving support device according to claim 1, wherein, At least one of the plurality of vibrating elements is disposed on the pedal operating component of the vehicle. The vibration control unit performs vibration control on the vibration unit installed on the pedal operating member when the vehicle's driving mode is manual driving mode or when switching from automatic driving mode to manual driving mode.

8. The driving support device according to claim 1, wherein, At least one of the plurality of vibrating elements is disposed on the pedal operating component of the vehicle. The vibration control unit performs vibration control on the vibration unit installed on the pedal operating member when the driver's foot comes into contact with the pedal operating member.

9. A driving support method, wherein, The driving support method causes the computer to perform the following processing: Identify the vehicle's surroundings; and Based on the relative position of the identified objects in the surrounding environment to the vehicle, and the orientation of the objects relative to the vehicle, at least one of a plurality of vibrating parts that applies vibration-based stimuli to the occupants of the vehicle is vibrated. At least one of the plurality of vibrating elements is disposed at a position capable of transmitting vibrations to the feet of the occupants of the vehicle.

10. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Identify the vehicle's surroundings; and Based on the relative position of the identified objects in the surrounding environment to the vehicle, and the orientation of the objects relative to the vehicle, at least one of a plurality of vibrating parts that applies vibration-based stimuli to the occupants of the vehicle is vibrated. At least one of the plurality of vibrating elements is disposed at a position capable of transmitting vibrations to the feet of the occupants of the vehicle.

Images