Occupant monitoring device for vehicle

Abstract

The invention provides an occupant monitoring device for a vehicle, which can restrain unnecessary alarms from being frequently issued during the driving of the vehicle in the vehicle equipped with a driving support control device and an occupant monitoring device, and can help to improve the safety during the driving of the vehicle. The vehicle occupant monitoring device includes an imaging unit that acquires an image of a driver of a vehicle, and an occupant monitoring control unit that includes at least a behavior change detection unit and an alarm output unit. And a behavior change detection unit that detects a behavior change of the driver on the basis of the image acquired by the imaging unit, and an alarm output unit that outputs predetermined alarm information on the basis of the detection result of the behavior change detection unit. The behavior change detection unit sets a detection duration for detecting a behavior change of the driver in accordance with a traveling speed of the vehicle and a type of driving assistance control being executed.

Description

Technical Field

[0001] The present invention relates to a vehicle occupant monitoring device, which ensures the safety of the vehicle by monitoring the driver in a vehicle such as an automobile and issuing a predetermined alarm at a predetermined timing under a predetermined situation. Background Art

[0002] In recent years, in vehicles such as automobiles, occupant monitoring devices have been put into practical use. The occupant monitoring device is configured to monitor the faces and behaviors of occupants including the driver (also referred to as the driver) using a photographing device such as a camera, and to issue a predetermined alarm when it recognizes, for example, the driver's distraction or drowsiness. By installing such an occupant monitoring device in a vehicle, for example, the driver can concentrate on driving the vehicle, thereby enabling safe operation of the vehicle.

[0003] Generally speaking, it is known that the higher the vehicle speed, the shorter the time until an accident caused by distraction, drowsiness, etc. occurs. Therefore, in such an occupant monitoring device, it is desirable to set the time until, for example, the driver's behavior is recognized and determined as distraction based on the recognition result to be shorter as the vehicle speed increases. However, if the alarm is frequently issued at the same timing as when the vehicle is traveling at a high speed even when the vehicle speed is low, the driver may feel annoyed.

[0004] Therefore, in conventional occupant monitoring devices, various techniques for devising timing for issuing a warning or the like according to the situation have been proposed, for example, in Japanese Patent Application Laid-Open No. 2014-89589.

[0005] The occupant monitoring device disclosed in Japanese Patent Application Laid-Open No. 2014-89589 changes the timing of issuing an alarm when the driver's distraction or drowsiness is detected, based on the distance to the preceding vehicle, the vehicle speed of the host vehicle, etc. This can prevent the alarm from being issued frequently.

[0006] On the other hand, in recent years, in vehicles such as automobiles, various driving assistance control devices that perform control to assist the driver in performing a plurality of driving operations have been proposed and put into practical use in order to reduce the burden of driving operations on the driver and improve vehicle safety.

[0007] Generally speaking, the driver's driving load is reduced during the execution of the driving assistance control, so the driver has room to monitor the situation in a wider visual field.

[0008] Prior art literature Patent Literature Patent Document 1: Japanese Patent Application Publication No. 2014-89589 Summary of the invention

[0009] Technical issues However, there is a possibility that the driver's behavior for monitoring the surrounding conditions is mistakenly recognized as distraction in the previous occupant monitoring device. In this case, the occupant monitoring device frequently issues unnecessary alarms. Therefore, in this case, the driver may also feel annoyed. In addition, if an unnecessary alarm is issued, the driver's attention may be directed to the alarm source, resulting in a narrowing of the driver's field of vision. Therefore, in this case, the safety of the vehicle may also be hindered.

[0010] In the conventional occupant monitoring device disclosed in Japanese Patent Application Laid-Open No. 2014-89589 and the like, control during driving assistance control is not considered, and the above-mentioned problem may occur.

[0011] An object of the present invention is to provide a vehicle occupant monitoring device which, in a vehicle equipped with a driving assistance control device and an occupant monitoring device, can suppress frequent issuance of unnecessary alarms during vehicle travel and contribute to improving vehicle travel safety.

[0012] Technical Solution In order to achieve the above-mentioned purpose, an occupant monitoring device of a vehicle according to one embodiment of the present invention comprises a photographing unit and an occupant monitoring control unit, wherein the photographing unit acquires an image of a driver of the vehicle, and the occupant monitoring control unit has at least a behavior change detection unit and an alarm output unit, wherein the behavior change detection unit detects behavior changes of the driver based on the image acquired by the photographing unit, and the alarm output unit outputs predetermined alarm information based on the detection result of the behavior change detection unit, and the behavior change detection unit sets a detection duration for detecting behavior changes of the driver according to the driving speed of the vehicle and the type of driving assistance control being executed.

[0013] Technical Effects According to the present invention, a vehicle occupant monitoring device can be provided, which can suppress the frequent issuance of unnecessary alarms during vehicle travel in a vehicle equipped with a driving assistance control device and an occupant monitoring device, and contribute to improving the safety of the vehicle during travel. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a block diagram showing a schematic configuration of a vehicle equipped with the occupant monitoring device according to the first embodiment of the present invention.

[0015] Figure 2 It is conceptually shown Figure 1 A simplified block diagram of the structure of an occupant monitoring device.

[0016] Figure 3This is a graph conceptually showing the setting of the detection duration time in the occupant monitoring device according to the first embodiment of the present invention.

[0017] Figure 4 This is a flowchart showing the operation of the occupant monitoring device according to the first embodiment of the present invention.

[0018] Figure 5 This is a flowchart showing a part of the operation of the occupant monitoring device according to the second embodiment of the present invention.

[0019] Explanation of symbols 1…Vehicle 10…Driving assistance control device (camera unit) 11…Stereo camera 11a…Main Camera 11b…Secondary camera 12…Image Processing Unit (IPU) 13…Image recognition unit 14…Control unit 21…Cockpit control unit (CP_ECU) 22…Engine control unit (E / G_ECU) 23…Transmission control unit (T / M_ECU) 24…Brake control unit (BK_ECU) 25…Power steering control unit (PS_ECU) 31…Human Machine Interface (HMI) 31a…Steering wheel touch sensor (ST touch sensor) 31b…Direction indicator rod 31d…Notification device 32…Throttle actuator 33…Hydraulic control circuit 34…Brake actuator 35…Electric power steering motor 36…Positioner unit 36a…GNSS sensor 36b…Road Map DB 37…Vehicle-mounted radar device 38…Rear sensor 39…Vehicle speed detection device 41…Photography Department 42…DMS_ECU (Driving Monitoring Control Unit) 42a…Behavior Change Detection Unit 42b…Alarm output T…Benchmark test duration T1… First detection duration T2…Second detection duration T3…The third detection duration α…First time β…Second time γ…Third Time DETAILED DESCRIPTION

[0020] Hereinafter, the present invention will be described with reference to the embodiments shown in the drawings.

[0021] Before describing in detail the occupant monitoring device according to the first embodiment of the present invention, Figure 1 A schematic configuration of a vehicle equipped with the occupant monitoring device will be described. Figure 1 It is a block diagram showing a schematic configuration of a vehicle equipped with the occupant monitoring device according to the first embodiment of the present invention.

[0022] It should be noted that Figure 1 Only the main components directly related to the present invention are shown in the figure, and other components are omitted from the figure as having substantially the same components as the conventional vehicles of the same type. In addition, only the main components directly related to the present invention will be described below.

[0023] like Figure 1 As shown, the occupant monitoring device of this embodiment is mounted (in Figure 1 The vehicle 1 denoted by reference numeral 31 c ) is equipped with at least a driving assistance control device (camera unit 10 ; described in detail later), a vehicle speed detection device 39 , an occupant monitoring device (DMS 31 c ), and a notification device 31 d .

[0024] The driving assistance control device is a device that can selectively perform multiple types of driving assistance controls to assist the driving operation of the vehicle or perform several driving assistance controls at the same time, and is a constituent unit. Therefore, among the constituent devices that perform the driving operation of the vehicle, the driving assistance control device mainly assists the driving operation of the accelerator, brake (brake device), steering wheel (steering device), and performs acceleration and deceleration control and steering control of the vehicle.

[0025] The driving assistance control device can be set to multiple driving modes, such as: (1) Manual driving mode, in which a notification or warning for driving assistance (hereinafter referred to as assistance notification) is provided when the driver is actively performing various driving operations (accelerator, brake, steering wheel, etc.) and is in the process of accelerating, decelerating, steering, etc., in a predetermined situation and at a predetermined timing; (2) Assisted driving mode, which is based on active driving by the driver and intervenes in driving assistance controls such as acceleration and deceleration control and steering assistance control under predetermined circumstances and at predetermined times; and (3) An automatic driving mode in which the vehicle is automatically driven without requiring any driving operation by the driver; etc.

[0026] These multiple driving modes can be arbitrarily and selectively set by the driver. It should be noted that the automatic driving mode (3) mentioned above may not necessarily be provided in the driving assistance control device.

[0027] The specific driving assistance controls executed in the multiple driving modes include, for example, the following controls: (A) The first driving assistance control only performs acceleration and deceleration control; (B) a second driving assistance control that performs acceleration / deceleration control and steering control at a steering angle smaller than a predetermined steering angle; and (C) A third driving assistance control is to perform acceleration / deceleration control and steering control at a predetermined steering angle or more.

[0028] The driver can also set some of these multiple driving assistance controls selectively or simultaneously as desired.

[0029] Here, in this embodiment, (A1) The first driver assistance control is equivalent to the following vehicle distance control (ACC; Adaptive Cruise Control), (A2) Secondary driving assistance control is equivalent to lane centering control (ALKC; Active Lane Keep Centering Control), (A3) The third driving assistance control is equivalent to the automatic lane changing control (ALC; Auto Lane Changing Control).

[0030] It should be noted that the steering angle less than the predetermined one in (B) above refers to a steering angle assumed to be made, for example, to cause the vehicle to leave its own lane. In addition, the steering angle greater than the predetermined one in (C) above refers to a steering angle assumed to be made, for example, to cause the vehicle to change lanes from its own lane to an adjacent lane.

[0031] By executing the first to third driving assistance controls in predetermined situations and at predetermined timings, the host vehicle can be automatically and stably driven along the determined driving lane while maintaining the inter-vehicle distance between the host vehicle and the preceding vehicle.

[0032] It should be noted that, in general, the following settings are often used: (a) Only the first driving assistance control setting (ACC) is performed; (b) performing simultaneous setting of the first driving assistance control and the second driving assistance control (ACC+ALKC); (c) performing simultaneous setting of the first driving assistance control and the third driving assistance control (ACC+ALC); and (d) The first driving assistance control, the second driving assistance control, and the third driving assistance control are simultaneously set (ACC+ALKC+ALC).

[0033] In the setting of (a) above, control is performed such that the host vehicle continues to travel following the preceding vehicle while maintaining the inter-vehicle distance between the host vehicle and the preceding vehicle.

[0034] In the setting of (b) above, control is performed such that the host vehicle continues to travel along the determined travel lane while following the preceding vehicle while maintaining the inter-vehicle distance between the host vehicle and the preceding vehicle.

[0035] In the setting of (c) above, the following automatic lane change control is performed: while the vehicle is following the vehicle in front while maintaining the inter-vehicle distance between the vehicle and the vehicle in front, the vehicle changes lanes from the lane in which it is traveling to the adjacent lane when a lane change instruction signal is received. In this case, the lane change instruction signal is, for example, obtained by the driver operating the direction indicator lever 31b (see Figure 1 ) and the output signal.

[0036] In the setting of (d) above, the following automatic lane change control is performed: while maintaining the inter-vehicle distance between the vehicle and the preceding vehicle, the vehicle follows the preceding vehicle, and if a lane change instruction signal is received while the vehicle continues to travel along the determined driving lane, the vehicle changes lanes from the lane in which it is traveling to the adjacent lane at the timing of receiving the signal.

[0037] Specifically, the driving assistance control device is, for example, a camera unit 10 which is a vehicle-mounted camera device including a stereo camera 11 fixed to a front upper center portion in a cabin (not shown) of the vehicle 1 .

[0038] like Figure 1 As shown, the camera unit 10 is composed of a stereo camera 11 and an image processing unit (Image Processing Unit; in Figure 1 The image recognition unit 13 and the control unit 14 are represented as “IPU” in the figure.

[0039] The stereo camera 11 is formed to include two cameras, a main camera 11a and a sub-camera 11b. The main camera 11a and the sub-camera 11b are arranged, for example, in a vehicle cabin at a position symmetrical to the left and right with respect to the center in the vehicle width direction, facing the front of the vehicle. The main camera 11a and the sub-camera 11b are respectively composed of, for example, an imaging optical system, an imaging element such as a CMOS image sensor, and a processing circuit for processing imaging signals, etc. (the detailed configuration is omitted in the figure).

[0040] With such a configuration, the stereo camera 11 uses the main camera 11a and the auxiliary camera 11b to acquire two image data from two different viewpoints of the surrounding environment of a predetermined range in front of the vehicle in synchronization with each other. Then, stereo image data is generated based on the two image data thus acquired. The stereo image data is surrounding environment information representing the surrounding environment of the vehicle during driving. The surrounding environment information (image data) generated in the stereo camera 11 is output to the image processing unit 12.

[0041] The image processing unit 12 is a component or circuit unit that performs predetermined image processing on the surrounding environment information (image data representing the surrounding environment of the vehicle during travel) acquired by the stereo camera 11. The image processing unit 12 performs processing to detect the edges of various objects (objects or dividing lines, etc.) represented on the image, for example.

[0042] The image processing unit 12 acquires distance information based on the positional offset of the edge corresponding to the left and right images based on the stereoscopic image data, and generates image information (distance image information) including the distance information. The distance image information generated by the image processing unit 12 is output to the image recognition unit 13.

[0043] The image recognition unit 13 calculates the road curvature [1 / m] of the left and right dividing lines and the width between the left and right dividing lines (lane width) and the like of the road on which the vehicle is traveling (the own vehicle's traveling road) based on the distance image information and the like input from the image processing unit 12. The method of obtaining the road curvature and the lane width uses various known methods.

[0044] In addition, the image recognition unit 13 performs predetermined pattern matching based on the distance image information acquired by the stereo camera 11, and identifies three-dimensional objects extending along the road (such as guardrails and / or curbs and other surrounding vehicles, etc.), or parking frame lines marked by dividing lines on the road surface in buildings such as parking lots, or three-dimensional structures such as barriers that set parking demarcation areas, or gaps between other adjacent vehicles, etc.

[0045] Here, in the recognition of three-dimensional objects (object recognition) in the image recognition unit 13, for example, the type of the three-dimensional object, the height of the three-dimensional object, the width of the three-dimensional object, the distance from the vehicle to the three-dimensional object, the moving speed of the three-dimensional object, the relative speed between the three-dimensional object and the vehicle, the relative distance between the three-dimensional objects (for example, the lateral distance between the curb at the edge of the road and the dividing line located near it, etc.), etc. are recognized.

[0046] Various information recognized by the image recognition unit 13 is output as surrounding environment information to the control unit 14. In this case, the camera unit 10 including the image recognition unit 13 has a function as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle.

[0047] The control unit 14 included in the camera unit 10 is a structural unit or a circuit unit that controls the entire driving assistance control device together with the camera unit 10 to comprehensively perform driving assistance control of the vehicle 1 on which the driving assistance control device is mounted.

[0048] The control unit 14 is connected to various control units such as a cockpit control unit (CP_ECU) 21, an engine control unit (E / G_ECU) 22, a transmission control unit (T / M_ECU) 23, a brake control unit (BK_ECU) 24, and a power steering control unit (PS_ECU) 25 via an in-vehicle communication circuit such as a CAN (Controller Area Network).

[0049] The CP_ECU 21 is connected to a human-machine interface (in Figure 1 The HMI 31 is described as “HMI” in the figure) 31. The HMI 31 is configured to include, for example, various operation members (31b), various sensing devices (31a, 31c), and a notification device 31d.

[0050] Here, as various operating components, for example, in addition to an operating switch for indicating the start or stop of the occupant monitoring system (DMS31c), a driving mode switching switch for selecting or switching the driving mode, multiple operating switches for indicating the start or stop of the driving assistance control, etc. (not shown), it also includes a direction indicator lever 31b for issuing a lane change indication signal, etc.

[0051] Here, the plurality of driving modes include, for example, a manual driving mode accompanied by a power assistance notification, a power assistance driving mode that performs driving assistance control by intervening in a part of driving operation at a predetermined time under a predetermined situation, and an automatic driving mode that performs automatic driving control by intervening in all driving operations during vehicle travel, etc. It should be noted that the automatic driving mode may not necessarily be included.

[0052] As various sensing devices, there is a steering wheel touch sensor (in Figure 1 The steering wheel touch sensor 31a is a sensing device that detects the grip state of the steering wheel (steering wheel) operated by the driver.

[0053] The DMS 31c is a component unit, a circuit unit, or a sensing device that monitors the driver of the vehicle and detects changes in the driver's behavior, and outputs predetermined alarm information at a predetermined timing according to the detection result. The DMS 31c is configured as an in-vehicle monitoring system composed of, for example, an in-vehicle camera, etc. The DMS 31c can be started or stopped by arbitrarily operating a predetermined operation switch.

[0054] Here, the driver's behavior change refers to, for example, a change in the orientation of the face, a change in the direction of the line of sight, a change in the degree of eye opening, a change in facial expression, etc. The DMS 31c continuously monitors the driver's behavior to detect posture changes (posture collapse), a decrease in alertness (change in the degree of eye opening, etc.). In addition, the DMS 31c as an occupant monitoring device can also detect the riding conditions of occupants including the driver.

[0055] Here, use Figure 2 The configuration of the DMS 31c as the occupant monitoring device according to the present embodiment will be described. Figure 2 It is a schematic block diagram conceptually showing the configuration of the DMS 31c which is the occupant monitoring device according to the present embodiment.

[0056] The DMS 31 c is configured to include an imaging unit 41 and a DMS_ECU 42 . The imaging unit 41 acquires an image of passengers including the driver of the vehicle, performs predetermined image processing on the acquired image, and outputs the image to the DMS_ECU 42 .

[0057] The DMS_ECU 42 is an occupant monitoring control unit that controls the DMS 31 c. The DMS_ECU 42 is a component unit or a circuit unit that is constituted by, for example, an electronic control unit (ECU) or the like. The DMS_ECU 42 includes a behavior change detection unit 42 a and an alarm output unit 42 b.

[0058] The behavior change detection unit 42a is a component or circuit unit that detects a change in the driver's behavior, etc., based on, for example, an image acquired by the camera unit 41, outputs from other sensing devices (such as the ST touch sensor 31a, etc.) or various operating members (such as the direction indicator lever 31b, etc.). In addition, the behavior change detection unit 42a detects the riding conditions of passengers including the driver. In addition, the behavior change detection unit 42a also performs facial recognition of the driver to perform personal recognition, etc.

[0059] The alarm output unit 42b is a component unit or a circuit unit that outputs a predetermined alarm information signal to the notification device 31d based on the detection result of the behavior change detection unit 42a.

[0060] The notification device 31d is a device that notifies the driver of a predetermined alarm corresponding to the alarm information output from the occupant monitoring device (DMS 31c) in a predetermined manner (visual display or auditory display). The notification device 31d is composed of, for example, a touch panel display device (visual display device), a sound device including a speaker (auditory display device), a combination meter in which various measuring instruments are integrated, and the like.

[0061] The CP_ECU 21 is a component unit or a circuit unit that appropriately notifies the driver of various information in a predetermined manner (visual or auditory display) using a notification device 31 d included in the HMI 31 upon receipt of a control signal from the control unit 14 .

[0062] Here, as various information notified by the notification device 31d, there are various information related to, for example, various alarm information, the implementation status of driving assistance control, the surrounding environment of the vehicle, etc. In addition, the notification device 31d receives the alarm information signal output from the alarm output unit 42b, and notifies the driver of a predetermined attention reminder warning, etc.

[0063] Furthermore, the CP_ECU 21 outputs various input information such as instruction signals input by the driver (eg, instructions for turning on or off various driving assistance controls, instructions for switching or selecting a driving mode, etc.) to the control unit 14 using various operating members included in the HMI 31 .

[0064] A throttle actuator 32 that electronically controls a throttle valve and the like are connected to the output side of the E / G_ECU 22. Various sensors such as an accelerator sensor are connected to the input side of the E / G_ECU 22.

[0065] The E / G_ECU 22 is a driving device that generates the driving force of the vehicle by driving and controlling the throttle actuator 32 based on the control signal from the control unit 14 or the detection signal from various sensors, and is a component unit or a circuit unit. Thus, the E / G_ECU 22 adjusts the intake air amount of the engine to generate the desired engine output. In addition, the E / G_ECU 22 outputs signals such as the throttle opening detected by various sensors (such as the throttle sensor) to the control unit 14.

[0066] The hydraulic control circuit 33 is connected to the output side of the T / M_ECU 23. In addition, various sensors such as a shift position sensor (not shown) are connected to the input side of the T / M_ECU 23.

[0067] The T / M_ECU 23 is a component or circuit unit that performs hydraulic control on the hydraulic control circuit 33 based on the engine torque signal estimated in the E / G_ECU 22, the detection signals from various sensors, etc. Thus, the T / M_ECU 23 operates the friction engagement member, pulley, etc. provided in the automatic transmission to change the engine output at a desired speed ratio. In addition, the T / M_ECU 23 outputs signals such as the gear position detected by various sensors to the control unit 14.

[0068] The output side of the BK_ECU 24 is connected to a brake actuator 34 for adjusting the brake fluid pressure outputted from the brake cylinders provided at the respective wheels. In addition, the input side of the BK_ECU 24 is connected to various sensors (not shown) such as a brake pedal sensor, a yaw rate sensor, a front and rear acceleration sensor, and a vehicle speed sensor.

[0069] BK_ECU24 is a brake device that performs a braking control of the vehicle by driving and controlling the brake actuator 34 based on a control signal from the control unit 14 or a detection signal from various sensors, and is a component unit or a circuit unit. Thus, BK_ECU24 causes a braking force for a mandatory braking control and / or yaw rate control of the vehicle to be appropriately generated on each wheel. In addition, BK_ECU24 outputs signals such as a brake operation state detected by various sensors, or a yaw rate, a longitudinal acceleration, and a vehicle speed to the control unit 14.

[0070] The output side of the PS_ECU 25 is connected to an electric power steering motor 35 that applies a steering torque to the steering mechanism by the rotational force of the motor. In addition, the input side of the PS_ECU 25 is connected to various sensors (not shown) such as a steering torque sensor and a steering angle sensor.

[0071] The PS_ECU 25 is a steering device that controls the driving of the electric power steering motor 35 to control the steering of the vehicle based on the control signal from the control unit 14 or the detection signal from various sensors, and is a component unit or a circuit unit. Thus, the PS_ECU 25 generates a steering torque for the steering mechanism. In addition, the PS_ECU 25 outputs signals such as the steering torque and the steering angle detected by various sensors to the control unit 14.

[0072] In addition, various sensors such as a locator unit 36, a vehicle-mounted radar device 37, a rear sensor 38, and a vehicle speed detection device 39 are connected to the control unit 14. The locator unit 36 ​​includes a GNSS sensor 36a and a high-precision road map database (road map DB) 36b.

[0073] The GNSS sensor 36 a locates the position (latitude, longitude, altitude, etc.) of the host vehicle by receiving positioning signals transmitted from a plurality of positioning satellites.

[0074] The road map DB 36 b is a large-capacity storage medium such as a HDD (Hard Disk Drive) device or an SSD (Solid State Drive) device, and stores high-precision three-dimensional road map information (dynamic map).

[0075] The road map DB 36b has lane width data, lane center position coordinate data, lane travel azimuth data, speed limit, etc. as lane data required for automatic driving. The lane data is stored at intervals of several meters for each lane on the road map. In addition, in addition to traffic control and road construction, the road map DB 36b also includes dynamic information such as accidents and traffic jams that change all the time.

[0076] In addition, the locator unit 36 ​​can communicate with an external system (not shown) to obtain real-time information about the surrounding environment of the vehicle position located by the GNSS sensor 36a (for example, various information related to parking lots, etc. in addition to traffic jam information and weather information). In this case, the weather information also includes, for example, fog generation information, rainfall information, snowfall information, snow accumulation information, and temperature and humidity information in the area including the vehicle position.

[0077] In addition, the road map DB 36b has information on various facilities, parking lots, etc. The road map DB 36b outputs road map information of a set range based on the vehicle position located by the GNSS sensor 36a as surrounding environment information to the control unit 14, for example, based on a request signal from the control unit 14. Thus, in the present embodiment, the road map DB 36b has a function as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle together with the GNSS sensor 36a.

[0078] The vehicle-mounted radar device 37 is composed of multiple sensors, such as multiple millimeter wave radars. Here, multiple millimeter wave radars receive reflected waves from objects by outputting radio waves and analyze them, so that in addition to mainly detecting three-dimensional objects such as pedestrians and parallel vehicles, they also detect structures set on the edge of the road (for example, the edge of the shoulder side) (for example, curbs, guardrails, walls of buildings, plantings and other three-dimensional objects, etc.). In addition, multiple millimeter wave radars can also detect three-dimensional obstacles on the road. In this case, multiple millimeter wave radars detect the lateral width of the three-dimensional object, the position of the representative point of the three-dimensional object (relative position and relative distance with the vehicle) and the relative speed as specific information related to the three-dimensional object.

[0079] The multiple sensors (multiple millimeter wave radars, etc.) included in the vehicle-mounted radar device 37 are arranged, for example, on the left and right sides of the front bumper (referred to as front left and right side sensors), on the left and right sides of the rear bumper (referred to as rear left and right side sensors), etc. In addition, the front left and right side sensors detect three-dimensional objects existing in the areas obliquely in front and to the sides of the vehicle that are difficult to identify in the image of the stereo camera 11 as surrounding environment information. In addition, the rear left and right side sensors detect three-dimensional objects existing in the areas obliquely to the left and right sides and rear of the vehicle that are difficult to identify in the front left and right side sensors as surrounding environment information.

[0080] Thus, in this embodiment, the vehicle-mounted radar device 37 functions as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle. In addition, information acquired by each sensor of the vehicle-mounted radar device 37 is transmitted to the image recognition unit 13 through the control unit 14.

[0081] The rear sensor 38 is composed of a sonar device that uses ultrasonic waves to measure the distance and / or shape of an object. For example, at least one (or more) rear sensor 38 is provided on the rear bumper. The rear sensor 38 detects a three-dimensional object that is difficult to identify with the rear left and right side sensors in the rear area of ​​the vehicle as surrounding environment information. In this way, in the present embodiment, the rear sensor 38 has a function as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle.

[0082] The coordinates of each object outside the vehicle contained in the surrounding environment information identified by the image recognition unit 13, the locator unit 36, the vehicle-mounted radar device 37, the rear sensor 38, etc. are transformed into coordinates of a three-dimensional coordinate system with the center of the vehicle as the origin in the control unit 14.

[0083] The vehicle speed detection device 39 is a component unit or a circuit unit that detects the running speed of the vehicle based on a plurality of wheel speed sensors and the outputs of the plurality of wheel speed sensors. The vehicle speed detection device 39 detects the vehicle speed under the control of the control unit 14.

[0084] In this case, the plurality of wheel speed sensors included in the vehicle speed detection device 39 detect pulse signals (wheel speed pulses) generated in proportion to the number of revolutions of each wheel (generally four wheels) in the vehicle. The vehicle speed detection device 39 detects the vehicle speed based on the wheel rotation speed data of each wheel acquired by the plurality of wheel speed sensors, for example, by finding the average data of each wheel speed data. The vehicle speed data detected by the vehicle speed detection device 39 is output to the control unit 14.

[0085] The control unit 14 executes driving assistance control of the vehicle based on various information acquired by the camera unit 10 and various sensors (the locator unit 36 ​​, the vehicle-mounted radar device 37 , the rear sensor 38 , and the vehicle speed detection device 39 ).

[0086] The driving assistance control in this case is, for example, in addition to executing engine output control and / or torque distribution control of each drive wheel based on E / G_ECU22, it also executes forward or reverse travel direction control using transmission control based on T / M_ECU23, or brake control (Brake control) of each wheel based on BK_ECU24, etc., to achieve driving assistance control of the vehicle as appropriately required.

[0087] In addition, among the various types of sensors used to obtain surrounding environment information, in addition to the above-mentioned sensor types, there may also be included, for example, a laser radar (LiDAR; Light Detection And Ranging) device that uses laser to detect the distance and / or shape of an object, a near-infrared sensor, an external temperature sensor, etc.

[0088] In addition, the above-mentioned stereo camera 11 mainly uses the predetermined visual field range in the front as the observation object. On this basis, multiple camera devices of the same mode that use the predetermined visual field range in the side and rear as the object can also be further provided. In this way, the range of the whole circumference of the vehicle can be used as the observation object.

[0089] It should be noted that all or part of the image recognition unit 13 , the control unit 14 , the CP_ECU 21 , the E / G_ECU 22 , the T / M_ECU 23 , the BK_ECU 24 , the PS_ECU 25 , the DMS 31 c , and the like are configured by a processor including hardware.

[0090] Here, the processor, for example, is composed of a central processing unit (CPU), a RAM (Random Access Memory), a ROM (Read Only Memory), or a non-volatile memory, a non-volatile storage device, etc., and is also composed of well-known structures such as a non-transitory computer readable medium and its peripheral devices.

[0091] The software program executed by the CPU and / or fixed data such as a data table are stored in advance in the ROM, nonvolatile memory, nonvolatile storage device, etc. The CPU reads the software program stored in the ROM, etc. and expands it to the RAM for execution, and the software program appropriately refers to various data, etc., thereby realizing the functions of the above-mentioned components, components (13, 14, 21 to 25, 31c), etc.

[0092] In addition, the processor may be formed by a semiconductor chip such as an FPGA (Field Programmable Gate Array) etc. In addition, each of the above-mentioned components, components (13, 14, 21 to 25, 31c) etc. may be formed by an electronic circuit.

[0093] In addition, the software program as a computer program product may be in the form of being recorded in whole or in part on a non-transitory computer readable medium such as a portable plate-type medium such as a floppy disk, CD-ROM, DVD-ROM, a card-type memory, an HDD (Hard Disk Drive) device, an SSD (Solid State Drive) device, or the like.

[0094] Next, the operation of the DMS 31c as the passenger monitoring device of the present embodiment configured as described above will be described.

[0095] First, a vehicle 1 equipped with an occupant monitoring device (DMS 31c) of the present embodiment has a plurality of driving modes and is equipped with a driving assistance control device capable of selectively executing a plurality of driving assistance controls or executing several driving assistance controls simultaneously. With such a configuration, the vehicle 1 travels while executing one or more driving assistance controls selected by the driver according to a plurality of driving modes selected by the driver.

[0096] In addition, when the occupant monitoring device (DMS 31c) is activated during the driving of the vehicle 1, the DMS 31c sets a detection duration based on the vehicle speed data of the vehicle 1. Here, the detection duration refers to the time used by the DMS 31c to detect the driver's behavior change. When a predetermined behavior change is detected within the detection duration, an alarm is output. It should be noted that in this case, when one or more driving assist controls are being executed, the DMS 31c also sets a predetermined detection duration corresponding to the type of driving control being executed.

[0097] Figure 3 : is a diagram conceptually showing the setting of the detection duration in the occupant monitoring device (DMS 31c) of this embodiment. Figure 3 As shown, the detection duration (T) of the occupant monitoring device (DMS 31c) is basically set to be shorter as the vehicle speed (V) is faster. In the manual driving mode, when the DMS 31c drives the vehicle 1 without executing any of the plurality of driving assistance controls, the detection duration (T) is set only according to the vehicle speed (V). Figure 3 The graph shown as "BASE" in FIG. 1 is the detection duration at this time. In the following description, the detection duration at this time is referred to as the reference detection duration T.

[0098] Normally, a driver driving the vehicle 1 without executing driving assistance control often continues to travel the vehicle 1 while monitoring surrounding conditions. The detection time of the DMS 31c at this time is set as a reference (reference detection duration T).

[0099] In the assist driving mode, when the vehicle 1 is driven while any one of the first to third driving assistance controls is being executed, the DMS 31 c sets the detection duration (T) in addition to the vehicle speed (V) according to the type of driving assistance control being executed.

[0100] exist Figure 3The graph shown as "BASE+α" in FIG. 1 is a first detection continuation time T1 when the vehicle 1 is driven in a state where only the first driving assist control is executed. Here, the symbol α refers to a first time added to the reference detection continuation time T.

[0101] In general, the driver of the vehicle 1 that performs only the first driving assistance control mainly needs to monitor the surrounding conditions ahead (such as the vehicle ahead). In view of this, during the execution of the first driving assistance control, it can be inferred that the change in behavior made by the driver to monitor the surrounding conditions ahead is a behavior made by the driver according to his own will. A first detection duration T1 is set, to which a first time α as the time required for this is added.

[0102] exist Figure 3 The graph shown as "BASE+β" in FIG. 1 is a second detection continuation time T2 when the vehicle 1 is driven in a state where the second driving assist control is executed. Here, the symbol β refers to a second time added to the base detection continuation time T.

[0103] Under normal circumstances, the driver of the vehicle 1 in which the second driving assistance control is executed mainly needs to monitor the surrounding conditions in front and on the left and right (such as the vehicle ahead and the left and right dividing lines). Considering this, in the execution of the second driving assistance control, it can be inferred that the change in behavior made by the driver to monitor the surrounding conditions in front and on the left and right is a behavior made by the driver according to his own will. The second detection duration T2 is set by adding the second time β as the time required for this.

[0104] exist Figure 3 The graph shown as "BASE+γ" in FIG. 1 is a third detection continuation time T3 when the vehicle 1 is driven in a state where the third driving assist control is executed. Here, the symbol γ refers to a third time added to the base detection continuation time T.

[0105] Under normal circumstances, the driver of the vehicle 1 in which the third driving assistance control is executed needs to monitor the surrounding conditions in the rear (the vehicle in front, the left and right dividing lines, the following vehicles in the own lane and the adjacent lane, etc.) in addition to the front and left and right. Considering this, in the execution of the third driving assistance control, it can be inferred that the behavior change made by the driver to monitor the surrounding conditions in the front, left, right, and rear is the behavior made by the driver according to his own will. The third detection duration T3 is set to which the third time γ as the time required for this is added.

[0106] It should be noted that, here, the first time α, the second time β, and the third time γ are set to have a relationship of "α≤β≤γ". That is, the first time α is set to be less than the second time β. The second time β is set to be greater than the first time α and less than the third time γ. The third time γ is set to be greater than the second time β.

[0107] Thus, the reference detection duration T is set to a time shorter than the first detection duration T1. The first detection duration T1 is set to a time shorter than the second detection duration T2. ​​The second detection duration T2 is set to a time shorter than the third detection duration T3.

[0108] Figure 4 1 is a flowchart showing the operation of the occupant monitoring device (DMS 31c) according to the first embodiment of the present invention. Figure 4 The operation of the occupant monitoring device (DMS 31 c ) according to this embodiment will be described with reference to the flowchart of FIG.

[0109] exist Figure 4 In step S1, the control unit 14 confirms whether the DMS 31c is activated. Here, if it is confirmed that the DMS 31c is activated, the process proceeds to the next step S2. It should be noted that if the DMS 31c is not activated, this process is repeated. That is, the start of the flowchart is triggered by the activation of the DMS 31c.

[0110] In step S2, the control unit 14 confirms whether the setting of the driving mode in the driving assistance control device is the manual driving mode. Here, if it is confirmed that the manual driving mode is set, the process proceeds to step S3. In this case, the setting information of the driving mode is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. In addition, when a mode other than the manual driving mode is set, the process proceeds to step S4.

[0111] In step S3 , the DMS_ECU 42 sets a reference detection duration T.

[0112] In step S4, the control unit 14 confirms whether the setting of the driving mode in the driving assistance control device is the power-assisted driving mode. Here, if it is confirmed that the power-assisted driving mode is set, the process proceeds to step S5. In this case, the setting information of the driving mode is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. In addition, when a mode other than the power-assisted driving mode is set (that is, when the automatic driving mode is set), a series of processes are terminated (return).

[0113] In the vehicle 1 equipped with the occupant monitoring device (DMS 31c) of the present embodiment, when the automatic driving mode is set and the vehicle is driven, the occupant monitoring by the DMS 31c is not necessary. Therefore, in this flowchart, when the automatic driving mode is set, after a series of processing is completed, for example, it moves to other control processing corresponding to the automatic driving mode (not shown).

[0114] In step S5, the control unit 14 confirms the type of driving assistance control that is set. Here, when only the first driving assistance control is set, the process proceeds to step S6. In this case, the setting information of the driving assistance control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. In addition, when multiple driving assistance controls are set (including the first driving assistance control), the process proceeds to step S7.

[0115] In step S6 , the DMS_ECU 42 sets a first detection duration T1 .

[0116] In step S7, the control unit 14 confirms the type of driving assistance control that is set. Here, when the setting including the second driving assistance control is performed, the process proceeds to step S8. In this case, the setting information of the driving assistance control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. In addition, when a driving assistance control other than the second driving assistance control is set (it may be a plurality of driving assistance controls), the process proceeds to step S7.

[0117] In step S8 , the DMS_ECU 42 sets a second detection duration T2 .

[0118] In step S9, the control unit 14 confirms the type of driving assistance control that is set. Here, when the setting including the third driving assistance control is performed, the process proceeds to step S10. In this case, the setting information of the driving assistance control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. In addition, when a driving assistance control other than the third driving assistance control is set (it may be a plurality of driving assistance controls), a series of processes are terminated (return).

[0119] The driving assistance control device in this embodiment exemplifies the first to third driving assistance controls as a plurality of driving assistance controls. However, the driving assistance controls that can be executed in the driving assistance control device are not limited to the above three types, and other driving assistance controls may also exist. In this embodiment, for other driving assistance controls, after a series of processing is completed, the control processing corresponding to the other driving assistance controls is transferred to (not shown).

[0120] In step S10 , the DMS_ECU 42 sets a third detection duration T3 .

[0121] In step S11, the control unit 14 controls the vehicle speed detection device 39 to detect the current vehicle speed data. The vehicle speed data input from the vehicle speed detection device 39 to the control unit 14 is further output to the DMS_ECU 42 of the DMS 31c via the CP_ECU 21 (see Figure 2 Refer to [Vdata] for the symbol.

[0122] In response to this, in step S12, the DMS_ECU 42 sets a detection duration corresponding to the vehicle speed (refer to Figure 3 ). At the same time, the DMS_ECU 42 starts counting time using an internal clock (not shown) or the like.

[0123] Next, in step S13 , the DMS_ECU 42 checks whether a change in the driver's behavior is detected.

[0124] In step S14, the DMS_ECU 42 checks whether the detection duration time set in the process of step S12 has passed. If it is confirmed that the set time has passed, the process proceeds to step S15. Otherwise, if the set time has not passed, the process returns to step S13.

[0125] In step S15, the DMS_ECU 42 executes a predetermined alarm notification process. The predetermined alarm notification process performed here is the following process.

[0126] First, the alarm output unit 42b outputs a predetermined alarm information signal to the notification device 31d based on the detection result of the behavior change detection unit 42a. In response, the notification device 31d performs an alarm notification in a predetermined manner (visual or auditory display). After that, a series of processing ends (return).

[0127] As described above, according to the first embodiment, when the driving assist control is not executed, the DMS 31 c performs detection within the reference detection duration T. At this time, the reference detection duration T is set according to the vehicle speed.

[0128] On the other hand, when one or more of the plurality of driving assistance controls are being executed, the DMS 31 c sets the detection duration time according to the driving assistance control being executed, and also sets the detection duration time according to the vehicle speed.

[0129] Specifically, in the process of executing only the first driving assistance control, the first detection duration T1 is set by adding the first time α to the reference detection duration T. In the process of executing the driving assistance control including the second driving assistance control, the second detection duration T2 is set by adding the second time β to the reference detection duration T. In the process of executing the driving assistance control including the third driving assistance control, the third detection duration T3 is set by adding the third time γ to the reference detection duration T.

[0130] In the DMS 31c of the present embodiment, by performing such setting control, it is possible to suppress the frequent issuance of unnecessary alarms during vehicle travel, thereby suppressing the influence on the driver and contributing to improving the safety during vehicle travel.

[0131] However, since the third driving assistance control among the above-mentioned plurality of driving assistance controls performs acceleration / deceleration control and steering control at a predetermined steering angle or more, specifically, automatic lane change control (ALC) is mainly assumed.

[0132] Here, the third driving assistance control is a control started at a timing when the DMS_ECU 42 of the DMS 31 c receives a lane change instruction signal outputted by the driver operating the direction indicator lever 31 b .

[0133] Under normal circumstances, after the driver operates the direction indicator lever 31b in the desired direction for lane change, the driver visually checks the surrounding conditions in the front-rear direction (especially to the side and rear of the vehicle) on the adjacent lane on the side where the lane change is desired. At this time, the driver's behavior and posture, including the direction of the face and the direction of the line of sight, change significantly compared to the normal posture of the driver when driving forward.

[0134] In this case, when the direction of the behavior and posture change detected by the DMS 31c is consistent with the direction in which the lane change is expected, it can be inferred that the driver's behavior and posture change after the lane change instruction signal is issued is the behavior required by the third driving assistance control. Therefore, in this case, the DMS 31c does not output the warning notification information.

[0135] However, when the direction of the behavior and posture change detected by the DMS 31c is inconsistent with the direction in which the lane change is expected, it can be inferred that the driver's behavior is obviously unrelated to the third driving assistance control for the driver's behavior and posture change after the lane change instruction signal is issued. Therefore, in this case, the DMS 31c should output the early warning notification information.

[0136] The following uses Figure 5 Taking this into consideration, the operation of the occupant monitoring device of the second embodiment of the present invention will be briefly described. The configuration of the occupant monitoring device of the second embodiment is the same as that of the first embodiment described above. In this embodiment, only a part of the operation (a part of the processing sequence) in the DMS 31c is slightly different.

[0137] Figure 5 FIG. 2 is a flowchart showing a part of the operation of the occupant monitoring device according to the second embodiment of the present invention. The operation of the occupant monitoring device according to the present embodiment is basically the same as that of the first embodiment described above. Figure 5 In the flowchart of the first embodiment described above (refer to Figure 4 ) The same processing steps are assigned the same step numbers and their descriptions are omitted. Figure 5 In, with Figure 4 The same processing steps are partially omitted in the figure. Figure 5 Only the parts different from the first embodiment will be described.

[0138] Figure 5 The processing of steps S11, S12, and S13 is exactly the same as that of the first embodiment described above.

[0139] In the process of step S13 , when the behavior change detection unit 42 a of the DMS_ECU 42 detects a behavior change, the process proceeds to the next step S13A.

[0140] exist Figure 5 In step S13A, the DMS_ECU 42 confirms whether a lane change instruction signal issued by operating the direction indicator lever 31b is detected. The DMS_ECU 42 can confirm that the direction indicator lever 31b is operated by detecting the lane change instruction signal. It should be noted that in this case, the lane change instruction signal also includes information related to the operation direction of the direction indicator lever 31b, that is, the direction of the lane change desired by the driver.

[0141] Here, when a lane change instruction signal including lane change direction information is confirmed, the process proceeds to step S13B. On the other hand, when a lane change instruction signal is not confirmed, the process proceeds to step S14.

[0142] In step S13B, the DMS_ECU 42 checks whether the detection information such as the facial orientation detected in the process of the above step S13 is consistent with the indicated direction of the direction indicator rod 31b detected in the process of the above step S13A (the direction in which the lane change is expected). Here, if it is confirmed that the facial orientation is consistent with the desired direction of the lane change, the process proceeds to step S14. In this case, the setting of the third detection duration is maintained. On the other hand, if the facial orientation is inconsistent with the desired direction of the lane change, the process proceeds to step S13C.

[0143] In step S13C, the DMS_ECU 42 sets a reference detection duration T. Thereafter, the process proceeds to step S14. The subsequent processes are completely the same as those of the first embodiment described above.

[0144] As described above, according to the second embodiment, substantially the same effect as that of the first embodiment can be achieved. In addition, according to the present embodiment, when the third driving assist control is started, for the driver's behavior detected by the behavior change detection unit 42a of the DMS_ECU 42, in particular, when the orientation of the face is consistent with the direction in which the lane change is desired, the setting of the third detection duration is maintained, and when it is inconsistent, the setting of the reference detection duration is switched.

[0145] Therefore, by performing such control, it is possible to further suppress the issuance of unnecessary warnings during the execution of the third driving assistance control.

[0146] The present invention is not limited to the above-mentioned embodiments, and various modifications and applications can be implemented without departing from the scope of the main purpose of the invention. In addition, the inventions of various stages are included in the above-mentioned embodiments, and various inventions can be extracted by appropriate combinations of the disclosed multiple constituent elements. For example, even if several constituent elements are deleted from all the constituent elements shown in the above-mentioned embodiments, the problem to be solved by the invention can be solved and the effect of the invention can be obtained, and the structure in which the constituent elements are deleted can be extracted as an invention. In addition, the constituent elements in different embodiments can also be appropriately combined. The present invention is not limited to its specific embodiments except for the limitations of the attached claims.

Claims

1. A vehicle occupant monitoring device, characterized in that: have: a photographing unit that acquires an image of a driver of the vehicle; and an occupant monitoring control unit, comprising at least a behavior change detection unit and an alarm output unit, wherein the behavior change detection unit detects a behavior change of the driver based on the image acquired by the imaging unit, and the alarm output unit outputs predetermined alarm information based on a detection result of the behavior change detection unit, The behavior change detection unit sets a detection duration for detecting a behavior change of the driver according to a running speed of the vehicle and a type of driving assistance control being executed.

2. The vehicle occupant monitoring device according to claim 1, characterized in that: The types of driving assistance control include at least: A first driving assistance control, which performs only acceleration and deceleration control; a second driving assist control that performs acceleration / deceleration control and steering control at a steering angle smaller than a predetermined steering angle; and The third driving assist control performs acceleration / deceleration control and steering control at a steering angle equal to or greater than a predetermined steering angle.

3. The vehicle occupant monitoring device according to claim 2, characterized in that: When none of the first to third driving assistance controls is executed, the behavior change detection unit performs detection within a reference detection duration time set according to the driving speed of the vehicle, When only the first driving assistance control is executed, the behavior change detection unit performs detection within a first detection duration obtained by adding a first time to the reference detection duration, When the second driving assistance control is executed, the behavior change detection unit performs detection within a second detection duration obtained by adding a second time to the reference detection duration. When the third driving assistance control is executed, the behavior change detection unit performs detection within a third detection duration time obtained by adding a third time to the reference detection duration time.

4. The vehicle occupant monitoring device according to claim 3, characterized in that: The first time is set to be less than the second time, The second time is set to be longer than the first time and shorter than the third time, The third time is set to be longer than the second time, in addition, The reference detection duration is set to a time shorter than the first detection duration, The first detection duration is set to be shorter than the second detection duration, The second detection duration is set to a shorter time than the third detection duration.

5. The vehicle occupant monitoring device according to claim 3, characterized in that: The third driving assistance control is an automatic lane change control started by receiving an on signal of a direction indicator, In the execution of the third driving assistance control, The behavior change detection unit performs detection within a third detection duration when the direction in which the driver's face is facing is consistent with the direction indicated by the direction indicator, The behavior change detection unit performs detection within a reference detection duration when the direction in which the driver's face is facing does not match the direction indicated by the direction indicator.

Images