Occupant monitoring device for vehicle

Abstract

To provide an occupant monitoring device for a vehicle which is used for a vehicle having a driving support control device and an occupant monitoring device mounted on it and is capable of: preventing unnecessary warming from being issued frequently while the vehicle is traveling; and contributing to increase in safety during traveling of the vehicle.SOLUTION: An occupant monitoring device comprises: an imaging part for acquiring an image of a driver of a vehicle; a behavior change detection part for detecting a behavior change of the driver on the basis of the image acquired by the imaging part; and an occupant monitoring control unit which at least has a warning issuing part which outputs prescribed warning information on the basis of a detection result of the behavior change detection part. The behavior change detection part sets a detection continuation time for detecting a behavior change of the driver in accordance with a travel speed of the vehicle and a type of driving support control which is being executed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an occupant monitoring device for a vehicle such as an automobile, which monitors the driver and issues a predetermined alarm at a predetermined timing in a predetermined case, thereby ensuring the safety of the vehicle. [Background technology]

[0002] In recent years, passenger monitoring devices have been put to practical use in vehicles such as automobiles, which are configured to monitor the faces and behavior of passengers including the driver (also called the driver) using an imaging device such as a camera, and to issue a predetermined alarm when it detects, for example, the driver looking away, falling asleep, etc. By installing such a passenger monitoring device in a vehicle, for example, the driver can be allowed to concentrate on driving the vehicle, thereby enabling the vehicle to be driven safely.

[0003] It is generally known that the higher the vehicle speed, the shorter the time until an accident occurs due to inattentive driving, drowsiness, etc. Therefore, in this type of occupant monitoring device, for example, the time until the driver's behavior is recognized and it is determined based on the recognition result that the driver is inattentive, etc., is desirably set shorter as the vehicle speed increases. However, if warnings are issued frequently at the same timing as when the vehicle is traveling at high speed, even when the vehicle speed is low, the driver may find it annoying.

[0004] Therefore, in conventional passenger monitoring devices, various techniques have been proposed, such as in JP 2014-89589 A, for example, that devise the timing of issuing a warning or the like depending on the situation.

[0005] The occupant monitoring device disclosed in JP 2014-89589 A and the like changes the timing of issuing an alarm when it detects that the driver is looking away, dozing, etc., depending on the distance to the preceding vehicle, the vehicle speed of the vehicle, etc. This makes it possible to suppress frequent issuance of alarms.

[0006] Meanwhile, in recent years, in the field of vehicles such as automobiles, various driving assistance control devices have been proposed and are beginning to be put into practical use, which provide control to assist the driver in performing multiple driving operations in order to reduce the burden of driving operations on the driver and improve safety when operating the vehicle.

[0007] In general, the driver's driving load is reduced while driving assistance control is being executed, and therefore the driver can have time to monitor the situation in a wider field of vision. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2014-89589 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, the driver's behavior in monitoring the surroundings may be mistakenly recognized as inattentive driving by the conventional passenger monitoring device. In this case, the passenger monitoring device will frequently issue unnecessary alarms. As a result, the driver may feel annoyed in this case as well. Furthermore, when an unnecessary alarm is issued, the driver's attention may be directed to the alarm source, narrowing the driver's field of vision. Therefore, in this case, the safety of the vehicle may be impaired.

[0010] In the conventional occupant monitoring device disclosed in the above-mentioned JP 2014-89589 A and the like, no consideration is given to control during driving assistance control, and there is a possibility that the above-mentioned problems may occur.

[0011] The present invention aims to provide an occupant monitoring device for a vehicle equipped with a driving assistance control device and an occupant monitoring device, which can suppress the frequent issuance of unnecessary alarms while the vehicle is traveling and can contribute to improving safety while the vehicle is traveling. [Means for solving the problem]

[0012] In order to achieve the above-mentioned object, one embodiment of the present invention is an occupant monitoring device for a vehicle comprising an occupant monitoring control unit having at least an imaging unit that acquires an image of the vehicle driver, a behavior change detection unit that detects a change in behavior of the driver based on the image acquired by the imaging unit, and an alarm output unit that outputs predetermined alarm information based on the detection result of the behavior change detection unit, wherein the behavior change detection unit sets a detection duration for detecting a change in behavior of the driver depending on the traveling speed of the vehicle and the type of driving assistance control currently being executed. Effect of the Invention

[0013] According to the present invention, in a vehicle equipped with a driving assistance control device and an occupant monitoring device, it is possible to provide an occupant monitoring device for a vehicle that can suppress the frequent issuance of unnecessary alarms while the vehicle is traveling and can contribute to improving safety while the vehicle is traveling. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a vehicle in which an occupant monitoring device according to a first embodiment of the present invention is mounted; [Diagram 2] FIG. 2 is a block diagram conceptually illustrating the configuration of the occupant monitoring device of FIG. 1; [Diagram 3] Graph conceptually showing the setting of the detection duration in the passenger monitoring device according to the first embodiment of the present invention. [Figure 4] 1 is a flowchart showing the operation of the passenger monitoring device according to the first embodiment of the present invention. [Diagram 5] 11 is a flowchart showing a part of the operation of the passenger monitoring device according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present invention will now be described with reference to the illustrated embodiments. Before describing the details of the occupant monitoring device of the first embodiment of the present invention, the schematic configuration of a vehicle in which the occupant monitoring device is installed will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the schematic configuration of a vehicle in which the occupant monitoring device of the first embodiment of the present invention is installed.

[0016] 1, only the main components directly related to the present invention are shown, and other components are omitted because they are substantially similar to those of a conventional vehicle of the same type. Only the main components directly related to the present invention will be described below.

[0017] As shown in FIG. 1, a vehicle 1 equipped with an occupant monitoring device of this embodiment (referred to as DMS in FIG. 1; see reference symbol 31c) is equipped with at least a driving assistance control device (camera unit 10; described in detail below), a vehicle speed detection device 39, an occupant monitoring device (DMS 31c), and an alarm device 31d.

[0018] A driving assistance control device is a device and a component unit that can selectively execute multiple types of driving assistance control that assist the driving operation of a vehicle, or execute several driving assistance controls simultaneously. To this end, the driving assistance control device mainly assists the driving operation of the accelerator, brake (braking device), and steering (steering device) among the components that perform the driving operation of the vehicle, and performs acceleration / deceleration control and steering control of the vehicle.

[0019] The driving assistance control device may have a plurality of driving modes, e.g. (1) A manual driving mode in which a driver performs various driving operations (accelerator, brake, and steering operations) to accelerate, decelerate, steer, etc., and a notification or warning for driving assistance (hereinafter referred to as an assist notification) is provided at a predetermined timing in a predetermined case. (2) An assisted driving mode in which driving assistance controls such as acceleration / deceleration control and steering assistance control intervene at predetermined times under predetermined circumstances while assuming that the driver is actively driving the vehicle; (3) An autonomous driving mode that controls the vehicle to drive automatically without the need for driver operation. The driver can selectively set any of these driving modes. Note that the driving assistance control device does not necessarily have to include the automatic driving mode (3) above.

[0020] Specific driving assistance controls executed in such a plurality of driving modes include, for example, (A) a first driving assistance control that performs only acceleration / deceleration control; (B) a second driving assistance control that performs acceleration / deceleration control and steering control less than a predetermined steering angle; (C) Third driving assistance control that performs acceleration / deceleration control and steering control above a specified steering angle, etc. The driver can arbitrarily set these multiple driving assistance controls selectively or simultaneously set several driving assistance controls.

[0021] Here, in this embodiment, (A1) The first driving assistance control is Adaptive Cruise Control (ACC), (A2) The second driving assistance control is Active Lane Keep Centering Control (ALKC), (A3) The third driving assistance control is Auto Lane Changing Control (ALC), Each of them is considered to be equivalent.

[0022] In the above (B), the steering angle less than the predetermined angle is assumed to be a steering angle that prevents a traveling vehicle from departing from its own lane, while in the above (C), the steering angle equal to or greater than the predetermined angle is assumed to be a steering angle that causes a traveling vehicle to change lanes from its own lane to an adjacent lane.

[0023] These first to third driving assistance controls are executed at predetermined timings in predetermined cases, thereby enabling the host vehicle to automatically and stably travel along a designated travel lane while maintaining a distance between the host vehicle and a preceding vehicle.

[0024] In normal cases, for example, (a) Setting for performing only the first driving assistance control (ACC), (b) The first driving assistance control and the second driving assistance control are set simultaneously (ACC+ALKC), (c) The first driving assistance control and the third driving assistance control are set simultaneously (ACC+ALC), (d) The first driving assistance control, the second driving assistance control, and the third driving assistance control are set simultaneously (ACC+ALKC+ALC); Settings such as the following are often used.

[0025] In the above setting (a), control is performed to allow the host vehicle to continue traveling following the preceding vehicle while maintaining the distance between the host vehicle and the preceding vehicle.

[0026] In the setting of (b) above, the host vehicle is controlled to continue traveling along a determined travel lane while following the preceding vehicle and maintaining a distance between the host vehicle and the preceding vehicle.

[0027] In the setting of (c) above, when the vehicle is traveling following the preceding vehicle while maintaining the distance between the vehicle and the preceding vehicle, the vehicle performs an automated lane change control to change the lane of the vehicle from the lane in which the vehicle is traveling to an adjacent lane when a lane change instruction signal is received. In this case, the lane change instruction signal is a signal that is output by, for example, the driver operating the turn signal lever 31b (see FIG. 1).

[0028] In the setting of (d) above, when the vehicle is traveling along a determined driving lane while following the preceding vehicle and maintaining a distance between the vehicle and the preceding vehicle, if a lane change instruction signal is received, automatic lane change control is performed to change the lane of the vehicle from the lane in which the vehicle is traveling to an adjacent lane at the time of receiving the signal.

[0029] Specifically, the driving assistance control device is an in-vehicle camera device, or a camera unit 10, including a stereo camera 11 fixed, for example, in a front upper center portion within a cabin (not shown) of the vehicle 1.

[0030] As shown in FIG. 1, the camera unit 10 includes a stereo camera 11, an image processing unit (denoted as "IPU" in FIG. 1) 12, an image recognition unit 13, a control unit 14, and the like.

[0031] The stereo camera 11 is formed with two cameras, a main camera 11a and a sub camera 11b. The main camera 11a and the sub camera 11b are arranged, for example, in the vehicle cabin at symmetrical positions with respect to the center in the vehicle width direction, facing the front of the vehicle. Each of the main camera 11a and the sub camera 11b is composed of, for example, an imaging optical system, an imaging element such as a CMOS image sensor, and a processing circuit that processes imaging signals, etc. (detailed configuration is not shown).

[0032] With this configuration, the stereo camera 11 acquires two image data from two different viewpoints targeting a predetermined range of the surrounding environment in front of the vehicle using the main camera 11a and the sub camera 11b at a predetermined imaging period synchronized with each other. Then, stereo image data is generated based on the two image data acquired in this way. This stereo image data is surrounding environment information that represents the surrounding environment while the vehicle is traveling. The surrounding environment information (image data) generated by the stereo camera 11 is output to the image processing unit 12.

[0033] The image processing unit 12 is a configuration unit or circuit unit that performs predetermined image processing on the surrounding environment information (image data representing the surrounding environment while the vehicle is traveling) acquired by the stereo camera 11. The image processing unit 12 performs, for example, processing to detect edges of various objects (objects, lane markings, etc.) displayed on the image.

[0034] The image processing unit 12 also acquires distance information based on the amount of positional shift of corresponding edges on the left and right images based on the stereo image data, and generates image information including the distance information (distance image information). The distance image information generated in the image processing unit 12 is output to the image recognition unit 13.

[0035] The image recognition unit 13 calculates the road curvature [1 / m] of the left and right dividing lines of the road on which the vehicle is traveling (the vehicle's traveling path) and the width between the left and right dividing lines (lane width) based on the distance image information and the like input from the image processing unit 12. Various well-known means are used to calculate the road curvature and lane width.

[0036] In addition, the image recognition unit 13 performs a predetermined pattern matching based on the distance image information acquired by the stereo camera 11 to recognize three-dimensional objects extending along the road (such as guardrails, curbs, and other vehicles in the vicinity), parking frame lines marked by dividing lines on the road surface within a parking lot, etc., three-dimensional structures such as car stops that define parking space areas, and gap spaces between adjacent vehicles, etc.

[0037] Here, the recognition of three-dimensional objects (object recognition) in the image recognition unit 13 recognizes, for example, the type of three-dimensional object, the height of the three-dimensional object, the width dimension of the three-dimensional object, the distance from the vehicle to the three-dimensional object, the moving speed of the three-dimensional object, the relative speed between the three-dimensional object and the vehicle, and the relative distance between three-dimensional objects (for example, the lateral distance between a curb on the side of the road and a dividing line nearby).

[0038] Various pieces of information recognized by the image recognition unit 13 are output as surrounding environment information to the control unit 14. In this case, the camera unit 10 including the image recognition unit 13 functions as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle.

[0039] The control unit 14 included in the camera unit 10 is a configuration unit or circuit unit that controls the camera unit 10 as well as the entire driving assistance control device, and generally performs driving assistance control of the vehicle 1 in which the driving assistance control device is installed.

[0040] Various control units, such as a cockpit control unit (CP_ECU) 21, an engine control unit (E / G_ECU) 22, a transmission control unit (T / M_ECU) 23, a brake control unit (BK_ECU) 24, and a power steering control unit (PS_ECU) 25, are connected to the control unit 14 via an in-vehicle communication line such as a CAN (Controller Area Network).

[0041] A human-machine interface (referred to as "HMI" in FIG. 1) 31 disposed near the driver's seat is connected to the CP_ECU 21. The HMI 31 is configured to include, for example, various operating members (31b), various sensing devices (31a, 31c), and an alarm device 31d.

[0042] Here, the various operating members include, for example, an operating switch for instructing the starting or stopping of the occupant monitoring device (DMS31c), a driving mode change switch for selecting or switching the driving mode, a plurality of operating switches for instructing the starting or stopping of the driving assistance control (not shown), as well as a turn signal lever 31b for issuing a lane change instruction signal, etc.

[0043] Here, the multiple driving modes are, for example, a manual driving mode accompanied by an assist notification, an assisted driving mode in which driving assistance control is performed by intervening in some driving operations at a predetermined timing in a predetermined case, and an automatic driving mode in which automatic driving control is performed by intervening in all driving operations while the vehicle is traveling, etc. Note that the automatic driving mode does not necessarily have to be included.

[0044] The various sensing devices include a steering touch sensor (ST touch sensor in FIG. 1) 31a, a driver monitoring system (hereinafter abbreviated as DMS) 31c, which is an occupant monitoring device, etc. The steering touch sensor 31a is a sensing device that detects the steering state of the steering wheel (steered wheels) by the driver.

[0045] The DMS31c is a configuration unit, a circuit unit, or a sensing device that monitors the driver of the vehicle, detects changes in the driver's behavior, and outputs predetermined warning information or the like at a predetermined timing according to the detection result. The DMS31c is configured as an in-vehicle monitor system including, for example, an in-vehicle camera. The DMS31c can be started or stopped by arbitrarily operating a predetermined operation switch.

[0046] Here, the change in the driver's behavior is assumed to be, for example, a change in the direction of the face, a change in the direction of the line of sight, a change in the degree of eye opening, a change in facial expression, etc. The DMS31c continuously monitors the driver's behavior to detect a change in posture (loosening of posture) or a decrease in alertness (change in the degree of eye opening, etc.) The DMS31c, which is an occupant monitoring device, can also detect the riding conditions of occupants including the driver.

[0047] Here, the configuration of the DMS 31c which is the passenger monitoring device of this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram conceptually showing an outline of the configuration of the DMS 31c which is the passenger monitoring device of this embodiment.

[0048] The DMS 31c includes an imaging unit 41 and a DMS_ECU 42. The imaging unit 41 acquires images of occupants including a driver of the vehicle, performs predetermined image processing on the acquired images, and outputs the images to the DMS_ECU 42.

[0049] The DMS_ECU 42 is an occupant monitoring control unit that controls the DMS 31c. The DMS_ECU 42 is a configuration unit or a circuit unit including, for example, an electronic control unit (ECU) etc. The DMS_ECU 42 has a behavior change detection unit 42a and an alarm output unit 42b.

[0050] The behavior change detection unit 42a is a configuration unit or circuit unit that detects changes in the driver's behavior, etc., based on, for example, an image acquired by the imaging unit 41, or outputs from other sensing devices (such as the ST touch sensor 31a, etc.) or various operating members (such as the direction indicator lever 31b, etc.). The behavior change detection unit 42a also detects the riding conditions of passengers including the driver. Furthermore, the behavior change detection unit 42a also performs personal recognition by performing facial recognition of the driver.

[0051] The warning output unit 42b is a configuration unit or a circuit unit that outputs a predetermined warning information signal to the notification device 31d based on the detection result of the behavior change detection unit 42a.

[0052] The notification device 31d is a device that notifies the driver of a predetermined warning in a predetermined form (visual display or auditory display) according to the warning information output from the occupant monitoring device (DMS 31c). The notification device 31d is composed of, for example, a touch panel display device (visual display device), a sound device (auditory display device) including a speaker, etc., a combination meter that combines various instruments, etc.

[0053] The CP_ECU 21 is a configuration unit or circuit unit that, upon receiving a control signal from the control unit 14, appropriately notifies the driver of various information in a predetermined form (visual or audible display) using the notification device 31d included in the HMI 31.

[0054] Here, the various information notified by the notification device 31d includes, for example, various types of warning information, the implementation status of driving assistance control, various types of information related to the surrounding environment of the vehicle, etc. Furthermore, the notification device 31d notifies the driver of a predetermined attention-calling warning, etc., upon receiving a warning information signal, etc., output from the warning output unit 42b.

[0055] In addition, the CP_ECU 21 outputs various input information, such as instruction signals input by the driver using various operating members included in the HMI 31 (for example, on / off instructions for various driving assistance controls, switching or selection instructions for driving modes, etc.), to the control unit 14.

[0056] The output side of the E / G_ECU 22 is connected to a throttle actuator 32 of an electronically controlled throttle, etc. Also, the input side of the E / G_ECU 22 is connected to various sensors such as an accelerator sensor.

[0057] The E / G_ECU 22 is a drive device, a structural unit, or a circuit unit, that performs drive control on the throttle actuator 32 based on a control signal from the control unit 14 or detection signals from various sensors to generate a driving force for the vehicle. In this way, the E / G_ECU 22 adjusts the amount of intake air for the engine to generate a desired engine output. The E / G_ECU 22 also outputs signals such as the accelerator opening degree detected by various sensors (accelerator sensor, etc.) to the control unit 14.

[0058] An output side of the T / M_ECU 23 is connected to a hydraulic control circuit 33. In addition, an input side of the T / M_ECU 23 is connected to various sensors such as a shift position sensor (not shown).

[0059] The T / M_ECU 23 is a configuration unit or circuit unit that performs hydraulic control for the hydraulic control circuit 33 based on the engine torque signal estimated by the E / G_ECU 22 and detection signals from various sensors. As a result, the T / M_ECU 23 operates friction engagement elements, pulleys, and the like provided in the automatic transmission to shift the engine output at a desired gear ratio. The T / M_ECU 23 also outputs signals such as the shift position detected by the various sensors to the control unit 14.

[0060] A brake actuator 34 for adjusting the brake fluid pressure output to the brake wheel cylinders provided on the respective wheels is connected to the output side of the BK_ECU 24. In addition, various sensors (not shown) such as a brake pedal sensor, a yaw rate sensor, a longitudinal acceleration sensor, and a vehicle speed sensor are connected to the input side of the BK_ECU 24.

[0061] The BK_ECU 24 is a braking device, a structural unit or a circuit unit, that performs drive control of the brake actuator 34 to perform vehicle braking control based on a control signal from the control unit 14 or detection signals from various sensors. As a result, the BK_ECU 24 appropriately generates braking force on each wheel for performing forced braking control, yaw rate control, etc. on the vehicle. The BK_ECU 24 also outputs signals of the brake operation state, yaw rate, longitudinal acceleration, vehicle speed, etc. detected by various sensors to the control unit 14.

[0062] An electric power steering motor 35 that applies a steering torque to the steering mechanism by the rotational force of the motor is connected to the output side of the PS_ECU 25. In addition, various sensors (not shown) such as a steering torque sensor and a steering angle sensor are connected to the input side of the PS_ECU 25.

[0063] The PS_ECU 25 is a steering device, a structural unit or a circuit unit, that performs drive control of the electric power steering motor 35 to control the steering of the vehicle based on a control signal from the control unit 14 or detection signals from various sensors. In this way, the PS_ECU 25 generates a steering torque for the steering mechanism. The PS_ECU 25 also outputs signals of the steering torque, steering angle, etc. detected by the various sensors to the control unit 14.

[0064] In addition, various sensors, such as a locator unit 36, an on-board radar device 37, a rear sensor 38, and a vehicle speed detection device 39, are connected to the control unit 14. The locator unit 36 ​​is configured to include a GNSS sensor 36a and a high-precision road map database (road map DB) 36b.

[0065] The GNSS sensor 36a receives positioning signals transmitted from a plurality of positioning satellites to measure the position (latitude, longitude, altitude, etc.) of the vehicle.

[0066] The road map DB 36b is a large-capacity storage medium such as a hard disk drive (HDD) device or a solid state drive (SSD) device, and stores highly accurate three-dimensional road map information (dynamic map).

[0067] The road map DB36b contains lane data required for autonomous driving, such as lane width data, lane center position coordinate data, lane travel azimuth data, and speed limits. This lane data is stored at intervals of several meters for each lane on the road map. Furthermore, the road map DB36b contains dynamic information that changes from moment to moment, such as traffic regulations, road construction, accidents, and congestion.

[0068] Furthermore, the locator unit 36 ​​can acquire real-time information on the surrounding environment at the vehicle position measured by the GNSS sensor 36a (for example, traffic congestion information, weather information, and various information on parking lots, etc.) by communicating with an external system or the like (not shown). In this case, the weather information includes, for example, fog occurrence information in the area including the vehicle position, rainfall information, snowfall information, snow accumulation information, temperature and humidity information, etc.

[0069] The road map DB 36b also holds information on various facilities, parking lots, etc. For example, based on a request signal from the control unit 14, the road map DB 36b outputs road map information of a set range based on the vehicle position measured by the GNSS sensor 36a as surrounding environment information to the control unit 14. Thus, in this embodiment, the road map DB 36b, together with the GNSS sensor 36a, functions as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle.

[0070] The on-board radar device 37 is composed of a plurality of sensors, for example, a plurality of millimeter wave radars. Here, the plurality of millimeter wave radars detect three-dimensional objects such as pedestrians and vehicles traveling alongside, as well as structures (e.g., curbs, guard rails, walls of buildings, plants, and other three-dimensional objects) provided on the edge of the road (e.g., the edge on the shoulder side) by receiving and analyzing the reflected waves from objects in response to the outputted radio waves. Furthermore, the plurality of millimeter wave radars also detect three-dimensional obstacles present on the road. In this case, the plurality of millimeter wave radars detect the width of the three-dimensional object, the position of the representative point of the three-dimensional object (relative position and relative distance to the vehicle itself), the relative speed, and other specific information related to the three-dimensional object.

[0071] The multiple sensors (multiple millimeter wave radars, etc.) included in the on-board radar device 37 are disposed, for example, on the left and right sides of the front bumper (referred to as front left and right side sensors) and on the left and right sides of the rear bumper (referred to as rear left and right side sensors). The front left and right side sensors detect, as surrounding environment information, three-dimensional objects existing in areas diagonally forward and to the left and right and to the sides of the vehicle, which are difficult to recognize in the image from the stereo camera 11. The rear left and right side sensors detect, as surrounding environment information, three-dimensional objects existing in areas diagonally forward and to the left and right and to the rear of the vehicle, which are difficult to recognize with the front left and right side sensors.

[0072] In this manner, in this embodiment, the on-vehicle radar device 37 functions as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle. Information acquired by each sensor of the on-vehicle radar device 37 is sent to the image recognition unit 13 via the control unit 14.

[0073] The rear sensor 38 is, for example, configured with a sonar device that uses ultrasonic waves to measure the distance to and shape of an object. At least one (or multiple) rear sensor 38 is provided, for example, on the rear bumper. The rear sensor 38 detects, as surrounding environment information, a three-dimensional object present in an area behind the vehicle that is difficult to recognize using the rear left and right side sensors. Thus, in this embodiment, the rear sensor 38 functions as a surrounding environment recognition device that recognizes the surrounding environment of the vehicle.

[0074] The coordinates of each object outside the vehicle contained in the surrounding environment information recognized by the image recognition unit 13, the locator unit 36, the on-board radar device 37, the rear sensor 38, etc. are all converted by the control unit 14 into coordinates of a three-dimensional coordinate system with the center of the vehicle as the origin.

[0075] The vehicle speed detection device 39 is a component unit or a circuit unit that detects the traveling speed of the vehicle based on the outputs of a plurality of wheel speed sensors. The vehicle speed detection device 39 detects the vehicle speed under the control of the control unit 14.

[0076] In this case, a plurality of wheel speed sensors included in the vehicle speed detection device 39 detect pulse signals (wheel speed pulses) that are generated in proportion to the rotation speed of each wheel (generally four wheels) of the vehicle. Then, the vehicle speed detection device 39 detects the vehicle speed based on the wheel rotation speed data of each wheel acquired by the plurality of wheel speed sensors, for example, by calculating the average of each wheel speed data. The vehicle speed data detected by the vehicle speed detection device 39 is output to the control unit 14.

[0077] The control unit 14 executes driving assistance control of the vehicle based on various information acquired by the camera unit 10 and various sensors (a locator unit 36, an on-board radar device 37, a rear sensor 38, and a vehicle speed detection device 39), etc.

[0078] In this case, the driving assistance control includes, for example, engine output control by the E / G_ECU 22 and torque distribution control for each drive wheel, as well as forward or reverse direction control by control of the transmission by the T / M_ECU 23, and individual braking control (brake control) for each wheel by the BK_ECU 24, thereby providing vehicle driving assistance control as needed.

[0079] In addition to the above, the various sensors for acquiring surrounding environmental information may include, for example, a LiDAR (Light Detection And Ranging) device that uses laser light to measure the distance to and shape of an object, a near-infrared sensor, an outside air temperature sensor, and the like.

[0080] Furthermore, the above-mentioned stereo camera 11 mainly observes a predetermined field of view in front of the vehicle. In addition, a plurality of camera devices of the same form may be provided to observe predetermined field of view in the sides and rear of the vehicle. In this way, the entire surroundings of the vehicle can be observed.

[0081] All or some of the image recognition unit 13, control unit 14, CP_ECU 21, E / G_ECU 22, T / M_ECU 23, BK_ECU 24, PS_ECU 25, DMS 31c, etc. are configured by a processor including hardware.

[0082] Here, the processor is composed of a well-known configuration including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile memory, a non-volatile storage, a non-transitory computer readable medium, and the like, and peripheral devices thereof.

[0083] The software programs executed by the CPU and fixed data such as data tables are stored in advance in the ROM, non-volatile memory, non-volatile storage device, etc. Then, the CPU reads out the software programs stored in the ROM, etc., expands them in the RAM, and executes them, and the software programs appropriately refer to various data, etc., thereby realizing the functions of the above-mentioned components and components (13, 14, 21 to 25, 31c), etc.

[0084] The processor may be configured with a semiconductor chip such as a Field Programmable Gate Array (FPGA), etc. Each of the above components and components (13, 14, 21 to 25, 31c) may be configured with an electronic circuit.

[0085] Furthermore, the software program may be in a form in which the whole or part of the software program is recorded as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

[0086] The operation of the DMS 31c, which is the passenger monitoring device of this embodiment configured as described above, will be described below.

[0087] First, the vehicle 1 equipped with the occupant monitoring device (DMS31c) of this embodiment is equipped with a driving assistance control device that has a plurality of driving modes and can selectively execute a plurality of driving assistance controls or execute some driving assistance controls simultaneously. With this configuration, the vehicle 1 runs while executing one or a plurality of driving assistance controls selected by the driver according to the plurality of driving modes selected by the driver.

[0088] In addition, when the occupant monitoring device (DMS31c) is activated while the vehicle 1 is traveling, the DMS31c sets a detection duration in accordance with the vehicle speed data of the vehicle 1. Here, the detection duration refers to the time used by the DMS31c to detect a change in the driver's behavior. If a predetermined behavior change is detected within this detection duration, an alarm is output. In this case, when one or more driving assistance controls are being executed, the DMS31c further sets a predetermined detection duration in accordance with the type of driving control being executed.

[0089] FIG. 3 is a graph conceptually showing the setting of the detection duration in the occupant monitoring device (DMS31c) of this embodiment. As shown in FIG. 3, the detection duration (T) of the occupant monitoring device (DMS31c) is basically set to be shorter as the vehicle speed (V) increases. In the manual driving mode, when the vehicle 1 is traveling in a state where none of the driving assistance controls among the multiple driving assistance controls is being executed, the DMS31c sets the detection duration (T) according to only the vehicle speed (V). The graph shown as "BASE" in FIG. 3 is the detection duration at this time. In the following description, the detection duration at this time is referred to as the reference detection duration T.

[0090] Normally, when the driver is driving the vehicle 1 without executing the driving assistance control, the driver constantly monitors the surrounding situation while continuing to drive the vehicle 1. The detection time by the DMS 31c at this time is set as a reference (reference detection duration T).

[0091] When the vehicle 1 is traveling in the assisted driving mode with any one of the first to third driving assistance controls being executed among the multiple driving assistance controls, the DMS31c sets the detection duration time (T) according to the type of driving assistance control being executed in addition to the vehicle speed (V).

[0092] 3, the graph indicated as "BASE+α" is the first detection duration T1 when the vehicle 1 is traveling in a state where only the first driving assist control is being executed. Here, the symbol α indicates a first time period added to the reference detection duration T.

[0093] In a normal case, a driver who is driving the vehicle 1 while only the first driving assistance control is being executed needs to mainly monitor the surroundings ahead (such as a preceding vehicle). In consideration of this, it can be estimated that the behavior change performed by the driver to monitor the surroundings ahead while the first driving assistance control is being executed is a behavior performed by the driver of his / her own will. The first detection duration T1 is set by adding the first time α as the time required for this.

[0094] 3, the graph indicated as "BASE+β" is the second detection duration T2 when the vehicle 1 is traveling in a state in which the second driving assist control is being executed. Here, the symbol β indicates a second time period added to the reference detection duration T.

[0095] In normal cases, the driver of the vehicle 1 while the second driving assistance control is being executed needs to mainly monitor the surroundings in front and to the left and right (the preceding vehicle, etc., and the left and right lane markings, etc.). In consideration of this, while the second driving assistance control is being executed, it can be estimated that the behavioral changes that the driver makes to monitor the surroundings in front and to the left and right are behaviors that the driver makes of his / her own volition. The second detection duration T2 is set by adding the second time β as the time required for this.

[0096] 3, the graph indicated as "BASE+γ" is the third detection duration T3 when the vehicle 1 is traveling in a state in which the third driving assist control is being executed. Here, the symbol γ indicates a third time period added to the reference detection duration T.

[0097] Normally, a driver who is driving the vehicle 1 while the third driving assistance control is being executed needs to monitor the surroundings (preceding vehicles, etc., left and right dividing lines, etc., and following vehicles in the own lane and adjacent lanes, etc.) in addition to the front and left and right sides. In consideration of this, it can be estimated that the behavior change caused by the driver to monitor the surroundings in the front, left and right sides and rear while the third driving assistance control is being executed is a behavior that the driver is performing of his / her own will. A third detection duration T3 is set by adding a third time γ as the time required for this.

[0098] Here, the first time α, the second time β, and the third time γ are set to have a relationship of "α≦β≦γ." That is, the first time α is set to be equal to or less than the second time β. The second time β is set to be equal to or more than the first time α and equal to or less than the third time γ. The third time γ is set to be equal to or more than the second time β.

[0099] For this reason, the reference detection duration T is set to a time shorter than the first detection duration T1. The first detection duration T1 is set to a time shorter than the second detection duration T2. ​​The second detection duration T2 is set to a time shorter than the third detection duration T3.

[0100] 4 is a flowchart showing the operation of the passenger monitoring device (DMS31c) of the first embodiment of the present invention. Hereinafter, the operation of the passenger monitoring device (DMS31c) of this embodiment will be described with reference to the flowchart of FIG.

[0101] In step S1 of Fig. 4, the control unit 14 checks whether the DMS 31c is running. If it is checked that the DMS 31c is running, the process proceeds to the next step S2. If the DMS 31c is not running, the process is repeated. That is, the flowchart starts with the start of the DMS 31c as a trigger.

[0102] In step S2, the control unit 14 checks whether the driving mode setting in the driving assistance control device is the manual driving mode. If it is checked that the manual driving mode is set, the process proceeds to step S3. In this case, the driving mode setting information is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. If a driving mode other than the manual driving mode is set, the process proceeds to step S4.

[0103] In step S3, the DMS_ECU 42 sets a reference detection duration time T.

[0104] In step S4, the control unit 14 checks whether the driving mode setting in the driving assistance control device is the assisted driving mode. If it is checked that the assisted driving mode is set, the process proceeds to step S5. In this case, the driving mode setting information is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. If a driving mode other than the assisted driving mode is set (i.e., if the automatic driving mode is set), the process ends (returns).

[0105] In the vehicle 1 equipped with the occupant monitoring device (DMS31c) of this embodiment, when the vehicle is driven in the automatic driving mode, occupant monitoring by the DMS31c is not required. For this reason, in this flowchart, when the automatic driving mode is set, after a series of processes is completed, the process moves to another control process (not shown) corresponding to the automatic driving mode, for example.

[0106] In step S5, the control unit 14 checks the type of driving assist control that has been set. If only the first driving assist control has been set, the process proceeds to step S6. In this case, setting information for the driving assist control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. If multiple driving assist controls have been set (which may include the first driving assist control), the process proceeds to step S7.

[0107] In step S6, the DMS_ECU 42 sets a first detection duration time T1.

[0108] In step S7, the control unit 14 checks the type of driving assist control that has been set. If the setting includes the second driving assist control, the process proceeds to step S8. In this case, setting information for the driving assist control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. If a driving assist control other than the second driving assist control has been set (a plurality of driving assist controls may be set), the process proceeds to step S7.

[0109] In step S8, the DMS_ECU 42 sets a second detection duration T2.

[0110] In step S9, the control unit 14 checks the type of driving assist control that has been set. If the setting includes the third driving assist control, the process proceeds to step S10. In this case, setting information for the driving assist control is transmitted from the control unit 14 to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21. If a driving assist control other than the third driving assist control has been set (or multiple driving assist controls), the process ends (returns).

[0111] In the driving assistance control device in this embodiment, first to third driving assistance controls are illustrated as examples of multiple driving assistance controls. However, the driving assistance controls that can be executed by the driving assistance control device are not limited to the above three types, and other driving assistance controls may exist. In this embodiment, for the other driving assistance controls, after a series of processes is completed, a transition is made to a control process (not shown) corresponding to the other driving assistance control.

[0112] In step S10, the DMS_ECU 42 sets a third detection duration T3.

[0113] In step S11, the control unit 14 detects current vehicle speed data by controlling the vehicle speed detection device 39. The vehicle speed data input from the vehicle speed detection device 39 to the control unit 14 is further output to the DMS_ECU 42 of the DMS 31c through the CP_ECU 21 (see symbol [Vdata] in FIG. 2).

[0114] In response to this, in step S12, the DMS_ECU 42 sets a detection duration corresponding to the vehicle speed for the reference detection duration T or any of the first to third detection durations (T1 to T3) set in any of the processing steps S3, S6, S8, and S10 described above (see FIG. 3). At the same time, the DMS_ECU 42 starts timer timekeeping using an internal clock (not shown) or the like.

[0115] Next, in step S13, the DMS_ECU 42 checks whether or not a change in the driver's behavior has been detected.

[0116] In step S14, the DMS_ECU 42 checks whether the detection duration time set in the process of step S12 described above has elapsed. If it is confirmed that the set time has elapsed, the process proceeds to step S15. If the set time has not elapsed, the process returns to step S13.

[0117] In step S15, the DMS_ECU 42 executes a predetermined alarm notification process. The predetermined alarm notification process executed here is as follows.

[0118] First, the warning output unit 42b outputs a predetermined warning information signal to the notification device 31d based on the detection result of the behavior change detection unit 42a. In response to this, the notification device 31d issues a warning in a predetermined form (visual or audible display). After that, the series of processes is terminated (return).

[0119] As described above, according to the first embodiment, when driving assist control is not being executed, the DMS 31c performs detection for the reference detection duration T, and at that time, sets the reference detection duration T according to the vehicle speed.

[0120] On the other hand, when one or more of the driving assistance controls are being executed, the DMS31c sets the detection duration time according to the driving assistance control being executed, and sets the detection duration time according to the vehicle speed.

[0121] Specifically, when only the first driving assist control is being executed, a first detection duration T1 is set by adding a first time α to the reference detection duration T. When driving assist control including the second driving assist control is being executed, a second detection duration T2 is set by adding a second time β to the reference detection duration T. When driving assist control including the third driving assist control is being executed, a third detection duration T3 is set by adding a third time γ to the reference detection duration T.

[0122] In the DMS 31c of the present embodiment, by performing such setting control, it is possible to suppress the frequent issuance of unnecessary alarms while the vehicle is traveling, thereby suppressing the influence on the driver and contributing to improving the safety during the vehicle traveling.

[0123] Incidentally, the third driving assistance control among the multiple driving assistance controls described above performs acceleration / deceleration control and steering control above a predetermined steering angle, and specifically, is mainly intended as automated lane change control (ALC).

[0124] Here, the third driving assist control is a control that is started at the timing when the DMS_ECU 42 of the DMS 31c receives a lane change instruction signal that is output in response to the driver operating the direction indicator lever 31b.

[0125] In normal cases, after the driver operates the turn signal lever 31b in the desired direction to change lanes, he or she visually checks the surrounding conditions in the front and rear directions (particularly the sides and rear of the vehicle) in the adjacent lane on the side where the lane change is desired. At this time, the driver's behavior and posture, including the direction of the driver's face and the direction of the driver's line of sight, will change significantly from the normal posture when the driver is driving facing forward.

[0126] In this case, when the DMS 31c detects the driver's behavior and posture change after the lane change instruction signal is issued, if the detected direction of the behavior and posture change matches the desired direction of lane change, it can be estimated that the behavior is necessary for the third driving support control. Therefore, in this case, the DMS 31c does not output the warning notification information.

[0127] However, when the DMS 31c detects the driver's behavior and posture change after the lane change command signal is issued, if the detected behavior and posture change direction does not match the desired lane change direction, it can be estimated that the driver's behavior is clearly not related to the third driving support control. Therefore, in this case, the DMS 31c should output early warning notification information.

[0128] Taking the above into consideration, the operation of the passenger monitoring device according to the second embodiment of the present invention will be briefly described below with reference to Fig. 5. The configuration of the passenger monitoring device according to the second embodiment is similar to that of the first embodiment. In this embodiment, only a part of the operation (a part of the processing sequence) of the DMS 31c is slightly different.

[0129] Fig. 5 is a flowchart showing part of the operation of an occupant monitoring device of a second embodiment of the present invention. The operation of the occupant monitoring device of this embodiment is basically substantially the same as that of the first embodiment described above. Therefore, in the flowchart of Fig. 5, the same processing steps as those in the flowchart of the first embodiment described above (see Fig. 4) are given the same step numbers, and their explanations are omitted. Also, in Fig. 5, some of the processing steps that are the same as those in Fig. 4 are not illustrated. Hereinafter, the operation of the occupant monitoring device of this embodiment will be explained with reference to Fig. 5, with only the parts that differ from the first embodiment.

[0130] The processes in steps S11, S12, and S13 in FIG. 5 are entirely similar to those in the first embodiment described above.

[0131] If a behavior change is detected by the behavior change detection unit 42a of the DMS_ECU 42 in the process of step S13, the process proceeds to the next step S13A.

[0132] In step S13A of Fig. 5, the DMS_ECU 42 checks whether or not it has detected a lane change instruction signal issued by operating the turn signal lever 31b. By detecting the lane change instruction signal, the DMS_ECU 42 can check that the turn signal lever 31b has been operated. In this case, the lane change instruction signal also includes information regarding the operating direction of the turn signal lever 31b, i.e., the direction of the lane change desired by the driver.

[0133] If a lane change signal including lane change direction information is detected, the process proceeds to step S13B, whereas if a lane change signal is not detected, the process proceeds to step S14.

[0134] In step S13B, the DMS_ECU 42 checks whether the detection information such as the face direction detected in the process of step S13 described above matches the direction indicated by the turn signal lever 31b (the desired direction of lane change) detected in the process of step S13A described above. If it is confirmed that the face direction matches the desired direction of lane change, the process proceeds to step S14. In this case, the setting of the third detection duration is maintained. On the other hand, if the face direction does not match the desired direction of lane change, the process proceeds to step S13C.

[0135] In step S13C, the DMS_ECU 42 sets a reference detection duration time T. Then, the process proceeds to step S14. The subsequent processes are exactly the same as those in the first embodiment described above.

[0136] As described above, according to the second embodiment, it is possible to obtain substantially the same effects as those of the first embodiment. Furthermore, according to this embodiment, when the third driving assistance control is started, the setting of the third detection duration is maintained when the driver's behavior detected by the behavior change detection unit 42a of the DMS_ECU 42, particularly when the face direction and the desired lane change direction match, and when they do not match, the setting is switched to the reference detection duration.

[0137] Therefore, by performing such control, it is possible to further suppress the issuance of unnecessary warnings while the third driving assist control is being executed.

[0138] The present invention is not limited to the above-mentioned embodiments, and various modifications and applications can be implemented within the scope of the invention. Furthermore, the above-mentioned embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed components. For example, if some components are deleted from all the components shown in the above-mentioned embodiments, and the problem to be solved by the invention can be solved and the effect of the invention can be obtained, the configuration from which the components are deleted can be extracted as the invention. Furthermore, components from different embodiments may be appropriately combined. The present invention is not restricted by its specific embodiments except as limited by the attached claims. [Explanation of symbols]

[0139] 1. Vehicle 10...Driver assistance control device (camera unit) 11...Stereo camera 11a…Main camera 11b…Sub camera 12...Image processing unit (IPU) 13...Image recognition unit 14…Control unit 21...Cockpit control unit (CP_ECU) 22…Engine control unit (E / G_ECU) 23...Transmission control unit (T / M_ECU) 24...Brake control unit (BK_ECU) 25…Power steering control unit (PS_ECU) 31…Human Machine Interface (HMI) 31a…Steering touch sensor (ST touch sensor) 31b...Turn signal lever 31d…Alarm device 32...Throttle actuator 33...Hydraulic control circuit 34…Brake actuator 35...Electric power steering motor 36…Locator unit 36a…GNSS sensor 36b…Road map DB 37... Vehicle-mounted radar device 38…Rear sensor 39...Vehicle speed detection device 41...imaging unit 42…DMS_ECU (Occupant Monitoring Control Unit) 42a...Behavior change detection unit 42b…Alarm output section T: Reference detection duration T1: First detection duration T2: Second detection duration T3: Third detection duration α…First Time β…Second Time γ…The third time

Claims

1. An imaging unit that captures an image of a driver of the vehicle; an occupant monitoring control unit including at least a behavior change detection unit that detects a behavior change of the driver based on the image acquired by the imaging unit, and a warning output unit that outputs predetermined warning information based on a detection result of the behavior change detection unit; An occupant monitoring device comprising: The behavior change detection unit is Depending on the traveling speed of the vehicle and the type of driving assistance control being executed, Setting a detection duration for detecting a change in the driver's behavior A vehicle occupant monitoring device comprising:

2. The types of driving assistance control include at least: A first driving assistance control that performs only acceleration / deceleration control; A second driving assist control that performs acceleration / deceleration control and steering control less than a predetermined steering angle; A third driving assistance control that performs acceleration / deceleration control and steering control for a predetermined steering angle or more; 2. The vehicle occupant monitoring device according to claim 1, further comprising:

3. The behavior change detection unit is When none of the first to third driving assistance controls is being executed, performing detection for a reference detection duration time that is set according to the traveling speed of the vehicle; When only the first driving assistance control is being executed, performing detection for a first detection duration obtained by adding a first time to the reference detection duration; When the second driving assistance control is being executed, performing detection for a second detection duration obtained by adding a second time to the reference detection duration; When the third driving assistance control is being executed, The detection is performed for a third detection duration obtained by adding a third time to the reference detection duration.

3. A vehicle occupant monitoring device according to claim 2.

4. the first time period is set to be equal to or less than the second time period; the second time period is set to be equal to or greater than the first time period and equal to or less than the third time period; the third time period is set to be equal to or greater than the second time period; Also, the reference detection duration is set to a time shorter than the first detection duration, The first detection duration is set to a time shorter than the second detection duration, The second detection duration is set to a time shorter than the third detection duration.

4. A vehicle occupant monitoring device according to claim 3.

5. The third driving assistance control is an automobile lane change control that is started in response to an on signal of a turn indicator, The behavior change detection unit is During execution of the third driving assistance control, When the direction in which the driver's face is facing and the direction indicated by the turn signal coincide with each other, detection is performed for a third detection duration, When the direction in which the driver's face is facing does not match the direction indicated by the turn signal, detection is performed for a reference detection duration.

4. A vehicle occupant monitoring device according to claim 3.

Images