Mobile device control device, mobile device control method, and program

Abstract

To appropriately determine whether the detection area of ​​the detection device is blocked or not. [Solution] A mobile vehicle control device comprising: an acquisition unit that acquires characteristic points of an occupant obtained from the detection result of a detection device that detects an occupant riding in a mobile vehicle; and a determination unit that performs a driver state determination to determine whether the driver of the mobile vehicle is in a state in which the vehicle is drivable based on the characteristic points, wherein the determination unit determines that the detection area of ​​the detection device is blocked if the index based on the characteristic points is less than a preset threshold based on the characteristic points obtained from the detection result of a driver who is in a state in which the vehicle is drivable, and does not perform a driver state determination using the detection result of the blocked state, and determines that the vehicle is not in a blocked state if the index based on the characteristic points is equal to or greater than a preset threshold, and performs a driver state determination using the detection result of not being in a blocked state.

Description

Technical Field

[0001] The present invention relates to a movement control device, a movement control method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transport system that takes into account people in vulnerable positions among traffic participants. In order to achieve this, research and development have focused on further improving traffic safety and convenience through research and development of preventive safety technologies such as driving support that supports the driving of drivers. For example, a technique for determining whether a driver is in a state where the driver can drive a vehicle from an image of the driver driving the vehicle is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional driver state determination system, there are cases where, although the driver is in a state where the driver can drive the vehicle, it is determined that the driver is not in a state where the driver can drive the vehicle. For example, when the camera for imaging is blocked by an obstacle such as a towel or the driver's hand, it may be determined that the driver is not in a state where the driver can drive the vehicle. In the conventional driver state determination system, it may not be possible to appropriately determine between a state blocked by an obstacle and a state where the driver cannot drive the vehicle.

[0005] To solve the above-mentioned problems, this invention aims to provide a mobile vehicle control device, a mobile vehicle control method, and a program that can appropriately determine whether or not the detection area of ​​a detection device is blocked. More specifically, one of its objectives is to prevent the system from determining that the driver is not in a condition to operate the vehicle when an obstruction is present that blocks the detection area of ​​the detection device. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The mobile device control device, mobile device control method, and program according to this invention employ the following configuration. (1) A mobile vehicle control device according to one aspect of the present invention includes an acquisition unit that acquires characteristic points of an occupant obtained from the detection result of a detection device that detects an occupant riding in a mobile vehicle, and a determination unit that performs a driver state determination based on the characteristic points to determine whether the driver of the mobile vehicle is in a state in which the mobile vehicle is drivable, wherein the determination unit determines that the detection area of ​​the detection device is in a blocked state if the index based on the characteristic points is less than a preset threshold based on the characteristic points obtained from the detection result of a driver who is in a state in which the mobile vehicle is drivable, and does not perform the driver state determination using the detection result of the blocked state, and determines that the blockage state is not present if the index based on the characteristic points is equal to or greater than the preset threshold, and performs the driver state determination using the detection result of the non-blockage state.

[0007] (2): In the embodiment of (1) above, the determination unit determines the direction in which the obstructing object blocking the detection area appears based on the feature points, and changes the threshold according to the direction.

[0008] (3) In the embodiment of (2) above, if the determination unit determines that the obstruction state has occurred due to the obstruction appearing from the direction of the driver's seat of the moving body, it determines the obstruction state by comparing the index based on the feature points with a first threshold, and if the determination unit determines that the obstruction state has occurred due to the obstruction appearing from the direction of the passenger seat of the moving body, it determines the obstruction state by comparing the index based on the feature points with a second threshold that is smaller than the first threshold.

[0009] (4) In the embodiment of (2) above, the determination unit determines the cutoff state by comparing the index, which is the cumulative amount of the feature points, with the threshold.

[0010] (5) In the embodiment of (2) above, the determination unit determines that the detection area is in the blocked state if the index based on the feature points is less than the pre-set threshold for a set time or longer.

[0011] (6) In the embodiment of (5) above, if the determination unit determines that the obstruction state has occurred due to the obstruction appearing from the direction of the driver's seat of the moving body, it sets the setting time to the first setting time, and if the determination unit determines that the obstruction state has occurred due to the obstruction appearing from the direction of the passenger seat of the moving body, it sets the setting time to the second setting time, which is longer than the first setting time.

[0012] (7) In the embodiment of (1) above, the detection device is provided on or near the navigation device of the moving body, and the determination unit does not determine the blocked state or the driver state when operation of the navigation device of the moving body is detected.

[0013] (8) In the embodiment of (1) above, the determination unit includes a control unit that decelerates or stops the mobile body if the determination unit determines in the driver status determination that the driver is not able to drive the mobile body.

[0014] (9) A mobile vehicle control method according to one aspect of the present invention involves a computer acquiring characteristic points of an occupant obtained from the detection result of a detection device that detects an occupant riding in a mobile vehicle, performing a driver state determination based on the characteristic points to determine whether the driver of the mobile vehicle is in a state in which the mobile vehicle is drivable, determining that the detection area of ​​the detection device is blocked and that the driver state determination is not performed using the detection result of the blocked state, and determining that the driver state determination is not performed if the index based on the characteristic points is equal to or greater than the pre-set threshold, and determining that the state is not blocked and performing the driver state determination using the detection result of the state not being blocked.

[0015] (10): A program according to one aspect of the present invention causes a computer to perform the following processes: acquiring characteristic points of an occupant obtained from the detection result of a detection device for detecting an occupant riding in a mobile vehicle; and performing a driver state determination based on the characteristic points to determine whether the driver of the mobile vehicle is in a state in which the vehicle is drivable, wherein if the index based on the characteristic points is less than a pre-set threshold based on the characteristic points obtained from the detection result of a driver who is in a state in which the vehicle is drivable, the detection area of ​​the detection device is blocked and the detection result of the blocked state is not performed; and if the index based on the characteristic points is equal to or greater than the pre-set threshold, the detection area of ​​the detection device is not blocked and the detection result of the blocked state is performed. [Effects of the Invention]

[0016] According to the embodiments described in (1) to (10) above, the mobile object control device, mobile object control method, or program can appropriately determine whether or not the detection area of ​​the detection device is blocked by making a determination based on an index and a threshold based on feature points.

[0017] According to the embodiment of (2) above, the determination unit can improve the accuracy of determining whether or not the detection area of ​​the detection device is blocked by changing the threshold according to the direction in which the obstruction appears.

[0018] According to the embodiment of (3) above, the determination unit can make a careful determination when an obstruction appears from a direction that is unlikely to be blocked by setting the second threshold value to a value greater than the first threshold, thereby improving the accuracy of determining whether or not the detection area of ​​the detection device is blocked.

[0019] According to the embodiment of (4) above, the determination unit can appropriately determine whether or not the detection area of ​​the detection device is blocked by making a determination based on the accumulated amount of feature points.

[0020] According to the embodiment of (5) above, the determination unit can improve the accuracy of determining whether or not the detection area of ​​the detection device is blocked by making a determination based on the duration of the state in which the index is below the threshold.

[0021] According to the embodiment of (6) above, the determination unit can improve the accuracy of determining whether or not the detection area of ​​the detection device is blocked by setting the second setting time to a value greater than the first setting time. The determination unit can suppress the easy determination of the driver's state when an obstruction appears from the direction of the passenger seat, by setting the second time to be longer than when an obstruction appears from the driver's seat, thereby preventing the easy determination of the driver's state when the passenger's body blocks the detection area. For example, a passenger may operate the navigation device for, for example, more than the first time. In this case, the easy determination of the driver's state is suppressed.

[0022] According to the aspect of (7) above, when an operation of the navigation device is detected, the determination unit does not determine that it is in a blocked state or does not perform a driver state determination, so that it is possible to appropriately determine whether or not the detection area of the detection device is blocked. Specifically, even when the index is not less than the threshold value, if an operation of the navigation device is detected, since there is a very high possibility that there is an obstacle such as a hand due to the operation, it is determined that it is in a blocked state or a driver state determination is not performed.

[0023] According to the aspect of (8) above, when it is determined in the driver state determination that the driver is not able to drive the moving body, the control unit can appropriately control the moving body by decelerating or stopping the moving body.

Brief Description of the Drawings

[0024] [Figure 1] It is a configuration diagram of a vehicle system 1 using a vehicle control system according to an embodiment. [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​ [Modes for carrying out the invention]

[0025] [Overall structure] Figure 1 is a diagram showing the configuration of a vehicle system 1 utilizing a vehicle control system according to an embodiment. The vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharge power from a secondary battery or fuel cell. This embodiment is described as being applicable to a vehicle, but it may be applied to other mobile bodies instead of a vehicle.

[0026] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, an object recognition device 16, an in-cabin camera 18, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU 60, an operator 80, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. The configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The driving assistance device 100 is an example of a "mobile vehicle control device".

[0027] Camera 10 is a digital camera that utilizes a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is installed. When imaging the area in front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, etc. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0028] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to determine at least the position (distance and bearing) of the object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of the object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0029] LIDAR14 irradiates light (or electromagnetic waves with a wavelength close to light) around vehicle M and measures the scattered light. Based on the time from emission to reception, LIDAR14 detects the distance to the target. The irradiated light is, for example, pulsed laser light. LIDAR14 can be attached to any location on vehicle M.

[0030] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the position, type, speed, etc., of an object. The object recognition device 16 outputs the recognition results to the driver assistance device 100. The object recognition device 16 may output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the driver assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

[0031] The in-vehicle camera 18 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 18 is mounted in a position and orientation that allows it to capture the driver's face in the vehicle M. For example, the in-vehicle camera 18 is mounted near the HMI 30 or the navigation HMI 52 (display unit of the navigation device 50). The in-vehicle camera 18 may be mounted anywhere in the vehicle M as long as the above conditions are met. The in-vehicle camera 18 generates image information that captures the scenery inside the vehicle. The captured image information includes the driver's face. The image information may also include the state of occupants (passengers) seated in seats other than the driver's seat in the vehicle M. The in-vehicle camera 18 transmits the generated image information to the driver assistance device 100. The in-vehicle camera 18 is an example of a "detection device". Instead of (or in addition to) the in-vehicle camera 18, a detection device capable of detecting the conditions inside the vehicle may be provided. For example, a device capable of detecting the conditions inside the vehicle using optical methods may be provided, and an index obtained from the detection results of this detection device may be used in the processing described later.

[0032] The communication device 20 communicates with other vehicles in the vicinity of vehicle M, or with various server devices via a wireless base station, for example, by using a cellular network, Wi-Fi network, Bluetooth®, DSRC (Dedicated Short Range Communication), etc.

[0033] The HMI30 presents various information to the occupants of vehicle M and accepts input operations from the occupants. The HMI30 includes various display devices, speakers, buzzers, touch panels, switches, keys, etc. The HMI30 is equipped with a display device. The display device is a display device, so-called multi-information display, that displays various information in vehicle M, such as a speedometer that shows the vehicle's speed or a tachometer that shows the rotational speed of the internal combustion engine equipped in vehicle M, which is installed on the instrument panel of vehicle M.

[0034] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity around the vertical axis, a compass sensor for detecting the orientation of the vehicle M, and a sensor for detecting the steering rotation angle, etc.

[0035] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, speaker, touch panel, keys, etc. The navigation HMI 52 may be partially or completely shared with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter referred to as the route on the map) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52, by referring to the first map information 54. The first map information 54 is, for example, information in which the road shape is represented by links indicating roads and nodes connected by those links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be implemented, for example, by the functions of a terminal device such as a smartphone or tablet held by an occupant. The navigation device 50 may transmit the current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0036] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the map route provided by the navigation device 50 into multiple blocks (for example, every 100m with respect to the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 makes decisions such as which lane from the left the vehicle should travel in. If there is a branching point on the map route, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel a reasonable route to proceed to the branching point. For example, if the vehicle M reaches a predetermined distance before a branching road it is traveling on, the recommended lane determination unit 61 determines the lane connecting to the branching road as the recommended lane. The recommended lane determination unit 61 and the second map information 62 may be functional units or information included in other devices such as a driver assistance device 100. The recommended lane information is provided to the driver, for example, via the HMI.

[0037] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may also include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

[0038] The control element 80 includes, for example, a steering wheel, accelerator pedal, brake pedal, shift lever, and other control elements. The control element 80 is equipped with a sensor that detects the amount of operation or whether or not an operation is performed, and the detection result is output to the driver assistance device 100, or to some or all of the driving force output device 200, brake device 210, and steering device 220. The steering wheel does not necessarily have to be annular in shape and may take the form of an irregularly shaped steering wheel, joystick, buttons, etc.

[0039] The driver assistance device 100 includes, for example, a recognition unit 110, an acquisition unit 120, a determination unit 130, and a control unit 140. The recognition unit 110, the acquisition unit 120, the determination unit 130, and the control unit 140 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance in a storage unit (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driver assistance device 100 when the storage medium (non-transient storage medium) is mounted on the drive device.

[0040] The recognition unit 110 recognizes the position and state, such as speed and acceleration, of objects around the vehicle M based on surrounding information input from the camera 10, radar device 12, and LIDAR 14 via the object recognition device 16. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive axis) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or it may be represented by a region. The "state" of an object may include the object's acceleration, jerk, or "action state" (for example, whether or not it is changing lanes or attempting to change lanes).

[0041] The recognition unit 110 recognizes, for example, the vehicle M's path and other paths in its vicinity. A path is a route on a road that includes the lane in which the vehicle M travels and the opposing lane. For example, the recognition unit 110 recognizes a path by comparing the pattern of road markings obtained from the second map information 62 (for example, an arrangement of solid lines SL and dashed lines) with the pattern of road markings around the vehicle M recognized from the image captured by the camera 10. The recognition unit 110 may also recognize a path by recognizing not only road markings but also path boundaries (road boundaries) that include road markings, shoulders, curbs, median strips, guardrails, etc. In this recognition, the position of the vehicle M obtained from the navigation device 50 and the processing results from INS may be taken into consideration. The recognition unit 110 recognizes stop lines, obstacles, red lights, toll booths, and other road events that exist in the vicinity of the vehicle M.

[0042] The recognition unit 110 recognizes the behavior of the vehicle M based on the detection results of the vehicle sensor 40. For example, when recognizing a road, the recognition unit 110 recognizes the position and attitude of the vehicle M relative to the road. The recognition unit 110 may recognize, for example, the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of travel of the vehicle M, as the relative position and attitude of the vehicle M relative to the road. Alternatively, the recognition unit 110 may recognize the position of the vehicle M's reference point relative to one of the side edges of the road (road lane markings or road boundary) as the relative position of the vehicle M relative to the road.

[0043] The acquisition unit 120 acquires characteristic points of the occupants of vehicle M based on image information obtained from the in-vehicle camera 18 (an example of "detection result"). The image information is information about the scenery inside the vehicle and includes information about the occupants of vehicle M. The characteristic points are the outlines of parts such as the eyes and mouths of the occupants of vehicle M, and the boundaries of the occupants of vehicle M.

[0044] The determination unit 130 performs a driver state determination based on feature points. Driver state determination is the determination of whether or not the driver of vehicle M is in a state where they can drive vehicle M. Based on the positional relationship of multiple feature points in the driver's outline, the determination unit 130 derives driver information such as the driver's posture, facial expression, and gaze information, and performs a driver state determination based on the driver information. Driver state determination is a process that determines whether or not there is an abnormality in the driver, such as when the vehicle M is in motion, due to the driver's posture being in an unsuitable state for driving. For example, if the driver's posture is tilted significantly forward or to the side, or if the driver's posture is significantly deviated from that of driving, and this continues for a period of time or longer, it is presumed that there is an abnormality in the driver.

[0045] The control unit 140 controls all components included in the driver assistance device 100 and the vehicle system 1. For example, the control unit 140 controls the steering of the vehicle M, controls the speed of the vehicle M, and controls the HMI 30 to provide information to the driver. If there is an abnormality with the driver, the control unit 140 outputs a warning or slows down or stops the vehicle M. Details of the processing of the determination unit 130 and the control unit 140 will be described later.

[0046] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle M to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU that controls them. The ECU controls the above configuration according to information input from the driver assistance device 100 or from the operator 80.

[0047] The braking system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driver assistance device 100 or from the control element 80, so that brake torque corresponding to the braking operation is output to each wheel.

[0048] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driver assistance device 100 or from the control element 80.

[0049] [overview] This embodiment provides driving assistance when the driver of vehicle M is not in a state to drive vehicle M, and implements control to ensure the safety of the occupants of vehicle M and the surrounding area. For example, when vehicle M is in motion, the driving assistance device 100 supports safe driving by outputting a warning or slowing down or stopping vehicle M if the driver's posture is tilted significantly forward or to the side (when it is presumed that the posture is unsuitable for driving and an abnormality has occurred in the driver). In this embodiment, the determination unit 130 decides whether or not to perform a driver state determination based on the feature point information acquired by the acquisition unit 120.

[0050] [Processing details] In this embodiment, the determination of whether or not to perform a driver state determination is made based on the feature points. The determination unit 130 compares an index based on the feature points acquired by the acquisition unit 120 with a threshold value and decides whether or not to perform a driver state determination.

[0051] The acquisition unit 120 acquires multiple feature points of the occupants of vehicle M that are included in the image information of the interior of vehicle M, based on the image information of the interior of vehicle M captured by the in-vehicle camera 18.

[0052] The determination unit 130 compares an index, which is the cumulative amount of feature points, with a threshold. The threshold may be a pre-set value based on feature points obtained from image information including a driver who is in a state where the vehicle M can be driven. This driver may not be the driver that the driving assistance device 100 is trying to determine, but rather a driver who was the subject of experiments or simulations to set the threshold. The index is not limited to the cumulative value, but may also be an index obtained by statistically processing the cumulative amount, or an index obtained by performing a predetermined process on the feature points.

[0053] The determination unit 130 determines that the system is not in an obstructed state if the indicator is above the threshold, and determines that the system is in an obstructed state if the indicator is below the threshold. An obstructed state is a state in which at least a part of the detection area of ​​the in-vehicle camera 18 is obstructed. When an obstructed state is determined, the image information captured by the in-vehicle camera 18 includes the obstructing object.

[0054] If the determination unit 130 determines that the power is shut off, it does not perform a driver status determination. If it determines that the power is not shut off, it performs a driver status determination.

[0055] Figure 2 shows the interior of a vehicle according to this embodiment. When an occupant of vehicle M operates the navigation HMI 52 of the navigation device, they operate within the operation area AR (for example, the touch panel of the display unit). Operations within the operation area AR are performed, for example, by the driver D seated in the driver's seat or an occupant seated in the passenger seat.

[0056] Figure 3 is an enlarged view of the AR operating area. As shown in Figure 3, the in-vehicle camera 18 is located around the navigation HMI 52 (in Figure 3, at or near the center of the navigation HMI 52 in the width direction, in the vertical direction). Therefore, the images captured by the in-vehicle camera 18 may include the hands and arms of the occupant operating the navigation HMI 52 or the area around the navigation HMI 52 when the occupant of the vehicle M operates the navigation HMI 52 or the area around the navigation HMI 52.

[0057] Figure 4 shows an example of image information captured by the in-vehicle camera 18. The scene shown in Figure 4 is one in which the occupants of vehicle M are not operating the navigation HMI 52 or the area around the navigation HMI 52.

[0058] The acquisition unit 120 acquires feature points from the image information of Figure 4. Figure 4 does not include any obstructions in the image information, and includes a scene of driver D driving vehicle M and passenger C sitting in the passenger seat. At this time, the cumulative amount of feature points in the image information of Figure 4 is greater than or equal to the threshold. In the image information of Figure 4, the cumulative amount of feature points is greater than or equal to the threshold due to the faces and facial features of the driver and passenger in the image information. The determination unit 130 determines that the scene in Figure 4 is not an obstruction state because the cumulative amount of feature points is greater than or equal to the threshold, and performs a driver state determination.

[0059] Figure 5 shows an example of image information captured by the in-vehicle camera 18. The scene shown in Figure 5 depicts an occupant (occupant C) seated in the passenger seat of vehicle M operating the navigation HMI 52.

[0060] The acquisition unit 120 acquires feature points from the image information in Figure 5. Figure 5 includes the hand of occupant C (obstruction OB). The face of driver D is obscured by the obstruction OB. In Figure 5, since the face of driver D, which is formed from a large number of feature points, is obscured, the cumulative amount of feature points is less than when the obstruction OB is not present. At this time, the determination unit 130 determines that the cumulative amount of feature points is less than the threshold, so the scene in Figure 4 is an obstructed state, and does not perform a driver state determination.

[0061] In the example above, we described an image that includes a passenger in the front seat, but the image to be processed may also be one in which only the driver is captured and the passenger in the front seat is not captured.

[0062] (Flowchart 1) Figure 6 is a flowchart showing an example of the processing flow performed by the driver assistance device 100. The flowchart shown in Figure 6 is, for example, executed when vehicle M is in motion.

[0063] First, the acquisition unit 120 acquires feature points from the image information captured by the in-vehicle camera 18 (step S100). Next, the determination unit 130 calculates the cumulative amount of feature points (step S102). Next, the determination unit 130 determines whether or not the cumulative amount of feature points is less than a threshold (step S104).

[0064] If the determination unit 130 determines that the cumulative amount of feature points is less than the threshold, it determines that the system is in a blocked state and does not perform a driver state determination (step S106). If the determination unit 130 determines that the cumulative amount of feature points is not less than the threshold, that is, that the cumulative amount of feature points is greater than or equal to the threshold, it determines that the system is not in a blocked state and performs a driver state determination (step S108). This completes the processing of this flowchart.

[0065] As described above, the determination unit 130 can appropriately determine whether or not the detection area of ​​the detection device is blocked by making a determination based on the accumulated amount of feature points and a threshold. For example, if the image information includes the hand of an occupant operating the navigation HMI 52, which is an obstruction, the accumulated amount of feature points acquired will be less than when the image information does not include an obstruction. Therefore, the determination unit 130 can appropriately determine that an obstruction is present by comparing the accumulated amount of feature points with a threshold. If an obstruction is present, the determination unit 130 decides not to perform a driver state determination based on feature points, as it may not be possible to accurately determine the driver's state. Specifically, if a driver state determination is performed when the feature point index is below the threshold in an obstruction state, it will be determined that there is an abnormality in the driver. For example, if the driver's body or head is tilted forward or backward, the feature point index will be below the threshold, resulting in a feature point index that has a similar tendency to an obstruction state caused by an obstruction. In this embodiment, taking the above situation into consideration, the index of the feature point and the threshold are compared, and the blocking state is appropriately determined.

[0066] [Change the threshold] Furthermore, the threshold value may be changed depending on the direction from which the obstruction appears. The determination unit 130 calculates the direction from which the obstruction appeared based on the time change of the characteristic points of the obstruction. There are two directions from which the characteristic points appear: the direction of the driver's seat and the direction of the passenger seat. The determination unit 130 sets the threshold value as the first threshold value when the obstruction appears from the direction of the driver's seat, and as the threshold value as the second threshold value when the obstruction appears from the direction of the passenger seat. The second threshold value is smaller than the first threshold value.

[0067] Figure 7 shows an example of image information captured by the in-vehicle camera 18. The image in Figure 7, like that in Figure 5, shows that the driver D's face is obscured by the hand of passenger C, who is operating the navigation HMI 52. Feature point P1 is a set of feature points of the obstruction OB (passenger's hand) at time T. Feature point P2 is a set of feature points of the obstruction OB at time T+1. Feature point P3 is a set of feature points of the obstruction OB at time T+2.

[0068] In the scenario shown in Figure 7, the determination unit 130 determines that the multiple feature points of the obstruction OB have moved from the passenger side to the navigation HMI 52 side in the order of feature point P1, feature point P2, and feature point P3 as time progresses. In other words, the determination unit 130 determines that the direction in which the obstruction OB appeared is the direction of the passenger side. The determination unit 130 may determine the direction in which the obstruction OB appeared based on the time change of the multiple feature points of the obstruction OB, or it may determine it based on the time change of one feature point of interest determined by the determination unit 130 from among the multiple feature points of the obstruction OB.

[0069] (Flowchart 2) Figure 8 is a flowchart showing an example of the processing flow performed by the driver assistance device 100. The flowchart shown in Figure 8 is, for example, executed when vehicle M is in motion.

[0070] First, the acquisition unit 120 acquires feature points from the image information captured by the in-vehicle camera 18 (step S200). Next, the determination unit 130 calculates the cumulative amount of feature points (step S202). Next, the determination unit 130 determines whether the direction in which the obstruction appeared is in the direction of the driver's seat or not (step S204).

[0071] If the determination unit 130 determines that the direction in which the obstruction appeared is towards the driver's seat, it sets the threshold to the first threshold (step S206). If the determination unit 130 determines that the direction in which the obstruction appeared is not towards the driver's seat, that is, that the direction in which the obstruction appeared is towards the passenger seat, it sets the threshold to the second threshold (step S208).

[0072] The determination unit 130 determines whether the cumulative amount of feature points is less than a threshold (step S210). The threshold is either the first threshold or the second threshold determined according to the determination process in step S204. If it is determined that the cumulative amount of feature points is less than the threshold, the determination unit 130 determines that the system is in a blocked state and does not perform a driver state determination (step S212). If it is determined that the cumulative amount of feature points is not less than the threshold, that is, that the cumulative amount of feature points is greater than or equal to the threshold, the determination unit 130 determines that the system is not in a blocked state and performs a driver state determination (step S214). This completes the processing of this flowchart.

[0073] Note that the order of the above processes may be changed. For example, the process in step S202 may be executed after the process in step S206 or step S208. The same applies to the flowchart in Figure 10, which will be described later.

[0074] As described above, the determination unit 130 can improve the accuracy of determining whether the detection area of ​​the detection device is blocked or not by changing the threshold value based on the direction from which the obstruction appears. For example, if an obstruction appears from the passenger side, a smaller threshold value is used to determine whether or not the area is blocked than when the obstruction appears from the driver's side. This allows the determination unit 130 to make a careful determination when an obstruction appears from a direction where blocking is less likely (the passenger side), thereby improving the accuracy of determining whether or not the area is blocked.

[0075] [Decision based on set time] If the determination unit 130 determines that the indicator is below a predetermined threshold, it may determine that the system is not in a blocked state if the state in which the indicator is below the predetermined threshold continues for a set time or longer, and determine that the system is in a blocked state if the state in which the indicator is below the predetermined threshold does not continue for a set time or longer. The predetermined threshold may be the same value as the threshold, first threshold, and second threshold described above, or it may be a value greater than the value described above.

[0076] The set time is a time set in advance, for example, based on the time required when the occupant operates the navigation HMI 52 or its surroundings. The set time may also be changed depending on the direction from which the obstruction appears. The determination unit 130 sets the set time as the first set time when the obstruction appears from the driver's seat direction, and as the second set time when the obstruction appears from the passenger seat direction. The second set time is longer than the first set time.

[0077] Figure 9 shows the time required for each operation of the navigation system. In Figure 9, the vertical axis represents the time required, and the horizontal axis represents the type of operation. The horizontal axis shows the main operations performed on the navigation HMI 52 and its surroundings. Of the operations shown on the horizontal axis, the five underlined operations—"Selecting a song using Bluetooth Audio," "Making a call from the history," "Making a call from the phonebook," "Setting a destination and starting navigation," and "Searching for nearby facilities and starting navigation"—are operations that are likely to be performed by the passenger seat occupant (hereinafter referred to as the passenger), or are performed frequently (first operations). The other operations are operations that are likely to be performed by the driver, or are performed frequently (second operations). Comparing the time required for each operation, it can be seen that the time required for second operations is longer than the time required for first operations. Therefore, the second setting time A2 is set to be longer than the first setting time A1. The first setting time A1 is, for example, the first operation time due to the first operation, or the first operation time plus a buffer time. The first operation time is the time obtained by statistically processing the time during which the first operation is performed. The second set time A2 is, for example, the second operation time due to the second operation, or the time obtained by adding a buffer time to the second operation time. The second operation time is the time obtained by statistically processing the time during which the second operation is performed.

[0078] As described above, by determining whether or not a blocked state exists based on the duration of the state in which the indicator is below a predetermined threshold, the accuracy of determining whether or not a state of blockage by an obstruction can be improved. For example, if the driver scrolls the map screen on the navigation HMI 52 (when an obstruction appears from the driver's seat direction), and the duration of the state in which the indicator is below a predetermined threshold is less than the first set time A1, the determination unit 130 determines that a blocked state exists and does not perform a driver status determination. In this case, it is presumed that the blocked state is not due to a driver malfunction, but rather to the driver's operation of the navigation HMI 52. For example, if a passenger in the front passenger seat searches for nearby facilities and starts guidance on the navigation HMI 52 (when an obstruction appears from the passenger seat direction), and the duration of the state in which the indicator is below a predetermined threshold is less than the second set time A2, the determination unit 130 determines that a blocked state exists and does not perform a driver status determination. In this case, it is presumed that the blocked state is not due to a driver malfunction, but rather to the passenger's operation of the navigation HMI 52.

[0079] (Flowchart 3) Figure 10 is a flowchart showing an example of the processing flow performed by the driver assistance device 100. The explanation will focus on the processing that differs from the flowchart in Figure 8.

[0080] The processes for steps S200 to S206, S208, and S210 to S214 are the same as those described in flowchart 2 of Figure 8 above.

[0081] In step S204, if it is determined that the direction in which the obstruction appeared is the direction of the driver's seat, the determination unit 130 sets the threshold to the first threshold (step S206) and the set time to the first set time (step S207). In step S204, if it is determined that the direction in which the obstruction appeared is not the direction of the driver's seat, the determination unit 130 sets the threshold to the second threshold (step S208) and the set time to the second set time (step S209). Next, the determination unit 130 determines whether the accumulated amount of feature points is less than the threshold (step S210).

[0082] If the determination unit 130 determines that the cumulative amount of feature points is less than a predetermined threshold, it determines whether the duration of the state in which the cumulative amount of feature points is less than the predetermined threshold is less than the set time (step S211). If the determination unit 130 determines that the duration of the interrupted state is less than the set time, it determines that the state is interrupted and does not perform a driver state determination.

[0083] If the determination unit 130 determines in step S210 that the accumulated amount of feature points is not less than the threshold, or if it determines in step S211 that the duration of the interrupted state is not less than the set time, it determines that there is no interrupted state and performs a driver status determination. This completes the processing of this flowchart.

[0084] As described above, the determination unit 130 can improve the accuracy of determining whether or not the vehicle is blocked by an obstruction by changing the threshold and set time based on the direction from which the obstruction appears. For example, if an obstruction appears from the direction of the driver's seat, the determination unit 130 determines that the vehicle is blocked if the accumulated amount of feature points is less than the first threshold for less than the first set time. On the other hand, if an obstruction appears from the direction of the passenger seat, the determination unit 130 determines that the vehicle is blocked if the accumulated amount of feature points is less than the second threshold for less than the second set time. By setting the second set time to a longer time than the first set time, the determination unit 130 can suppress the easy determination of the driver's state due to the passenger's body blocking the detection area. For example, a passenger may operate the navigation system for more than the first hour. In this case, the easy determination of the driver's state is suppressed.

[0085] In the example above, the threshold and the set time were described as being changed, but the determination unit 130 may change the set time according to the direction in which the obstruction appears, without changing the threshold.

[0086] [others] The determination unit 130 may determine that the system is in a blocked state when it detects navigation operation, or it may decide not to perform a driver status determination. Navigation operation is, for example, the operation shown in Figure 9. The navigation device can be any operation in the operation area AR of the navigation HMI 52 or the like.

[0087] As described above, the determination unit 130 can appropriately determine whether the detection area of ​​the detection device is blocked or not by determining that the device is blocked when navigation operation is detected, or by not performing a driver status determination. For example, even if the indicator is not below the threshold, if operation of the navigation device is detected, there is a very high possibility that an obstruction exists, so by determining that the device is blocked, it is possible to avoid performing a driver status determination, which may have low determination accuracy.

[0088] If the determination unit 130 performs a driver status determination and determines that the driver is not in a state to drive the vehicle M, the control unit 140 may perform control (assistance control) of the vehicle M. Not being in a state to drive means, for example, that while the vehicle M is in motion, the driver's posture is tilted forward or to the side more than a predetermined amount, or that the driver's eyelids remain closed for a predetermined amount of time or longer. Not being in a state to drive means that the driver is determined to be unable to perform driving (control) operations such as steering or speed control of the vehicle M.

[0089] The control unit 140 may perform support control, such as automatically reducing the speed of vehicle M or automatically stopping vehicle M. Support control is a control that automatically and safely stops the vehicle M's movement according to the surrounding conditions, thereby ensuring the safety of the vehicle M's occupants and the surrounding area. The control unit 140 executes support control according to the surrounding conditions recognized by the recognition unit 110. Support control may also include warning control. The warning notifies the vehicle M's occupants and the surrounding area that the driver is not in a state to operate vehicle M, or that vehicle M is slowing down or stopping.

[0090] As described above, the control unit 140 can appropriately control the moving vehicle by performing support control when the driver is not in a state to drive the vehicle M. For example, if the driver's posture is tilted forward or to the side beyond a predetermined angle, the control unit 140 determines that the driver is not in a state to drive the vehicle M, and by performing support control, it can decelerate and stop the vehicle M according to the surrounding conditions, thereby ensuring the safety of the occupants of the vehicle M and the surrounding area.

[0091] According to the embodiments described above, the driver assistance device 100 determines whether or not the system is in a blocked state based on its characteristic points. This allows the in-vehicle camera 18 to appropriately determine whether or not the detection area of ​​the detection device is blocked.

[0092] The embodiments described above can be expressed as follows. A memory device that stores the program, Equipped with a hardware processor, The hardware processor executes the program stored in the memory device, The characteristic points of the occupants obtained from the detection results of a detection device that detects occupants on board a moving vehicle are acquired. Based on the aforementioned characteristic points, it is determined whether the driver of the mobile body is in a state where the mobile body can be driven. If the index based on the aforementioned feature points is less than a pre-set threshold based on the feature points obtained from the detection result of a driver who is in a state where the moving object can be driven, it is determined that the detection area of ​​the detection device is in a blocked state, and the driver state determination using the detection result of the blocked state is not performed. If the index based on the aforementioned feature points is equal to or greater than the pre-set threshold, it is determined that the state is not shut off, and the driver state determination is performed using the detection result that the state is not shut off. Mobile device control system.

[0093] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0094] 1. Vehicle System 10 Cameras 12 Radar equipment 14 LIDAR 18. In-car camera 50 Navigation System 120 Acquisition Department 130 Judgment section 140 Control Unit

Claims

1. An acquisition unit that acquires characteristic points of the occupant obtained from the detection results of a detection device that detects occupants on board a moving vehicle, The system includes a determination unit that performs a driver state determination based on the aforementioned characteristic points to determine whether the driver of the mobile body is in a state where the mobile body can be driven, The determination unit, If the index based on the feature points is less than a pre-set threshold based on the feature points obtained from the detection result of a driver who is in a state where the moving object can be driven, it is determined that the detection area of ​​the detection device is in a blocked state, and the driver state determination using the detection result of the blocked state is not performed. If the index based on the aforementioned feature points is equal to or greater than the pre-set threshold, it is determined that the state is not shut off, and the driver state determination is performed using the detection result that the state is not shut off. Mobile device control system.

2. The determination unit determines, based on the feature points, the direction in which the obstructing object blocking the detection area appears, and changes the threshold according to the direction. The mobile device control device according to claim 1.

3. The determination unit, If it is determined that the obstruction state has occurred due to the obstruction appearing from the direction of the driver's seat of the moving body, the obstruction state is determined by comparing the index based on the characteristic points with the first threshold, If it is determined that the obstruction state has occurred due to the obstruction appearing from the direction of the passenger seat of the moving body, the obstruction state is determined by comparing the index based on the feature points with a second threshold that is smaller than the first threshold. The mobile device control device according to claim 2.

4. The determination unit determines the blocking state by comparing the index, which is the cumulative amount of the feature points, with the threshold. The mobile device control device according to claim 2.

5. The determination unit, If the index based on the feature points remains below the pre-set threshold for a set period of time or longer, it is determined that the detection area is blocked. The mobile device control device according to claim 2.

6. The determination unit, If it is determined that the obstruction state has occurred due to the obstruction appearing from the direction of the driver's seat of the moving body, the set time is set to the first set time. If it is determined that the obstruction state has occurred due to the obstruction appearing from the direction of the passenger seat of the moving body, the set time is set to a second set time which is longer than the first set time. The mobile device control device according to claim 5.

7. The detection device is provided on or near the navigation device of the moving object. The determination unit does not perform the determination of the blocked state or the determination of the driver's state if it detects that the navigation device of the moving body is being operated. The mobile device control device according to claim 1.

8. If the determination unit determines in the driver status determination that the driver is not able to operate the mobile body, the control unit is provided to decelerate or stop the mobile body. The mobile device control device according to claim 1.

9. Computers The characteristic points of the occupants obtained from the detection results of a detection device that detects occupants on board a moving vehicle are acquired. Based on the aforementioned characteristic points, a driver status determination is performed to determine whether the driver of the mobile body is in a state where the mobile body can be driven. If the index based on the feature points is less than a pre-set threshold based on the feature points obtained from the detection result of a driver who is in a state where the moving object can be driven, it is determined that the detection area of ​​the detection device is in a blocked state, and the driver state determination using the detection result of the blocked state is not performed. If the index based on the aforementioned feature points is equal to or greater than the pre-set threshold, it is determined that the state is not shut off, and the driver state determination is performed using the detection result that the state is not shut off. A method for controlling a mobile object.

10. On the computer, A process for acquiring characteristic points of the occupant obtained from the detection results of a detection device that detects the occupant on board a moving vehicle, In a process of determining whether the driver of the mobile body is in a state where he or she can drive the mobile body, based on the aforementioned characteristic points, If the index based on the feature points is less than a pre-set threshold based on the feature points obtained from the detection result of a driver who is in a state where the moving object can be driven, the detection area of ​​the detection device is determined to be in a blocked state, and the driver state determination using the detection result of the blocked state is not performed. If the index based on the aforementioned feature points is greater than or equal to the pre-set threshold, it is determined that the state is not shut off, and the driver state determination is performed using the detection result that the state is not shut off. A program that executes the command.

Images