Vehicle control device, vehicle control method, and storage medium

Abstract

The present invention provides a vehicle control device, a vehicle control method, and a storage medium capable of improving the accuracy of determining whether a driver is monitoring the front. The vehicle control device of an embodiment includes: a recognition unit that recognizes other vehicles in the vicinity of a host vehicle; a detection unit that detects the direction of the face or line of sight of the driver of the host vehicle; an output control unit that, based on the position of other vehicles in a second lane adjacent to a first lane where the host vehicle is located, causes the output unit to output prescribed information indicating the presence of other vehicles in the second lane; a driving control unit that performs automatic driving based on the recognition result of the recognition unit and the detection result of the detection unit, and lowers the level of the automatic driving if the direction is outside the permissible range; and a determination unit that determines the permissible range based on the direction detected by the detection unit after the output unit outputs the prescribed information.

Description

Technical Field

[0001] The present invention relates to a vehicle control device, a vehicle control method and a storage medium. Background Art

[0002] The obligation for drivers to monitor the road ahead is being studied depending on the level of autonomous driving. In this regard, there is a known technology for determining the driver's state, such as looking around (see, for example, Patent Document 1).

[0003]

Prior technical literature

[0004] [Patent Literature]

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-91272 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] However, conventional technologies have not been able to determine with sufficient accuracy whether the driver is looking ahead or looking around.

[0008] One aspect of the present invention has been made in consideration of such circumstances, and one object thereof is to provide a vehicle control device, a vehicle control method, and a storage medium capable of improving the accuracy of determining whether the driver is monitoring the front.

[0009] Solutions to Problems

[0010] The vehicle control device, vehicle control method, and storage medium of the present invention employ the following structures.

[0011] One solution (1) of the present invention is a vehicle control device, wherein the vehicle control device comprises: an identification unit that identifies other vehicles existing in the vicinity of the vehicle; a detection unit that detects the direction of the face or line of sight of the driver of the vehicle; an output control unit that, based on the position of other vehicles existing in a second lane adjacent to a first lane where the vehicle is located among the other vehicles identified by the identification unit, causes the output unit to output prescribed information indicating the presence of other vehicles in the second lane; a driving control unit that performs automatic driving to control at least one of the speed and steering of the vehicle based on the identification result of the identification unit and the detection result of the detection unit, and reduces the level of the automatic driving when the direction detected by the detection unit is outside the permissible range; and a determination unit that determines the permissible range based on the direction detected by the detection unit after the output unit outputs the prescribed information.

[0012] In the scheme (2), based on the vehicle control device of the scheme (1) above, the output unit is an indicator provided on the mirror surface of the door rearview mirror of the vehicle or near the door rearview mirror in the cabin of the vehicle, and the output control unit lights up or flashes the indicator when the recognition unit recognizes that the other vehicle is located at a position behind the vehicle in the second lane, and the decision unit determines the allowable range to include the indicator when the detection unit detects that the driver has turned his face or gaze toward the indicator after the indicator is lit or flashed.

[0013] In the aspect (3), based on the vehicle control device of the aspect (2) above, the determination unit determines the permissible range when the number of times the driver turns his face or eyes toward the indicator reaches a predetermined number.

[0014] In the scheme (4), based on the vehicle control device of any one of the schemes (1) to (3) above, the driving control unit reduces the level of the automatic driving when the driver directs his face or gaze outside the permissible range for more than a specified time.

[0015] Another embodiment (5) of the present invention is a vehicle control method, wherein the vehicle control method causes a computer mounted on the vehicle to perform the following processing: identifying other vehicles existing in the vicinity of the vehicle; detecting the direction of the face or line of sight of the driver of the vehicle; causing an output unit to output prescribed information indicating the presence of other vehicles in a second lane adjacent to a first lane where the vehicle is located, based on the position of the other vehicles among the identified other vehicles; performing automatic driving to control at least one of the speed and steering of the vehicle based on the recognition result of the other vehicles and the detection result of the direction of the driver's face or line of sight, and reducing the level of the automatic driving when the detected direction is outside the permissible range; and determining the permissible range based on the direction detected after the output unit outputs the prescribed information.

[0016] Another embodiment (6) of the present invention is a storage medium storing a program, wherein the program is used to cause a computer mounted on the vehicle to execute the following processing: identifying other vehicles existing in the vicinity of the vehicle; detecting the direction of the face or line of sight of the driver of the vehicle; causing an output unit to output prescribed information indicating the presence of other vehicles in a second lane adjacent to the first lane where the vehicle is located, based on the position of other vehicles among the identified other vehicles; performing automatic driving to control at least one of the speed and steering of the vehicle based on the recognition result of the other vehicles and the detection result of the direction of the driver's face or line of sight, and reducing the level of the automatic driving when the detected direction is outside the permissible range; and determining the permissible range based on the direction detected after the output unit outputs the prescribed information.

[0017] Effects of the Invention

[0018] According to any of the above aspects, the accuracy of determining whether the driver is monitoring the front can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a configuration diagram of a vehicle system 1 using the vehicle control device according to the embodiment.

[0020] Figure 2 1 is a diagram showing an example of the BSI indicator 90 .

[0021] Figure 3 FIG. 1 is a diagram showing another example of the BSI indicator 90 .

[0022] Figure 4 1 is a functional configuration diagram of the first control unit 120 and the second control unit 160 .

[0023] Figure 5 1 is a diagram showing an example of the correspondence between the driving mode and the control state and task of the host vehicle M.

[0024] Figure 6 This is a flowchart showing an example of a flow of a series of processes performed by the automatic driving control device 100 according to the embodiment.

[0025] Figure 7 This is a diagram for explaining a method for determining the allowable range θ.

[0026] Figure 8 This is a diagram for explaining a method for determining the allowable range θ.

[0027] Figure 9 This is a diagram showing an example of the hardware configuration of the automatic driving control device 100 according to the embodiment.

[0028] Description of reference numerals:

[0029] 1…Vehicle system, 10…Camera, 12…Radar device, 14…LIDAR, 16…Object recognition device, 20…Communication device, 30…HMI, 40…Vehicle sensor, 50…Navigation device, 60…MPU, 70…Driver monitoring camera, 80…Driving operation element, 82…Steering wheel, 84…Steering wheel grip sensor, 90…BSI indicator, 100…Automatic driving control device, 120…First control unit, 130…Recognition unit, 140…Action plan generation unit, 150…Mode control unit, 152…Driver state determination unit, 154…Mode determination unit, 156…Device control unit, 158…Range determination unit, 160…Second control unit, 162…Acquisition unit, 164…Speed ​​control unit, 166…Steering control unit, 180…Storage unit, M…Own vehicle DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of a vehicle control device, a vehicle control method, and a storage medium according to the present invention will be described with reference to the accompanying drawings.

[0031] [Overall structure]

[0032] Figure 1 This is a structural diagram of a vehicle system 1 utilizing a vehicle control device according to an embodiment. The vehicle equipped with the vehicle system 1 (hereinafter referred to as the host vehicle M) is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its driving source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator coupled to the internal combustion engine or power discharged from a secondary battery or fuel cell.

[0033] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driver monitoring camera 70, a driving operating element 80, a BSI (Blind Spot Information) indicator 90, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. It should be noted that Figure 1The structure shown is only an example, and a part of the structure may be omitted or other structures may be added. The automatic driving control device 100 is an example of a "vehicle control device."

[0034] The camera 10 is, for example, a digital camera that utilizes a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is installed at any part of the vehicle equipped with the vehicle system 1. For example, when photographing the front of the vehicle M, the camera 10 is installed at the upper portion of the front windshield, the back of the rearview mirror inside the vehicle, etc. When photographing the rear of the vehicle M, the camera 10 is installed at the upper portion of the rear windshield, etc. When photographing the right or left side of the vehicle M, the camera 10 is installed at the right side or left side of the rearview mirror on the vehicle body or the door, etc. The camera 10 periodically and repeatedly photographs the surroundings of the vehicle M. The camera 10 can also be a stereo camera.

[0035] The radar device 12 transmits radio waves, such as millimeter waves, toward the periphery of the host vehicle M and detects the radio waves (reflected waves) reflected by objects to detect at least the object's position (range and direction). The radar device 12 is mounted anywhere on the host vehicle M. The radar device 12 can also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0036] LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to light) around the vehicle M and measures the scattered light. LIDAR 14 detects the distance to an object based on the time from light emission to light reception. The irradiated light is, for example, a pulsed laser. LIDAR 14 is mounted anywhere on the vehicle M.

[0037] The object recognition device 16 performs sensor fusion processing on some or all of the detection results from the camera 10, radar device 12, and LIDAR 14 to identify the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the automatic driving control device 100. The object recognition device 16 can directly output the detection results from the camera 10, radar device 12, and LIDAR 14 to the automatic driving control device 100. The object recognition device 16 can also be omitted from the vehicle system 1.

[0038] The communication device 20 communicates with other vehicles around the host vehicle M using, for example, a cellular network, Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or communicates with various server devices via a wireless base station.

[0039] The HMI 30 presents various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. For example, the HMI 30 may include a display device, switches, speakers, a buzzer, a touch panel, etc. For example, a occupant inputs the destination of the vehicle M into the HMI 30. The HMI 30 is an example of an "output unit."

[0040] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a gyro sensor for detecting angular velocity, and an orientation sensor for detecting the orientation of the vehicle M. The gyro sensor may include, for example, a yaw rate sensor for detecting angular velocity about a vertical axis.

[0041] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or a flash memory.

[0042] The GNSS receiver 51 receives radio waves from each of a plurality of GNSS satellites (artificial satellites) and determines the position of the host vehicle M based on the received radio wave signals. The GNSS receiver 51 outputs the determined position of the host vehicle M to the path determination unit 53, or outputs the determined position of the host vehicle M directly to the automatic driving control device 100 or indirectly to the automatic driving control device 100 via the MPU 60. The position of the host vehicle M may also be determined or supplemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40.

[0043] The navigation HMI 52 includes a display device, a speaker, a touch panel, buttons, and the like. The navigation HMI 52 may partially or entirely share the aforementioned HMI 30. For example, instead of or in addition to inputting the destination of the vehicle M into the HMI 30, the occupant may input the destination of the vehicle M into the navigation HMI 52.

[0044] The route determination unit 53 refers to the first map information 54 to determine a route (hereinafter referred to as a route on a map) from the position of the vehicle M determined by the GNSS receiver 51 (or an input arbitrary position) to the destination input by the occupant using the HM 30 or the navigation HMI 52 .

[0045] The first map information 54 is information representing the shape of a road by, for example, representing road links and nodes connected by the links. The first map information 54 may also include road curvature, POI (Point of Interest) information, etc. The route on the map is output to the MPU 60 .

[0046] The navigation device 50 can also provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 can also be implemented as a function of a terminal device such as a smartphone or tablet computer held by the passenger. The navigation device 50 can also transmit the current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0047] The MPU 60 includes, for example, a recommended lane determination unit 61, which stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 is implemented by having a hardware processor such as a CPU (Central Processing Unit) execute a program (software). Furthermore, the recommended lane determination unit 61 can be implemented using hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or through a combination of software and hardware. The program can be pre-stored in the MPU 60's storage device (a storage device with a non-transitory storage medium) or stored in a removable storage medium such as a DVD or CD-ROM. The program can be installed in the MPU 60's storage device by attaching the storage medium (non-transitory storage medium) to a drive device.

[0048] The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (e.g., every 100 meters in the vehicle's travel direction) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines the lane from the left to be driven. If the route on the map has a branch, the recommended lane determination unit 61 determines a recommended lane so that the host vehicle M can travel on a reasonable route to the branch destination.

[0049] The second map information 62 is more accurate than the first map information 54. The second map information 62 includes, for example, information about lane centers and lane boundaries. Furthermore, the second map information 62 may include road information, traffic restriction information, address information (address, postal code), facility information, and telephone number information. The second map information 62 can be updated at any time by communicating with other devices via the communication device 20.

[0050] The driver monitoring camera 70 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS. The driver monitoring camera 70 is mounted at any location within the vehicle M in a position and orientation such that it can capture a frontal image of the occupant (i.e., the driver) seated in the driver's seat of the vehicle M. For example, the driver monitoring camera 70 is mounted on the dashboard of the vehicle M.

[0051] Driving control elements 80 include, for example, a steering wheel 82, an accelerator pedal, a brake pedal, a shift lever, and other operating elements. Sensors are mounted on driving control elements 80 to detect the amount of operation or the presence or absence of an operation. The sensor's detection results are output to the automatic driving control device 100 or to some or all of the driving force output device 200, the braking device 210, and the steering device 220.

[0052] The steering wheel 82 need not necessarily be annular; it may also be in the form of a special-shaped steering wheel, a joystick, or buttons. A steering wheel grip sensor 84 is mounted on the steering wheel 82. The steering wheel grip sensor 84 is, for example, a capacitance sensor. The steering wheel grip sensor 84 detects whether the driver is gripping the steering wheel 82 (i.e., contacting the steering wheel while applying force) and outputs a signal indicating the detection result to the automatic driving control device 100.

[0053] The BSI indicator 90 is a driving support device that illuminates or flashes to notify the driver that the host vehicle M is currently approaching a surrounding vehicle or will approach a surrounding vehicle at a certain time in the future. By illuminating or flashing the BSI indicator 90, the driver is urged to visually check for vehicles approaching the host vehicle M. The BSI indicator 90 is an example of an "output unit."

[0054] Figure 2 and Figure 3 1 is a diagram showing an example of a BSI indicator 90. The BSI indicator 90 can be Figure 2 As in the example of the rearview mirror embedded in the door, the mirror surface can also be Figure 3 Hereinafter, as an example, the structure in which the BSI indicator 90 is embedded in the mirror surface of the door upper rearview mirror will be described.

[0055] The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, and a storage unit 180. The first control unit 120 and the second control unit 160 are respectively implemented by, for example, causing a hardware processor such as a CPU to execute a program (software). In addition, some or all of these components can be implemented by hardware (including a circuit unit: circuitry) such as an LSI, an ASIC, an FPGA, or a GPU, or can be implemented by the cooperation of software and hardware. The program can be pre-stored in a storage device such as an HDD or a flash memory of the automatic driving control device 100 (a storage device having a non-temporary storage medium), or can be stored in a removable storage medium such as a DVD or a CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by assembling the storage medium (non-temporary storage medium) in a drive device.

[0056] The storage unit 180 is realized by, for example, a HDD, a flash memory, an EEPROM, or a ROM (or a RAM, etc.) The storage unit 180 stores, for example, programs read and executed by the processor.

[0057] Figure 4 This is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130, an action plan generation unit 140, and a mode control unit 150. The combination of the action plan generation unit 140 and the second control unit 160, or the combination of the action plan generation unit 140, the mode control unit 150, and the second control unit 160, is an example of a "driving control unit."

[0058] The first control unit 120, for example, concurrently implements functions based on AI (artificial intelligence) and pre-defined models. For example, the "intersection recognition" function can be implemented by concurrently executing intersection recognition based on deep learning and other methods and recognition based on pre-defined conditions (such as the presence of signals or road signs that can be pattern-matched). Scoring and comprehensively evaluating both functions allows for the "intersection recognition" function. This ensures the reliability of autonomous driving.

[0059] The recognition unit 130 identifies the surrounding conditions or environment of the host vehicle M. For example, the recognition unit 130 identifies objects around the host vehicle M based on information input from the camera 10, radar device 12, and LIDAR 14 via the object recognition device 16. Examples of objects recognized by the recognition unit 130 include bicycles, motorcycles, four-wheeled vehicles, pedestrians, road markings, road signs, dividing lines, utility poles, guardrails, and fallen objects. Furthermore, the recognition unit 130 identifies the object's state, including its position, velocity, and acceleration. The object's position is, for example, identified as a relative coordinate position (i.e., relative position relative to the host vehicle M) with a representative point (such as the center of gravity or the center of the drive shaft) of the host vehicle M as the origin, and is used for control. The object's position can be represented by a representative point, such as the object's center of gravity or corner, or by a represented area. The object's "state" can also include its acceleration, jerk, or "behavior" (e.g., whether it is currently changing lanes or intending to change lanes).

[0060] Furthermore, the recognition unit 130 recognizes, for example, the lane in which the host vehicle M is traveling (hereinafter referred to as the host lane), adjacent lanes adjacent to the host lane, and the like. For example, the recognition unit 130 obtains the second map information 62 from the MPU 60 and compares the pattern of road dividing lines (e.g., the arrangement of solid and dashed lines) included in the obtained second map information 62 with the pattern of road dividing lines around the host vehicle M recognized from the image of the camera 10, thereby recognizing the space between the dividing lines as the host lane and the adjacent lane.

[0061] The recognition unit 130 is not limited to road dividing lines. It can also recognize lanes such as the vehicle's own lane and adjacent lanes by recognizing road dividing lines, roadside boundaries (road boundaries) including shoulders, curbs, medians, and guardrails. This recognition can also take into account the position of the vehicle M obtained from the navigation device 50 and the processing results of the INS. Furthermore, the recognition unit 130 can recognize temporary stop signs, obstacles, red lights, toll booths, and other road features.

[0062] Furthermore, when recognizing the host lane, the recognition unit 130 recognizes the relative position and posture of the host vehicle M relative to the host lane. For example, the recognition unit 130 may recognize the deviation of the host vehicle M's reference point from the lane center and the angle formed by the host vehicle M's travel direction relative to a line connecting the coordinate points in the lane center as the relative position and posture of the host vehicle M relative to the host lane. Alternatively, the recognition unit 130 may recognize the position of the host vehicle M's reference point relative to either end of the host lane (road dividing line or road boundary) as the relative position of the host vehicle M relative to the host lane.

[0063] The action plan generation unit 140 generates a future target trajectory for the vehicle M to automatically (without relying on the driver's operation) travel in the recommended lane determined by the recommended lane determination unit 61 in principle and in a manner that can respond to the surrounding conditions of the vehicle M in the driving state specified by the events described later.

[0064] The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a trajectory in which the locations (track points) that the vehicle M should arrive at are arranged in sequence. Track points are locations that the vehicle M should arrive at at each specified driving distance (e.g., a few meters) along the way. Separately, target speeds and target accelerations at each specified sampling time (e.g., a few tenths of a second) are generated as part of the target trajectory. Alternatively, a track point can be the location that the vehicle M should arrive at at the sampling moment of each specified sampling time. In this case, the target speed and target acceleration information are expressed at intervals between track points.

[0065] In order to cope with the surrounding conditions of the vehicle M, the action plan generation unit 140 may exceptionally generate a target trajectory such that the vehicle M travels in a lane other than the recommended lane (for example, a lane adjacent to the recommended lane). That is, the priority of lanes other than the recommended lane is relatively lower than the priority of the recommended lane. For example, the recommended lane has the highest priority (priority 1), the other lanes adjacent to the recommended lane (hereinafter referred to as adjacent lanes) have the second highest priority (priority 2), and the other lanes adjacent to the adjacent lane have the third highest priority (priority 3). In this way, the action plan generation unit 140 generates a target trajectory such that the vehicle M travels in the recommended lane with the highest priority in principle, and exceptionally generates a target trajectory such that the vehicle M travels in a lane with a lower priority than the recommended lane according to the surrounding conditions of the vehicle M.

[0066] When generating the target trajectory, the action plan generation unit 140 determines an event for autonomous driving (including some driving support) along the route for which the recommended lane is determined. An autonomous driving event is information that specifies the behavior that the vehicle M should take during autonomous driving (partial driving support), that is, the state (or form) of the vehicle during driving.

[0067] Automatic driving events include, for example, constant speed driving events, low-speed following driving events, lane change events, and overtaking events. A constant speed driving event is an event in which the vehicle M is driven in the same lane at a constant speed. A low-speed following driving event is an event in which the vehicle M is driven to follow another vehicle (hereinafter referred to as the preceding vehicle) that is within a specified distance (e.g., within 100 [m]) in front of the vehicle M and is closest to the vehicle M. "Following" can be, for example, a driving state in which the relative distance (vehicle distance) between the vehicle M and the preceding vehicle is maintained constant, or a driving state in which the vehicle M is driven in the center of the lane in addition to maintaining the relative distance between the vehicle M and the preceding vehicle constant. A lane change event is an event in which the vehicle M is driven to change lanes from the lane to the adjacent lane. An overtaking event is an event in which the vehicle M is driven to change lanes to the adjacent lane temporarily and then overtakes the preceding vehicle in the adjacent lane, and then is driven to change lanes again to the original lane.

[0068] Furthermore, autonomous driving events include diverging events, merging events, lane reduction events, and takeover events. A diverging event is an event in which, when the vehicle M is traveling on a main road and its destination is on an extension of a branch road (hereinafter referred to as a branch lane) that branches off from the main road, the vehicle M is guided to change lanes from the main road to the branch lane at the diverging point. A merging event is an event in which, when the vehicle M is traveling on a branch road (hereinafter referred to as a merging lane) that merges with the main road and its destination is on an extension of the main road, the vehicle M is guided to change lanes from the merging lane to the main road at the merging point. A lane reduction event is an event in which, when traveling on a route where the number of lanes decreases, the vehicle M is guided to change lanes to another lane. A takeover event is an event in which the autonomous driving mode (Mode A, described below) is terminated and switched to the driving support mode (Modes B, C, and D, described below) or the manual driving mode (Mode E, described below). For example, there are cases where the dividing line is interrupted before a toll booth on a highway, making it impossible to identify the relative position of the vehicle M. In such a case, a takeover event is determined (planned) with respect to the section immediately before the toll booth.

[0069] The action plan generating unit 140 sequentially determines the aforementioned plurality of events on the route to the destination, and generates a target trajectory for causing the vehicle M to travel in a state defined by each event while taking into account the surrounding conditions of the vehicle M.

[0070] The mode control unit 150 determines the driving mode of the vehicle M to be one of a plurality of driving modes. Each of the plurality of driving modes assigns different tasks to the driver. The mode control unit 150 includes, for example, a driver state determination unit 152, a mode determination unit 154, a device control unit 156, and a range determination unit 158. Their respective functions will be described below. The combination of the driver monitoring camera 70 and the driver state determination unit 152 is an example of a "detection unit." The device control unit 156 is an example of an "output control unit."

[0071] Figure 5 : This is a diagram showing an example of the correspondence between the driving mode and the control state and task of the vehicle M. The driving mode of the vehicle M includes five modes, namely mode A to mode E. As for the control state, that is, the degree of automation (control level) of the driving control of the vehicle M, mode A is the highest, followed by mode B, mode C, and mode D in order, and mode E is the lowest. On the contrary, for the tasks assigned to the driver, mode A is the lightest, followed by mode B, mode C, and mode D in order, and mode E is the heaviest. It should be noted that since the control state becomes a non-automatic driving state in modes D and E, the automatic driving control device 100 performs its duties before terminating the control involved in the automatic driving and transferring to driving assistance or manual driving. The following is an example of the content of each driving mode.

[0072] In mode A, the vehicle is in an automatic driving state, and the driver is not required to monitor the front or hold the steering wheel 82 (steering wheel holding in the figure). However, even in mode A, the driver is required to have a body posture that can quickly switch to manual driving according to the requirements from the system centered on the automatic driving control device 100. It should be noted that the automatic driving mentioned here refers to controlling any one of steering and acceleration and deceleration without relying on the driver's operation. The front refers to the space in the direction of travel of the vehicle M that is visually identified through the front windshield. Mode A is a driving mode that can be executed when the following conditions are met, such as when the vehicle M is traveling at a speed below a specified speed (for example, about 50 [km / h]) on a motor vehicle-only road such as an expressway and there is a preceding vehicle to be followed. It is sometimes also called TJP (Traffic Jam Pilot). If this condition is not met, the mode control unit 150 changes the driving mode of the vehicle M to mode B.

[0073] In mode B, the state is set to driving support, and the driver is assigned the task of monitoring the front of the vehicle M (hereinafter referred to as front monitoring), but is not assigned the task of holding the steering wheel 82. In mode C, the state is set to driving support, and the driver is assigned the task of monitoring the front and the task of holding the steering wheel 82. Mode D is a driving mode in which a certain degree of driver's driving operation is required for at least one of the steering and acceleration and deceleration of the vehicle M. For example, in mode D, driving support such as ACC (Adaptive Cruise Control) and LKAS (Lane Keeping Assist System) is performed. In mode E, the state is set to manual driving in which the driver's driving operation is required for both steering and acceleration and deceleration. Of course, both modes D and E assign the driver the task of monitoring the front of the vehicle M.

[0074] The automatic driving control device 100 (and the driving support device (not shown)) executes automatic lane changes according to the driving mode. Automatic lane changes include automatic lane changes based on system requirements (1) and automatic lane changes based on driver requirements (2). Automatic lane changes (1) include automatic lane changes for overtaking when the speed of the preceding vehicle is greater than or equal to a certain threshold compared to the speed of the vehicle itself, and automatic lane changes for traveling toward the destination (automatic lane changes performed by changing the recommended lane). In automatic lane changes (2), when conditions related to speed, positional relationship with surrounding vehicles, etc. are met, when the driver operates the direction indicator, the vehicle M is caused to change lanes in the direction of operation.

[0075] The automatic driving control device 100 does not execute either automatic lane change (1) or (2) in mode A. The automatic driving control device 100 executes either automatic lane change (1) or (2) in modes B and C. The driving support device (not shown) does not execute automatic lane change (1) but executes automatic lane change (2) in mode D. In mode E, neither automatic lane change (1) or (2) is executed.

[0076] Return to Figure 4 If the driver does not perform the task related to the determined driving mode, the mode control unit 150 changes the driving mode of the host vehicle M to a driving mode with a heavier task.

[0077] For example, in mode A, the following control is performed: when the driver is in a body posture that is unable to switch to manual driving in response to a request from the system (for example, when the driver continues to look around outside the allowable range θ described later, or when a sign of driving difficulty is detected), the mode control unit 150 uses the HMI30 to urge the driver to switch to manual driving. If the driver does not respond, the vehicle M is brought close to the shoulder of the road and gradually stopped, and the automatic driving is stopped. After the automatic driving is stopped, the vehicle enters the state of mode D or E, and the vehicle M can be started by manual operation of the driver. The same applies to "stopping automatic driving" below. In mode B, the following control is performed: when the driver is not monitoring the front, the mode control unit 150 uses the HMI30 to urge the driver to monitor the front. If the driver responds, the vehicle M is brought close to the shoulder of the road and gradually stopped, and the automatic driving is stopped. In mode C, the following control is performed: when the driver is not monitoring the front or is not holding the steering wheel 82, the mode control unit 150 uses HMI30 to urge the driver to monitor the front and / or hold the steering wheel 82. If the driver does not respond, the vehicle M is brought close to the shoulder of the road and gradually stops, and the automatic driving is stopped.

[0078] In order to perform the above-mentioned mode change, the driver state determination unit 152 determines whether the driver is in a state capable of performing the task based on the image of the driver monitoring camera 70 and the detection signal of the steering wheel grip sensor 84 .

[0079] For example, the driver state determination unit 152 analyzes the image of the driver monitoring camera 70 to estimate the driver's posture, and determines based on the estimated posture whether the driver is in a body posture that allows transition to manual driving in response to a request from the system.

[0080] The driver state determination unit 152 analyzes the image of the driver monitoring camera 70 to estimate the driver's line of sight or facial orientation, and determines whether the driver is monitoring the front of the host vehicle M based on the estimated line of sight or facial orientation.

[0081] For example, the driver status determination unit 152 uses methods such as template matching to detect the positional relationship between the driver's head and eyes, the combination of the eye reference point and the moving point, etc. based on the image from the driver monitoring camera 70. Furthermore, the driver status determination unit 152 estimates the orientation of the face based on the relative position of the eyes relative to the head. Furthermore, the driver status determination unit 152 estimates the orientation of the driver's line of sight based on the position of the moving point relative to the reference point. For example, if the reference point is the corner of the eye, the moving point is the iris. Alternatively, if the reference point is the corneal reflection area, the moving point is the pupil.

[0082] For example, the mode determination unit 154 may determine that the driver is not monitoring the front of the host vehicle M when the driver's face or line of sight is directed outside the permissible range θ for a predetermined time or longer.

[0083] The driver state determination unit 152 determines whether the driver is gripping the steering wheel 82 based on the detection signal of the steering wheel grip sensor 84 .

[0084] The mode determination unit 154 determines the driving mode of the host vehicle M based on the determination result of the driver state determination unit 152 .

[0085] The device control unit 156 controls the HMI 30, the BSI indicator 90, and the like based on the driving mode of the host vehicle M determined by the mode determination unit 154, the determination result of the driver state determination unit 152, and the recognition result of the recognition unit 130. For example, the device control unit 156 causes the HMI 30 to output information urging the driver to perform tasks corresponding to each driving mode. Furthermore, if the recognition unit 130 recognizes that a surrounding vehicle is in an adjacent lane and is positioned behind the host vehicle M, the device control unit 156 causes the BSI indicator 90 to illuminate or flash.

[0086] Range determination unit 158 ​​determines the permissible range θ based on the driver's line of sight or facial orientation estimated by driver state determination unit 152. Permissible range θ refers to the angular range of the driver's line of sight or facial orientation within which it can be considered that the driver is fulfilling their forward monitoring obligations (performing their forward monitoring tasks). Permissible range θ may be an angular range related to the horizontal direction or an angular range related to both the horizontal and vertical directions.

[0087] The second control unit 160 controls the driving force output device 200 , the braking device 210 , and the steering device 220 so that the host vehicle M passes through the target trajectory generated by the action plan generation unit 140 at a predetermined timing.

[0088] The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires the information of the target track (track point) generated by the action plan generation unit 140 and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed element attached to the target track stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target track stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is implemented, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 combines and executes feedforward control corresponding to the curvature of the road in front of the vehicle M and feedback control based on deviation from the target track.

[0089] The driving force output device 200 outputs driving force (torque) to the drive wheels for driving the vehicle. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls these components. The ECU controls the above components based on information input from the second control unit 160 or from the driving operating element 80.

[0090] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor based on information input from the second control unit 160 or from the driver's operating element 80, and outputs a braking torque corresponding to the braking operation to each wheel. The braking device 210 may include a mechanism that transmits the hydraulic pressure generated by operating the brake pedal included in the driver's operating element 80 to the hydraulic cylinder via a master hydraulic cylinder as a backup. It should be noted that the braking device 210 is not limited to the structure described above and may also be an electronically controlled hydraulic braking device that controls the actuator based on information input from the second control unit 160 and transmits the hydraulic pressure of the master hydraulic cylinder to the hydraulic cylinder.

[0091] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to, for example, a rack-and-pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor based on information input from the second control unit 160 or information input from the driving operating element 80 to change the direction of the steering wheel.

[0092] [Overall processing flow]

[0093] Hereinafter, the flow of a series of processes performed by the automatic driving control device 100 according to the embodiment will be described using a flowchart. Figure 6 This is a flowchart showing an example of a series of processes performed by the automatic driving control device 100 according to the embodiment. The processes in this flowchart can be repeatedly executed at a predetermined cycle. In addition, the processes in this flowchart can be executed every time the driver changes.

[0094] First, the device control unit 156 determines whether the recognition unit 130 recognizes that a surrounding vehicle is present in an adjacent lane and that the surrounding vehicle is located behind the host vehicle M (step S100 ).

[0095] When the recognition unit 130 recognizes that a surrounding vehicle is present in an adjacent lane and that the surrounding vehicle is located behind the host vehicle M, the device control unit 156 lights or flashes the BSI indicator 90 (step S102 ).

[0096] For example, when a surrounding vehicle is in an adjacent lane to the left of the vehicle M and is located behind the vehicle M, the device control unit 156 causes the BSI indicator 90 on the rearview mirror on the right door to light up or flash when viewed from the driver.

[0097] Next, the range determination unit 158 ​​determines whether the driver's face or line of sight is directed toward the BSI indicator 90 based on the direction of the driver's line of sight or face estimated by the driver state determination unit 152 (step S104 ).

[0098] When the driver directs his face or his line of sight toward the BSI indicator 90 , the range determination unit 158 ​​stores the direction of the driver's face or line of sight in the storage unit 180 (step S106 ).

[0099] On the other hand, if the driver does not direct his face or line of sight toward the BSI indicator 90 , the range determination unit 158 ​​skips S106 and advances the process to S108 .

[0100] Next, the range determination unit 158 ​​determines whether the process of storing the orientation of the face or the line of sight in the storage unit 180 (ie, the process of S106 ) has been repeated a predetermined number of times (eg, 10 times) (step S108 ).

[0101] For example, the range determination unit 158 ​​determines whether the process of S106 has been repeated a predetermined number of times for each of the right and left BSI indicators 90 .

[0102] If the process of S106 has not been repeated a predetermined number of times for each of the right and left BSI indicators 90 , the range determination unit 158 ​​returns the process to S100 and repeats the series of processes from S100 to S106 .

[0103] On the other hand, when the process of S106 is repeated a predetermined number of times for each of the right and left BSI indicators 90, the range determination unit 158 ​​determines the permissible range θ based on the orientation of the face or line of sight for the predetermined number of times stored in the storage unit 180 (step S110). The process of this flowchart is thus terminated.

[0104] Figure 7 and Figure 8 is a diagram for explaining the method of determining the allowable range θ. Figure 7As shown, the permissible range θ is typically determined to include the windshield as the target space for forward monitoring. For example, if the driver's line of sight or facial orientation does not fall within the permissible range θ, the driver state determination unit 152 determines that the driver is not monitoring the forward direction of the host vehicle M. If the driver's line of sight or facial orientation falls within the permissible range θ, the driver determines that the driver is monitoring the forward direction of the host vehicle M.

[0105] As described above, when the driver is not monitoring the road ahead, the mode determination unit 154 changes the driving mode of the host vehicle M to a driving mode with a lower degree of automation (control level). Figure 7 In the example shown in FIG, the driver directs his or her gaze or face toward the right door mirror where the BSI indicator 90 is illuminated. In such a case, when a uniform permissible range θ is used, the driver's gaze or face may be directed outside the permissible range θ, and the driver may be judged to be neglecting the task of monitoring the front (i.e., looking around), and the driving mode may be switched. The BSI indicator 90 notifies the driver of surrounding vehicles in adjacent lanes approaching from behind the host vehicle M, and from the perspective of calling attention, it is common with the task of monitoring the front required in a specific driving mode. Therefore, it is preferable that directing the gaze or face toward the BSI indicator 90 is not treated as looking around, but rather as part of the task of monitoring the front.

[0106] In light of this, in the present embodiment, when the driver directs his or her line of sight or face toward the BSI indicator 90 after the BSI indicator 90 is illuminated, the range determination unit 158 ​​determines the range extending to the direction of the driver's line of sight or face as the new permissible range θ′. Figure 7 In the example shown in FIG1 , since the driver's gaze or face is directed toward the right door mirror where the BSI indicator 90 is illuminated, the range determination unit 158 ​​expands the permissible range θ to include the right door mirror and sets this as the new permissible range θ′. If the BSI indicator 90 in the left door mirror is illuminated and the driver's gaze or face is directed toward the left BSI indicator 90, the range determination unit 158 ​​similarly expands the permissible range θ to include the right door mirror and sets this as the new permissible range θ′.

[0107] In this case, when expanding the permissible range θ, the range determination unit 158 ​​may determine the extent to which the permissible range θ is expanded based on the number of times (or the duration) the driver directs his or her gaze or face toward the BSI indicator 90. For example, the range determination unit 158 ​​may further expand the permissible range θ if the driver directs his or her gaze or face toward the BSI indicator 90 more times (or for a longer duration).

[0108] By expanding the permissible range θ in this manner, it is easier to determine that the driver is performing the forward monitoring task even if the driver turns his face or eyes toward the BSI indicator 90. As a result, frequent switching of driving modes can be suppressed, enabling more comfortable autonomous driving.

[0109] According to the embodiment described above, the automatic driving control device 100 identifies surrounding vehicles around the host vehicle M and estimates the direction of the driver's face or line of sight based on the image from the driver monitoring camera 70. Based on the position of surrounding vehicles in adjacent lanes, the BSI indicator 90 is illuminated or flashed. If the direction of the driver's face or line of sight is outside the permissible range θ, the level of automatic driving is lowered, and the permissible range θ is expanded based on the direction of the driver's face or line of sight after the BSI indicator 90 is illuminated or flashed. This improves the accuracy of determining whether the driver is monitoring the front. As a result, frequent switching of driving modes can be suppressed, enabling more comfortable automatic driving.

[0110] [Hardware structure]

[0111] Figure 9 : This is a diagram showing an example of the hardware structure of the automatic driving control device 100 of the embodiment. As shown in the figure, the automatic driving control device 100 is a structure in which a communication controller 100-1, a CPU 100-2, a RAM 100-3 used as a working memory, a ROM 100-4 storing a boot program, etc., a storage device 100-5 such as a flash memory and an HDD, a drive device 100-6, etc. are interconnected via an internal bus or a dedicated communication line. The communication controller 100-1 communicates with components other than the automatic driving control device 100. A program 100-5a for execution by the CPU 100-2 is stored in the storage device 100-5. The program is expanded to the RAM 100-3 via a DMA (Direct Memory Access) controller (not shown) and is executed by the CPU 100-2. In this way, part or all of the first control unit and the second control unit 160 are realized.

[0112] The above-described embodiment can be expressed as follows.

[0113] A vehicle control device, comprising:

[0114] a memory storing a program; and

[0115] processor,

[0116] The processor executes the program to perform the following processing:

[0117] Identify other vehicles around the vehicle;

[0118] detecting the direction of the face or sight of the driver of the host vehicle;

[0119] causing an output unit to output predetermined information indicating the presence of another vehicle in a second lane adjacent to the first lane in which the host vehicle is located, based on a position of the other vehicle among the identified other vehicles;

[0120] performing autonomous driving to control at least one of the speed and steering of the own vehicle based on the recognition result of the other vehicle and the detection result of the orientation of the driver's face or gaze, and lowering the level of the autonomous driving if the detected orientation is outside an allowable range; and

[0121] The permissible range is determined based on the direction detected after the output unit outputs the predetermined information.

[0122] While specific embodiments of the present invention have been described above, the present invention is not limited to these embodiments at all, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A vehicle control device, wherein: The vehicle control device comprises: a recognition unit that recognizes other vehicles existing around the host vehicle; a detection unit that detects the direction of the face or sight of the driver of the host vehicle; an output control unit that causes an output unit to output predetermined information indicating the presence of another vehicle in a second lane adjacent to the first lane in which the host vehicle is located, based on a position of the other vehicle identified by the identification unit and located in the second lane; a driving control unit that performs automatic driving to control at least one of the speed and the steering of the host vehicle based on the recognition result of the recognition unit and the detection result of the detection unit, and lowers a level of the automatic driving if the direction detected by the detection unit is outside an allowable range; as well as a determination unit that determines the allowable range based on the orientation detected by the detection unit after the output unit outputs the prescribed information, The output unit is an indicator provided on a mirror surface of a door rearview mirror of the host vehicle or near the door rearview mirror in the cabin of the host vehicle. The output control unit lights or flashes the indicator when the recognition unit recognizes that the other vehicle is located behind the host vehicle in the second lane. The determination unit determines the allowable range to include the indicator when the detection unit detects that the driver has directed his face or eyesight toward the indicator after the indicator has been lit or flashed.

2. The vehicle control device according to claim 1, wherein: The determination unit determines the permissible range when the number of times the driver directs his face or gaze toward the indicator reaches a predetermined number.

3. The vehicle control device according to claim 1 or 2, wherein: The driving control unit lowers the level of the automatic driving when the driver keeps the face or the sight directed outside the permissible range for a predetermined time or longer.

4. A vehicle control method, wherein: The vehicle control method causes a computer mounted on the vehicle to execute the following processing: identifying other vehicles existing around the host vehicle; detecting the direction of the face or sight of the driver of the host vehicle; causing an output unit to output predetermined information indicating the presence of another vehicle in a second lane adjacent to the first lane in which the host vehicle is located, based on a position of the other vehicle among the identified other vehicles; performing autonomous driving to control at least one of the speed and steering of the host vehicle based on the recognition result of the other vehicle and the detection result of the orientation of the driver's face or gaze, and lowering the level of the autonomous driving if the detected orientation is outside an allowable range; as well as determining the permissible range based on the direction detected after the output unit outputs the prescribed information, The output unit is an indicator provided on a mirror surface of a door rearview mirror of the host vehicle or near the door rearview mirror in the cabin of the host vehicle. When it is recognized that the other vehicle is located behind the host vehicle in the second lane, the indicator is illuminated or flashed. When it is detected that the driver has directed his face or eyesight toward the indicator after the indicator has been lit or flashed, the allowable range is determined to include the indicator.

5. A storage medium storing a program, wherein: The program is used to cause the computer mounted on the vehicle to execute the following processing: identifying other vehicles existing around the host vehicle; detecting the direction of the face or sight of the driver of the host vehicle; causing an output unit to output predetermined information indicating the presence of another vehicle in a second lane adjacent to the first lane in which the host vehicle is located, based on a position of the other vehicle among the identified other vehicles; performing autonomous driving to control at least one of the speed and steering of the host vehicle based on the recognition result of the other vehicle and the detection result of the orientation of the driver's face or gaze, and lowering the level of the autonomous driving if the detected orientation is outside an allowable range; as well as determining the permissible range based on the direction detected after the output unit outputs the prescribed information, The output unit is an indicator provided on a mirror surface of a door rearview mirror of the host vehicle or near the door rearview mirror in the cabin of the host vehicle. When it is recognized that the other vehicle is located behind the host vehicle in the second lane, the indicator is illuminated or flashed. When it is detected that the driver has directed his face or eyesight toward the indicator after the indicator has been lit or flashed, the allowable range is determined to include the indicator.

Images