Information processing device, information processing method, program, and vehicle

Abstract

To secure safety when a visual line direction of a driver is inappropriate according to a driving state.SOLUTION: A visual line direction of a driver of a vehicle is detected, thereby to determine whether the detected visual line direction is a safe visual line direction. When it is determined that the visual line direction is not the safe visual line direction, the vehicle is controlled to be in a safe state. The present disclosure is applied to an on-vehicle system.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, a program, and a vehicle, and in particular to an information processing device, an information processing method, a program, and a vehicle that can ensure safety when the driver's line of sight is inappropriate for the driving condition. [Background technology]

[0002] Driving assistance technologies that detect surrounding information and assist the driver in driving a vehicle, and autonomous driving technologies that automatically control driving, are attracting attention.

[0003] Among these, a driving assistance technology that assists driving has been proposed, which detects the driver's line of sight and determines whether to accept driving operations by the driver based on whether changes in the line of sight direction are appropriate (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-100562 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, for example, when the vehicle is traveling straight, even if it is detected that the line of sight is not in the appropriate direction due to inattentive driving or drowsy driving, etc., in relation to the already accepted driving state of traveling straight, no driving control is performed.

[0006] The present disclosure has been made in consideration of such circumstances, and in particular, realizes driving control to ensure safety based on whether the line of sight direction is appropriate for the driving state. [Means for solving the problem]

[0007] An information processing device, program, and vehicle according to one aspect of the present disclosure include an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering, an acquisition unit that acquires detection results from an in-vehicle sensor for detecting the line of sight of the driver of the vehicle, an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on the detection results, and a feedback control unit that, if the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver urging them to change their line of sight to an appropriate direction; and, if the in-vehicle environment recognition unit determines that the driver's line of sight has not improved after the feedback control unit has provided the feedback, the operation control unit controls the operation of the vehicle to slow down the driving speed or change lanes to a safe lane.

[0008] An information processing method according to one aspect of the present disclosure includes an operation control process for controlling the operation of a vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering operations; an in-vehicle environment recognition process for determining whether the driver's line of sight is appropriate based on the detection results of an in-vehicle sensor for detecting the line of sight of the driver of the vehicle; and a feedback control process for, if the in-vehicle environment recognition process determines that the line of sight is inappropriate, feeding back information to the driver to urge the driver to change the line of sight to an appropriate direction; and if the in-vehicle environment recognition process determines that the driver's line of sight has not improved after the feedback control process, the operation control process controls the operation of the vehicle to reduce the driving speed or change lanes to a safe lane.

[0009] In one aspect of the present disclosure, the operation of a vehicle is controlled by coordinating driving operations related to acceleration and deceleration and driving operations related to steering, and a detection result from an in-vehicle sensor for detecting the line of sight of the driver of the vehicle is obtained, and based on the detection result, it is determined whether the driver's line of sight is appropriate. If it is determined that the line of sight is inappropriate, information is fed back to the driver to urge them to change their line of sight to an appropriate direction. If it is determined that the line of sight of the driver has not improved after the feedback has been provided, the operation of the vehicle is controlled so that the driving speed is reduced or the vehicle is changed to a safe lane. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 10 is a diagram illustrating a configuration for detecting a line-of-sight direction. [Figure 2] FIG. 10 is a diagram illustrating an example of safe driving control by deceleration when the line of sight direction is inappropriate in the case of autonomous driving level 1. [Figure 3] FIG. 10 is a diagram illustrating an example of safe driving control by lane changing when the line of sight direction is inappropriate in the case of autonomous driving level 2. [Figure 4] 1 is a block diagram illustrating a configuration example of a vehicle control system according to the present disclosure. [Figure 5] FIG. 5 is a block diagram illustrating an example of a configuration that realizes the safe driving control in FIG. 4. [Figure 6] 10 is a flowchart illustrating an automatic driving control process. [Figure 7] 10 is a flowchart illustrating a safe driving control process. [Figure 8] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] Hereinafter, embodiments of the present technology will be described in the following order. 1. Overview of this Disclosure 2. Configuration example of a vehicle control system that controls a vehicle according to the present disclosure 3. Configuration that realizes driving control to ensure safety according to line of sight 4.Autonomous driving control processing 5.Safe driving control processing 6. Software implementation example

[0013] <<1. Overview of this Disclosure>> <Detection of gaze direction> An overview of the present disclosure will be described.

[0014] The vehicle of the present disclosure detects the driver's line of sight and performs driving control to ensure safety based on whether the line of sight is appropriate.

[0015] The gaze direction is detected, for example, as shown in Figure 1, by using a camera Cam located above the steering wheel and facing the driver H to capture an image of an area V around the face of the driver H, and based on the position of the driver's eyes in the captured image.

[0016] It is desirable that the camera Cam be configured to use not only a so-called image sensor but also a ToF (Time of Flight) sensor.

[0017] More specifically, in the case of images obtained by an image sensor, if the entire face is exposed to strong light from the headlights of oncoming vehicles at night, the image of the entire face may become overexposed, making it impossible to recognize the various organs that make up the face.

[0018] In contrast, in the case of a distance image obtained by a ToF sensor, information on the distance from the camera Cam to each position on the surface of the face can be obtained, making it possible to recognize the unevenness of the surface of the face, and therefore each organ that makes up the face can be recognized based on the information on the unevenness of the surface of the face.

[0019] For these reasons, it is desirable that the camera Cam be configured to include both an image sensor and a ToF sensor, and be able to selectively use either the image sensor's image or the ToF sensor's distance image depending on the situation. Furthermore, combining the image sensor and the ToF sensor may improve robustness and shorten processing time.

[0020] <Driving control according to the level of autonomous driving> The vehicle disclosed herein is assumed to be driven by a driver and to employ autonomous driving technology of level 2 or lower, but can also be applied to vehicles with autonomous driving level 3 or higher.

[0021] Here, the autonomous driving level is a classification of the level of autonomous driving, and is classified into levels 0 to 5, for example.

[0022] Level 0 is an autonomous driving level in which the driver performs all driving operations, and is not essentially autonomous driving.

[0023] Level 1 is an autonomous driving level in which driving operations related to acceleration / deceleration and steering are controlled.

[0024] Level 2 is an autonomous driving level in which driving operations related to acceleration and deceleration and steering operations are controlled in coordination.

[0025] Level 3 is an automated driving level in which all driving operations are controlled in specific locations, such as on expressways. However, at Level 3, it is assumed that driving operations in emergencies will be performed by the driver.

[0026] Level 4 is an autonomous driving level in which all driving operations are controlled, including in emergencies, in specific locations such as highways.

[0027] Level 5 is a level of autonomous driving in which all driving operations are controlled in all situations, making it what is known as fully autonomous driving.

[0028] As mentioned above, technologies that realize automated driving levels 0 to 2 are generally referred to as driving assistance technologies, since they involve the driver primarily performing driving operations.

[0029] In contrast, technologies that achieve levels 3 to 5 of autonomous driving are generally referred to as autonomous driving technologies, as they eliminate the need for the driver to primarily perform driving operations in certain locations.

[0030] The driving control disclosed herein controls driving behavior to ensure safety for vehicles that achieve autonomous driving at levels 0 to 2, i.e., vehicles to which driving assistance technology is applied, based on whether the line of sight direction is appropriate for the driving conditions.

[0031] More specifically, when the autonomous driving level is 0, as described above, the vehicle is not actually autonomous and no control of driving behavior is performed, so the driver is given feedback (presentation) that their line of sight is inappropriate, and is encouraged to improve their line of sight.

[0032] Furthermore, in the case of autonomous driving level 1, for example, if the detected gaze direction is not directly ahead while driving straight ahead, it is suspected that the driver is looking away or drowsy at the wheel, and the gaze direction is therefore deemed to be inappropriate.

[0033] In such a case, the driver is provided with feedback (presentation) that the line-of-sight direction is inappropriate, and an operation is performed to prompt improvement of the line-of-sight direction. Further, when no improvement is observed, as shown in FIG. 2, the driving operation is controlled so as to decelerate by a predetermined speed.

[0034] By decelerating in this way, it is possible to delay the occurrence of accidents caused by distracted driving or drowsy driving, and to ensure safety by reducing the damage level in the event of an accident.

[0035] In addition, in FIG. 2, as shown in the left part, it is shown that the vehicle C1 is traveling straight ahead at a traveling speed V in the front. Further, in the right part of FIG. 2, in the traveling state of the vehicle C1, when the line-of-sight direction is considered inappropriate and no improvement is observed after prompting improvement of the line-of-sight direction, a vehicle C1' traveling at a decelerated traveling speed V' (<V) is represented.

[0036] Furthermore, when the automatic driving level is 2, when the line-of-sight direction is inappropriate and distracted driving or drowsy driving is suspected, and no improvement is observed after an operation is performed to prompt improvement of the line-of-sight direction, as shown in FIG. 3, the driving operation is controlled so as to change lanes to a safer driving lane.

[0037] By changing lanes to a lane safer than the currently traveling lane in this way, it is possible to delay the occurrence of accidents caused by distracted driving or drowsy driving, and to ensure safety by reducing the damage level in the event of an accident.

[0038] In addition, in the left part of FIG. 3, it is shown that the vehicle C2 is traveling in the overtaking lane L1, and in the right part, a vehicle C2' traveling in a state where it has changed lanes to the traveling lane L2, which is a safer lane, as a result of the line-of-sight being considered inappropriate, is represented.

[0039] <<2. Configuration example of vehicle control system for controlling vehicle of the present disclosure>> Next, the vehicle control system of the vehicle of the present disclosure will be described with reference to the block diagram of FIG.

[0040] FIG. 4 is a block diagram showing an example of a schematic functional configuration of a vehicle control system 100 of a vehicle 11 to which the present technology can be applied.

[0041] In the following description, when the vehicle 11 in which the vehicle control system 100 is installed is to be distinguished from other vehicles, it will be referred to as the host vehicle or the host vehicle.

[0042] The vehicle control system 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, in-vehicle devices 104, an output control unit 105, an output unit 106, a drivetrain control unit 107, a drivetrain system 108, a body system control unit 109, a body system 110, a memory unit 111, and an automatic driving control unit 112. The input unit 101, the data acquisition unit 102, the communication unit 103, the output control unit 105, the drivetrain control unit 107, the body system control unit 109, the memory unit 111, and the automatic driving control unit 112 are connected to each other via a communication network 121. The communication network 121 is formed of an in-vehicle communication network or bus conforming to any standard such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark). Note that the components of the vehicle control system 100 may be directly connected to each other without using the communication network 121.

[0043] In the following, when each unit of the vehicle control system 100 communicates via the communication network 121, the description of the communication network 121 will be omitted. For example, when the input unit 101 and the automatic driving control unit 112 communicate via the communication network 121, it will simply be described as the input unit 101 and the automatic driving control unit 112 communicating with each other.

[0044] The input unit 101 includes a device used by the passenger to input various data, instructions, etc. For example, the input unit 101 includes operation devices such as a touch panel, buttons, a microphone, switches, and levers, as well as operation devices that allow input by voice, gestures, or other means other than manual operation. Furthermore, for example, the input unit 101 may be a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that supports operation of the vehicle control system 100. The input unit 101 generates an input signal based on the data, instructions, etc. input by the passenger, and supplies the input signal to each component of the vehicle control system 100.

[0045] The data acquisition unit 102 includes various sensors and the like that acquire data used for processing by the vehicle control system 100 , and supplies the acquired data to each unit of the vehicle control system 100 .

[0046] For example, the data acquisition unit 102 includes various sensors for detecting the state of the vehicle, etc. Specifically, for example, the data acquisition unit 102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the motor rotation speed, or the rotation speed of the wheels, etc.

[0047] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting information outside the vehicle. Specifically, for example, the data acquisition unit 102 includes imaging devices such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. Furthermore, for example, the data acquisition unit 102 includes an environmental sensor for detecting weather or climate, and a surrounding information detection sensor for detecting objects around the vehicle. The environmental sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, etc. The surrounding information detection sensor includes, for example, an ultrasonic sensor, radar, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), sonar, etc.

[0048] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 102 includes a GNSS (Global Navigation Satellite System) receiver that receives GNSS signals from GNSS satellites.

[0049] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 102 includes an imaging device (such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras) that captures an image of the driver, a biosensor that detects biometric information of the driver, and a microphone that collects sound inside the vehicle. The biosensor is provided, for example, on the seat or the steering wheel, and detects biometric information of a passenger sitting in the seat or a driver gripping the steering wheel.

[0050] The communication unit 103 communicates with the in-vehicle device 104 as well as various devices, servers, base stations, etc. outside the vehicle, transmits data supplied from each part of the vehicle control system 100, and supplies received data to each part of the vehicle control system 100. Note that the communication protocol supported by the communication unit 103 is not particularly limited, and the communication unit 103 can also support multiple types of communication protocols.

[0051] For example, the communication unit 103 performs wireless communication with the in-vehicle device 104 using wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), WUSB (Wireless USB), etc. Also, for example, the communication unit 103 performs wired communication with the in-vehicle device 104 using USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), etc. via a connection terminal (and a cable, if necessary) not shown.

[0052] Furthermore, for example, the communication unit 103 communicates with devices (e.g., application servers or control servers) present on an external network (e.g., the Internet, a cloud network, or a network specific to a carrier) via a base station or an access point. Furthermore, for example, the communication unit 103 communicates with terminals present near the vehicle (e.g., terminals of pedestrians or stores, or MTC (Machine Type Communication) terminals) using P2P (Peer To Peer) technology. Furthermore, for example, the communication unit 103 performs V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. Furthermore, for example, the communication unit 103 includes a beacon receiver that receives radio waves or electromagnetic waves transmitted from radio stations or the like installed on roads, and acquires information such as the current location, congestion, traffic restrictions, and required travel time. Furthermore, the communication unit 103 is controlled by E-call 203 (FIG. 5) described later, and transmits the location of the accident (GPS coordinates) to a center on an external network that contacts the police, hospitals, etc. in conjunction with the operating status of sensors that detect collisions such as airbags.

[0053] The in-vehicle devices 104 include, for example, mobile devices or wearable devices owned by the passengers, information devices carried into or attached to the vehicle, and navigation devices that search for routes to any destination.

[0054] The output control unit 105 controls the output of various types of information to passengers in the vehicle or to the outside of the vehicle. For example, the output control unit 105 generates an output signal including at least one of visual information (e.g., image data) and auditory information (e.g., audio data) and supplies the output signal to the output unit 106, thereby controlling the output of the visual information and audio information from the output unit 106. Specifically, for example, the output control unit 105 synthesizes image data captured by different imaging devices in the data acquisition unit 102 to generate an overhead image, a panoramic image, or the like, and supplies an output signal including the generated image to the output unit 106. Furthermore, for example, the output control unit 105 generates audio data including a warning sound or a warning message against danger such as a collision, contact, or entry into a dangerous area, and supplies an output signal including the generated audio data to the output unit 106.

[0055] The output unit 106 includes a device capable of outputting visual or auditory information to the occupants of the vehicle or to the outside of the vehicle. For example, the output unit 106 includes a display device (including the information display unit 252 (FIG. 5)), an instrument panel, audio speakers, headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, a lamp, etc. The display device included in the output unit 106 may be a device having a normal display, or may be a device that displays visual information within the driver's field of view, such as a head-up display (HUD), a see-through display, or a device with an augmented reality (AR) display function. The output unit 106 also includes a configuration having a feedback function for waking the driver up and directing their gaze in an appropriate direction when the driver's gaze direction is in an inappropriate direction and the driver is predicted to be distracted or drowsy while driving. The components having a feedback function include, for example, a device that displays visual information within the driver's field of vision (information display unit 252 (Fig. 5) etc.), a speaker 253 (Fig. 5) that outputs audio, a conversation agent unit 254 (Fig. 5) that uses a microphone and speaker to converse with the driver and determine whether the conversation can proceed normally, a mechanism for vibrating the seat belt, a mechanism for vibrating the steering wheel, a mechanism for vibrating the seat, and an irritating odor generating unit 251 (Fig. 5) that generates an irritating odor.

[0056] The drivetrain control unit 107 generates various control signals and supplies them to the drivetrain system 108, thereby controlling the drivetrain system 108. In addition, the drivetrain control unit 107 supplies control signals to each unit other than the drivetrain system 108 as necessary, and notifies the control status of the drivetrain system 108, etc.

[0057] The drivetrain system 108 includes various devices related to the drivetrain of the vehicle, such as a drive force generating device for generating drive force from an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle, a braking device for generating braking force, an ABS (Antilock Brake System), an ESC (Electronic Stability Control), and an electric power steering device.

[0058] The body system control unit 109 generates various control signals and supplies them to the body system 110, thereby controlling the body system 110. Furthermore, the body system control unit 109 supplies control signals to each unit other than the body system 110 as necessary, and notifies the control status of the body system 110, etc.

[0059] The body system 110 includes various body system devices mounted on the vehicle body, such as a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioning system, and various lamps (for example, head lamps, backup lamps, brake lamps, blinkers, fog lamps, etc.).

[0060] The storage unit 111 includes, for example, a magnetic storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage unit 111 stores various programs, data, and the like used by each unit of the vehicle control system 100. For example, the storage unit 111 stores map data such as a three-dimensional high-precision map such as a dynamic map, a global map that is less accurate than a high-precision map and covers a wide area, and a local map that includes information about the surroundings of the vehicle.

[0061] The autonomous driving control unit 112 performs control related to autonomous driving, such as autonomous driving or driving assistance. Specifically, for example, the autonomous driving control unit 112 performs cooperative control aimed at realizing functions of an Advanced Driver Assistance System (ADAS), including collision avoidance or impact mitigation for the host vehicle, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, collision warning for the host vehicle, or lane departure warning for the host vehicle. Furthermore, for example, the autonomous driving control unit 112 performs cooperative control aimed at autonomous driving, which travels autonomously without relying on driver operation. The autonomous driving control unit 112 includes a detection unit 131, a self-position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an operation control unit 135.

[0062] The detection unit 131 detects various types of information necessary for controlling autonomous driving. The detection unit 131 includes an outside-vehicle information detection unit 141, an inside-vehicle information detection unit 142, and a vehicle state detection unit 143.

[0063] The outside-vehicle information detection unit 141 performs a process of detecting information outside the vehicle based on data or signals from each unit of the vehicle control system 100. For example, the outside-vehicle information detection unit 141 performs a process of detecting, recognizing, and tracking objects around the vehicle, as well as a process of detecting the distance to the objects. Objects to be detected include, for example, vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. Furthermore, for example, the outside-vehicle information detection unit 141 performs a process of detecting the environment around the vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, and road surface conditions. The outside-vehicle information detection unit 141 supplies data indicating the results of the detection process to the self-position estimation unit 132, the map analysis unit 151, traffic rule recognition unit 152, and situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, etc.

[0064] The interior information detection unit 142 performs a process of detecting information about the interior of the vehicle based on data or signals from each unit of the vehicle control system 100. For example, the interior information detection unit 142 performs a process of authenticating and recognizing the driver, a process of detecting the driver's state, a process of detecting passengers, and a process of detecting the interior environment of the vehicle. The driver's state to be detected includes, for example, physical condition, alertness, concentration, fatigue, and gaze direction. The interior environment to be detected includes, for example, temperature, humidity, brightness, and odor. The interior information detection unit 142 supplies data indicating the results of the detection process to the situation recognition unit 153 of the situation analysis unit 133 and the emergency situation avoidance unit 171 of the operation control unit 135.

[0065] The vehicle state detection unit 143 performs a process of detecting the state of the vehicle based on data or signals from each unit of the vehicle control system 100. The vehicle state to be detected includes, for example, the speed, acceleration, steering angle, the presence or absence and content of an abnormality, the state of driving operation, the position and tilt of the power seat, the state of door locks, and the state of other in-vehicle devices. The vehicle state detection unit 143 supplies data indicating the results of the detection process to the situation recognition unit 153 of the situation analysis unit 133 and the emergency situation avoidance unit 171 of the operation control unit 135, etc.

[0066] The self-position estimation unit 132 performs estimation processing of the position, attitude, etc. of the vehicle based on data or signals from each unit of the vehicle control system 100, such as the outside-vehicle information detection unit 141 and the situation recognition unit 153 of the situation analysis unit 133. Furthermore, the self-position estimation unit 132 generates a local map (hereinafter referred to as a self-position estimation map) used for estimating the self-position as necessary. The self-position estimation map is a high-precision map using technology such as SLAM (Simultaneous Localization and Mapping). The self-position estimation unit 132 supplies data indicating the result of the estimation processing to the map analysis unit 151, the traffic rule recognition unit 152, the situation recognition unit 153, etc. of the situation analysis unit 133. Furthermore, the self-position estimation unit 132 stores the self-position estimation map in the storage unit 111.

[0067] The situation analysis unit 133 performs an analysis process of the situation of the vehicle and its surroundings. The situation analysis unit 133 includes a map analysis unit 151, a traffic rule recognition unit 152, a situation recognition unit 153, and a situation prediction unit 154.

[0068] The map analysis unit 151 analyzes various maps stored in the storage unit 111, using data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132 and the outside-vehicle information detection unit 141, as needed, to construct a map including information necessary for autonomous driving processing. The map analysis unit 151 supplies the constructed map to the traffic rule recognition unit 152, the situation recognition unit 153, the situation prediction unit 154, and the route planning unit 161, the behavior planning unit 162, the operation planning unit 163, etc. of the planner 134.

[0069] The traffic rule recognition unit 152 performs a recognition process of traffic rules around the vehicle based on data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132, the outside vehicle information detection unit 141, and the map analysis unit 151. This recognition process recognizes, for example, the positions and states of traffic signals around the vehicle, the details of traffic regulations around the vehicle, and available lanes. The traffic rule recognition unit 152 supplies data indicating the results of the recognition process to the situation prediction unit 154 and the like.

[0070] The situation recognition unit 153 performs recognition processing of the situation related to the vehicle based on data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132, the vehicle outside information detection unit 141, the vehicle inside information detection unit 142, the vehicle state detection unit 143, and the map analysis unit 151. For example, the situation recognition unit 153 performs recognition processing of the situation of the vehicle, the situation around the vehicle, the situation of the driver of the vehicle, etc. Furthermore, the situation recognition unit 153 generates a local map (hereinafter referred to as a situation recognition map) used to recognize the situation around the vehicle, as needed. The situation recognition map is, for example, an occupancy grid map.

[0071] The vehicle's conditions to be recognized include, for example, the vehicle's position, posture, and movement (e.g., speed, acceleration, direction of movement, etc.), as well as the presence or absence of abnormalities and their details. The vehicle's surrounding conditions to be recognized include, for example, the type and position of surrounding stationary objects, the type, position, and movement (e.g., speed, acceleration, direction of movement, etc.) of surrounding moving objects, the configuration and road surface condition of surrounding roads, and the surrounding weather, temperature, humidity, and brightness. The driver's conditions to be recognized include, for example, physical condition, alertness, concentration, fatigue, gaze (gaze direction) movement, driving operation, etc.

[0072] The situation recognition unit 153 supplies data indicating the result of the recognition process (including a situation recognition map, if necessary) to the self-position estimation unit 132, the situation prediction unit 154, etc. Furthermore, the situation recognition unit 153 stores the situation recognition map in the storage unit 111.

[0073] The situation prediction unit 154 performs prediction processing of the situation regarding the vehicle based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151, the traffic rule recognition unit 152, and the situation recognition unit 153. For example, the situation prediction unit 154 performs prediction processing of the situation of the vehicle, the situation around the vehicle, the situation of the driver, and the like.

[0074] The vehicle conditions to be predicted include, for example, the vehicle's behavior, the occurrence of an abnormality, and the driving distance. The conditions around the vehicle to be predicted include, for example, the behavior of moving objects around the vehicle, changes in traffic lights, and changes in the environment such as the weather. The driver conditions to be predicted include, for example, the driver's behavior and physical condition.

[0075] The situation prediction unit 154 supplies data indicating the results of the prediction process to the route planning unit 161, behavior planning unit 162, and operation planning unit 163 of the planning unit 134, together with data from the traffic rule recognition unit 152 and the situation recognition unit 153.

[0076] The route planning unit 161 plans a route to the destination based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the route planning unit 161 sets a route from the current location to a specified destination based on a global map. Furthermore, for example, the route planning unit 161 changes the route as appropriate based on conditions such as congestion, accidents, traffic restrictions, and construction, as well as the driver's physical condition. The route planning unit 161 supplies data indicating the planned route to the action planning unit 162 and the like.

[0077] The behavior planning unit 162 plans the behavior of the vehicle to travel safely along the route planned by the route planning unit 161 within the planned time based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the behavior planning unit 162 plans starting, stopping, traveling direction (for example, moving forward, backward, turning left, turning right, changing direction, etc.), traveling lane, traveling speed, overtaking, etc. The behavior planning unit 162 supplies data indicating the planned behavior of the vehicle to the operation planning unit 163, etc.

[0078] The action planning unit 163 plans the action of the host vehicle to realize the action planned by the action planning unit 162, based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the action planning unit 163 plans acceleration, deceleration, a driving trajectory, etc. The action planning unit 163 supplies data indicating the planned action of the host vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, etc. of the action control unit 135.

[0079] The operation control unit 135 controls the operation of the host vehicle. The operation control unit 135 includes an emergency situation avoidance unit 171, an acceleration / deceleration control unit 172, and a direction control unit 173.

[0080] The emergency situation avoidance unit 171 performs detection processing for an emergency situation such as a collision, contact, entry into a dangerous area, driver abnormality, or vehicle abnormality based on the detection results of the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, and the vehicle state detection unit 143. When the emergency situation avoidance unit 171 detects the occurrence of an emergency situation, it plans an operation of the vehicle to avoid the emergency situation such as a sudden stop or a sharp turn. The emergency situation avoidance unit 171 supplies data indicating the planned operation of the vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, etc.

[0081] The acceleration / deceleration control unit 172 performs acceleration / deceleration control to realize the operation of the host vehicle planned by the operation planning unit 163 or the emergency situation avoidance unit 171. For example, the acceleration / deceleration control unit 172 calculates a control target value of a driving force generation device or a braking device to realize the planned acceleration, deceleration, or sudden stop, and supplies a control command indicating the calculated control target value to the drive train control unit 107.

[0082] The direction control unit 173 performs direction control to realize the operation of the host vehicle planned by the action planning unit 163 or the emergency situation avoidance unit 171. For example, the direction control unit 173 calculates a control target value of the steering mechanism to realize the traveling trajectory or sharp turn planned by the action planning unit 163 or the emergency situation avoidance unit 171, and supplies a control command indicating the calculated control target value to the drive train control unit 107.

[0083] <<3. Configuration for realizing driving control to ensure safety according to line of sight>> Next, a configuration for realizing driving control to ensure safety in accordance with the detected line of sight will be described with reference to the block diagram of Fig. 5. Fig. 5 shows an excerpt of the configuration for realizing driving control to ensure safety in accordance with the detected line of sight from the example of functional configurations for realizing vehicle control system 100 described with reference to Fig. 4.

[0084] The data acquisition unit 102 includes an in-vehicle sensor 231 for detecting the line of sight. Specifically, the in-vehicle sensor 231 here corresponds to the camera Cam having the functions of an image sensor and a ToF sensor (ToF camera) described with reference to FIG.

[0085] That is, the in-vehicle sensor 231 acquires an image of the vicinity of the face of the driver H from an image acquired by an image sensor or a distance image acquired by a ToF sensor, and outputs the image to the in-vehicle information detection unit 142 of the detection unit 131 in the autonomous driving control unit 112. The in-vehicle information detection unit 142 detects the line of sight of the driver H based on the position of the pupils in the image, and outputs the detected line of sight to the situation recognition unit 153 of the situation analysis unit 133.

[0086] In addition, the outside vehicle information detection unit 141 performs detection processing, recognition processing, and tracking processing of objects around the vehicle supplied from the data acquisition unit 102, as well as detection processing of the distance to the objects, and based on the processing results, generates vehicle surroundings information at the time when the line of sight direction is detected and outputs it to the situation recognition unit 153 of the situation analysis unit 133.

[0087] The situation recognition unit 153 includes an in-vehicle environment recognition unit 201, a feedback control unit 202, and an E-call control unit (E-call) 203 as components for realizing driving control according to the detected line of sight direction.

[0088] The in-vehicle environment recognition unit 201 determines whether the driver's gaze direction inside the vehicle is appropriate for the driving situation (whether the driver is gazing at the appropriate gaze direction for an appropriate period of time) based on the vehicle surroundings information supplied from the outside-vehicle information detection unit 141 of the detection unit 131 and the gaze direction information supplied from the inside-vehicle information detection unit 142. Then, the in-vehicle environment recognition unit 201 outputs the determination result to the situation prediction unit 154 and the feedback control unit 202.

[0089] More specifically, the in-vehicle environment recognition unit 201 acquires, as vehicle surroundings information, the results of processing such as detection processing, recognition processing, and tracking processing of objects around the vehicle, as well as processing to detect the distance to the object, along with information on the line of sight direction acquired by the in-vehicle sensor 231 from the detection unit 131.

[0090] Here, information about objects around the vehicle includes, for example, vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. Furthermore, information about the environment around the vehicle includes weather, temperature, humidity, brightness, road surface conditions, etc.

[0091] Furthermore, the driver's line of sight changes to an appropriate direction depending on the driving situation (driving state).

[0092] For example, in a driving situation where the driver is traveling straight ahead on a road such as an expressway where the legal speed limit is relatively high, the appropriate line of sight is looking straight ahead.

[0093] However, when driving on a narrow alley with a lot of people, it is appropriate to keep your eyes directed toward pedestrians and cyclists walking on the shoulder of the road, as well as oncoming vehicles.

[0094] In addition, in a driving situation where a lane change is required, the appropriate gaze direction is one in which the gaze shifts sequentially to the rearview mirror, then to the door mirror, and finally to the lane to be changed, as is generally described in driving manuals, etc. Furthermore, regarding the duration of gaze, it is determined that it is not appropriate to gaze continuously at the rearview mirror or door mirror for more than a few seconds.

[0095] In addition, when the door mirrors or rearview mirrors are replaced by cameras that capture the range visible through the door mirrors or rearview mirrors, the range in which the image captured by the camera installed in place of the door mirrors or rearview mirrors is displayed becomes the appropriate line of sight direction corresponding to the direction of the door mirrors or rearview mirrors.

[0096] Based on the vehicle surroundings information and gaze direction information supplied from the detection unit 131, the in-vehicle environment recognition unit 201 determines whether the driver's gaze direction is appropriate for the driving situation, and outputs the determination result to the situation prediction unit 154.

[0097] In the following description, a line of sight direction that is deemed appropriate for the driving state based on the vehicle surroundings information will be referred to as a safe line of sight direction.

[0098] For example, when it is recognized from the vehicle surrounding information that the vehicle is traveling on a highway, if the driver's line of sight is directed straight ahead, the driver's line of sight is considered to be the safe line of sight direction.

[0099] Furthermore, when the vehicle surroundings information indicates that the vehicle is traveling on a busy alley, for example, if the driver's gaze is directed toward pedestrians, bicycles, oncoming vehicles, etc., the driver's gaze direction is considered to be the safe gaze direction.

[0100] Furthermore, when it is recognized from the vehicle surroundings information that a lane change or a right or left turn is to be made, for example, by turning on a blinker, the driver's line of sight is considered to be a safe line of sight direction when the driver's line of sight is changed in a time-series manner to a direction required at each timing, such as the rearview mirror, the door mirror (if the door mirror or the rearview mirror is a camera, the area where an image captured by the camera is displayed), the direction of the lane to be changed, or the direction of the road to be turned right or left. Here, the direction (position) and order of the line of sight that should be changed in a time-series manner may be the direction (position) and order described in, for example, a manual on driving a car.

[0101] In this way, the safe line of sight direction changes depending on the driving situation (driving state) recognized from the vehicle surroundings information.

[0102] If the gaze direction is not appropriate based on the judgment result supplied from the in-vehicle environment recognition unit 201, the feedback control unit 202 sends feedback to the driver from the output unit 106 via the output control unit 105 to prompt the driver to direct their gaze in an appropriate direction.

[0103] In other words, if the direction of gaze is not appropriate, there is a possibility that the driver is in a state of dangerous driving, such as looking away from the road or falling asleep at the wheel, so the feedback control unit 202 provides feedback to the driver to wake him up and adjust the direction of gaze to the appropriate direction.

[0104] More specifically, the feedback control unit 202 controls the irritating odor generating unit 251 of the output unit 106 via the output control unit 105 to generate an irritating odor, stimulate the driver's sense of smell to wake him up, and provide feedback to the driver to direct his gaze in an appropriate direction.

[0105] In addition, the feedback control unit 202 controls the information display unit 252 of the output unit 106 via the output control unit 105 to present image information (including text, moving images, etc.) that indicates that the line of sight is not appropriate and prompts the driver to turn his / her line of sight to an appropriate direction, thereby providing feedback to the driver to turn his / her line of sight in an appropriate direction.

[0106] Furthermore, the feedback control unit 202 controls the speaker 253 of the output unit 106 via the output control unit 105 to output a voice message informing the driver that the line of sight is inappropriate and to prompt the driver to change the line of sight to an appropriate direction, thereby providing feedback to the driver to change the line of sight to an appropriate direction. The feedback control unit 202 controls the conversation agent unit 254 of the output unit 106 via the output control unit 105 to ask the driver a question, have the driver respond, thereby awakening the driver's consciousness, and provide feedback to the driver to change the line of sight to an appropriate direction. The question to the driver may be, for example, the driver's name or date of birth, and the driver is prompted to respond to awaken the driver, thereby providing feedback to the driver to change the line of sight to an appropriate direction.

[0107] The feedback control unit 202 provides feedback to the driver via at least one of the irritating odor generating unit 251, the information display unit 252, the speaker 253, and the conversation agent unit 254, to encourage the driver to direct his or her line of sight in an appropriate direction.

[0108] As for feedback, in addition to the irritating odor generated by the irritating odor generating unit 251, the image displayed by the information display unit 252, the sound output from the speaker 253, and the conversation by the conversation agent unit 254, other methods may be used as long as they can present information that prompts the driver to turn their gaze in an appropriate direction.

[0109] Other more specific feedback configurations may include, for example, a HUD (Head Up Display) that displays information encouraging the driver to look in an appropriate direction, a configuration that vibrates the seat belt, a configuration that vibrates the steering wheel, and a configuration that vibrates the seat.

[0110] When the detection unit 131 detects information such as an impact detected when a collision accident occurs, the E-call control unit 203 acquires information about the vehicle's own position from the self-position estimation unit 132 and controls the communication unit 103 to notify a center that notifies the police, hospitals, etc. of the occurrence of the accident of information indicating that an accident has occurred, including information about the vehicle's own position.

[0111] The situation prediction unit 154 predicts the situation from the in-vehicle environment of the driver based on the determination result based on the line of sight supplied from the in-vehicle environment recognition unit 201, and outputs a command according to the prediction result to the action planning unit 163 of the planning unit 134. More specifically, if the line of sight direction is inappropriate, the situation prediction unit 154 predicts the possibility that the driver is in a dangerous driving state, such as inattentive driving or drowsy driving, and transmits a command to the planning unit 134 to execute safe driving control processing based on the prediction result.

[0112] When a command instructing safe driving control processing is sent from the situation prediction unit 154, the operation planning unit 163 of the planning unit 134 plans an operation to realize safe driving control processing according to the autonomous driving level.

[0113] More specifically, when the autonomous driving level is level 2 and there are multiple lanes on the road currently being traveled, the operation planning unit 163 controls the communication unit 103 to obtain information about the lanes of the road currently being traveled from a cloud server or the like.

[0114] The lane information acquired here includes the proportion of parked vehicles in each lane at each time, the proportion of vehicles entering and leaving nearby facilities at each time, congestion forecasts at each time, and passing lane information at each time. The proportion of parked vehicles in each lane at each time also includes cases where a lane is impassable due to an accident or other reason. Passing lane information is information when there are multiple lanes. It includes information on whether the lane is a driving lane, and if so, how far the lane is from the driving lane, the legal speed limit for each lane, etc.

[0115] The operation planning unit 163 calculates a risk assessment value for each lane based on the lane information, for example, using the following formula (1).

[0116] S x =P t ×w P +L t ×w L +T t ×w T +O t ×w O ···(1)

[0117] where S x is the risk assessment value of lane x at time t, and P t is the proportion of parked vehicles at time t, and w P is the weight for the proportion of parked vehicles present at time t.

[0118] Also, L t is the rate of incoming and outgoing goods at the surrounding facility at time t, and w L is the weight for the ratio of incoming and outgoing goods to the surrounding facilities at time t.

[0119] Furthermore, T t is the traffic congestion forecast at time t, and w T is the weight for the congestion prediction at time t.

[0120] Also, O t is the passing lane information at time t, and w O is the weight for the passing lane information at time t.

[0121] The operation planning unit 163 calculates the current risk evaluation value S for each of the multiple lanes. x The lane with the smallest distance is considered to be the safest lane, and a driving operation is planned to change lanes to the safest lane.

[0122] Furthermore, if the smallest lane is the lane currently being driven on, if the road is a relatively narrow alley, or if there is only one lane, the operation planning unit 163 plans a driving operation that controls the current driving speed to slow down by a predetermined rate.

[0123] In both lane changes and slowing down the driving speed, it is possible to secure time to delay the occurrence of an accident caused by inattentive driving or drowsy driving, and to reduce the level of damage in the unlikely event that an accident does occur, thereby realizing safe driving control.

[0124] Furthermore, when the autonomous driving level is level 1, the action planning unit 163 plans a driving action to decelerate the current driving speed by a predetermined rate.

[0125] Furthermore, when the autonomous driving level is level 0, all driving operations are performed by the driver, so no driving actions are planned, but the action planning unit 163 presents information urging the driver to improve their line of sight to an appropriate direction since the line of sight is not appropriate.

[0126] The motion planning unit 163 supplies the planned motion plan to the vehicle control unit 211 of the motion control unit 135.

[0127] The vehicle control unit 211 controls the acceleration / deceleration control unit 172 and the direction control unit 173 to control the operation of the vehicle in accordance with the operation plan.

[0128] That is, when an operation plan for decelerating the traveling speed is supplied, the acceleration / deceleration control unit 172 controls the driving system 108 via the driving system control unit 107 so as to decelerate the traveling speed by a predetermined rate in accordance with the operation plan.

[0129] Furthermore, when an operation plan for changing lanes is supplied, the direction control unit 173 controls the direction in accordance with the operation plan so as to achieve a lane change to the safest lane.

[0130] At this time, the action planning unit 163 controls the information display unit 252 and the speaker 253 of the output unit 106 via the output control unit 105 to present information according to the planned action plan using images and sounds.

[0131] That is, in this case, the operation planning unit 163 controls the information display unit 252 and the speaker 253 of the output unit 106 via the output control unit 105 to present the driver with information such as changing lanes or slowing down the driving speed because the line of sight direction is not appropriate, by means of images and sounds.

[0132] In this way, by providing advance notice of lane changes and deceleration through the safe driving control process, even if the driver wakes up just before the lane change or deceleration occurs, the driver will be surprised by the sudden lane change or deceleration, and accidents caused by careless driving operations can be prevented.

[0133] <<4. Automatic Driving Control Processing>> Next, the automatic driving control process performed by the vehicle control system 100 of FIG. 5 will be described with reference to the flowchart of FIG.

[0134] In step S11, the interior sensor 231 of the data acquisition unit 102 captures an image of the area around the driver's face and outputs the captured image to the interior information detection unit 142 of the detection unit 131. The interior information detection unit 142 detects the driver's line of sight based on the position of the driver's eyes in the captured image, and outputs the detected direction of sight to the interior environment recognition unit 201 in the situation recognition unit 153. That is, through this processing, the interior environment recognition unit 201 acquires information on the line of sight.

[0135] More specifically, the in-vehicle information detection unit 142 detects the position of the driver's eyes based on, for example, an image captured by the image sensor of the in-vehicle sensor 231 or a distance image captured by a ToF sensor, and detects the driver's line of sight according to the position of the eyes.

[0136] Furthermore, the in-vehicle information detection unit 142 may detect the gaze direction from an image captured by the image sensor of the in-vehicle sensor 231 in a bright daytime environment, and may detect the gaze direction based on a distance image captured by a ToF sensor in a dark environment such as at night or during bad weather. Furthermore, the image sensor and the ToF sensor may be combined to improve robustness and shorten processing time.

[0137] In step S12, the data acquisition unit 102 detects information outside the vehicle using various sensors other than the interior sensor 231, and outputs the information to the exterior information detection unit 141 of the detection unit 131. The exterior information detection unit 141 acquires the information from the data acquisition unit 102, generates vehicle surroundings information, and outputs the information to the interior environment recognition unit 201 in the situation recognition unit 153. Through this process, the interior environment recognition unit 201 acquires the vehicle surroundings information.

[0138] In step S13, the vehicle interior environment recognition unit 201 determines whether the line of sight direction is appropriate for the driving situation based on information about the line of sight direction (including the time period during which the line of sight is fixed) and vehicle surroundings information.

[0139] As described above, based on the vehicle surroundings information, if the road currently being traveled is a highway or the like and the vehicle is traveling straight ahead on the road, the direction straight ahead is deemed to be the appropriate line of sight.

[0140] Furthermore, when driving on a narrow alley with a lot of people, the direction of gaze toward pedestrians and cyclists walking on the shoulder of the road, as well as toward oncoming vehicles, is deemed to be the appropriate direction of gaze based on the vehicle surroundings information.

[0141] Furthermore, when performing a driving maneuver to change lanes based on information about the vehicle's surroundings, the gaze direction is considered appropriate when there is a time-series change in the gaze direction required for safety, such as shifting the gaze direction to the rearview mirror, then to the door mirror, and finally to the lane to be changed, as is generally described in driving manuals, etc.

[0142] In step S13, if the gaze direction (assuming an appropriate gaze time) is not deemed appropriate for the driving situation (for example, including cases where the gaze direction is appropriate but the gaze time is inappropriate), processing proceeds to step S14.

[0143] In step S14, if the gaze direction is not appropriate based on the judgment result supplied from the in-vehicle environment recognition unit 201, the feedback control unit 202 provides feedback to the driver via the output control unit 105 using at least one of the irritating odor generation unit 251, information display unit 252, and speaker 253 of the output unit 106 to urge the driver to direct their gaze in an appropriate direction.

[0144] In step S15, the in-vehicle environment recognition unit 201 again acquires information on the line of sight direction, similar to the process in step S11.

[0145] In step S16, the vehicle interior environment recognition unit 201 again acquires vehicle surroundings information in the same manner as in the process of step S12.

[0146] In step S17, the in-vehicle environment recognition unit 201 determines whether the line of sight direction has been improved to a direction appropriate for the driving situation by the feedback from the feedback control unit 202.

[0147] In step S17, if the line of sight direction has not been improved to a direction appropriate for the driving situation, the process proceeds to step S18.

[0148] In step S18, the in-vehicle environment recognition unit 201 outputs information indicating that the line of sight direction is not appropriate for the driving situation and vehicle surrounding information to the situation prediction unit 154. If the line of sight direction is not appropriate, the situation prediction unit 154 predicts the possibility that the driver is engaging in dangerous driving, such as looking away from the road or falling asleep at the wheel, and transmits a command to the action planning unit 163 of the planner 134 to execute safe driving control processing based on the prediction result. At this time, the situation prediction unit 154 transmits the vehicle surrounding information together with the command to prompt the safe driving control processing to the action planning unit 163 of the planner 134.

[0149] In step S19, the operation planning unit 163 of the planning unit 134 executes a safe driving control process to ensure safety for the current driving situation, based on the command for executing the safe driving control process and the vehicle surrounding information.

[0150] The safe driving control process will be described in detail later with reference to the flowchart of FIG.

[0151] When the safe driving control process is performed in step S19 and safety is ensured, the process proceeds to step S20.

[0152] In step S20, the automatic driving control unit 112 determines whether an instruction to end automatic driving has been issued, and if it is determined that an instruction to end automatic driving has not been issued, the process returns to step S11 and the subsequent processes are repeated.

[0153] Furthermore, in step S13, if the line of sight direction is deemed to be appropriate for the driving situation, the processes of steps S14 to S19 are skipped.

[0154] Furthermore, in step S17, if the line of sight direction is deemed to be appropriate for the driving situation, the processes of steps S18 and S19 are skipped.

[0155] Then, in step S20, if an instruction to end the automatic driving is given, the automatic driving process ends.

[0156] Through the above processing, it is determined whether the line of sight direction is appropriate for the driving situation, and if it is inappropriate, processing is performed to prompt the driver to improve the line of sight direction, and if no improvement is made, safety driving control processing is executed to ensure safety.

[0157] <<5. Safe driving control processing>> Next, the safe driving control process will be described with reference to the flowchart of FIG.

[0158] In step S31, the operation planning unit 163 determines whether the current autonomous driving level is level 2. If it is determined in step S31 that the autonomous driving level is level 2, the process proceeds to step S32.

[0159] In step S32, the operation planning unit 163 determines whether the road on which the vehicle is currently traveling is a road with multiple lanes based on the vehicle surroundings information. If the road on which the vehicle is currently traveling is determined to be a road with multiple lanes in step S32, the process proceeds to step S33.

[0160] In step S33, the operation planning unit 163 controls the communication unit 103 to acquire information about the lanes of the road on which the vehicle is currently traveling from a cloud server or the like.

[0161] In step S34, the operation planning unit 163 calculates a risk assessment value for each lane based on the acquired lane information, for example, by the calculation expressed by the above-mentioned formula (1).

[0162] In step S35, the operation planning unit 163 identifies the lane with the smallest risk evaluation value as the safest lane.

[0163] In step S36, the operation planning unit 163 determines whether the safest lane, which has the smallest risk assessment value, is the lane the vehicle is currently traveling in and whether a lane change is necessary. If, in step S36, the lane the vehicle is currently traveling in is not the lane with the smallest risk assessment value and a lane change is necessary, the process proceeds to step S37.

[0164] In step S37, the operation planning unit 163 identifies the direction of movement for changing to the lane that minimizes the risk assessment value.

[0165] In step S38, the operation planning unit 163 controls the output control unit 105 to display an image and sound to the driver via the information display unit 252 and the speaker 253 of the output unit 106, informing the driver that the line of sight is not appropriate and that the driver should change lanes to ensure safety.

[0166] In step S39, the action planning unit 163 notifies the action control unit 135 of the movement direction for changing to the lane with the smallest risk evaluation value. In the action control unit 135, the direction control unit 173 of the vehicle control unit 211 controls the drivetrain control unit 107 to operate the drivetrain system 108 so that the vehicle moves in the notified movement direction for changing to the lane with the smallest risk evaluation value. As a result, the lane change to the lane with the smallest risk evaluation value is realized.

[0167] On the other hand, if the autonomous driving level is not level 2 in step S31, the process proceeds to step S40.

[0168] In step S40, the operation planning unit 163 determines whether the current autonomous driving level is level 1. If the current autonomous driving level is level 1 in step S40, the process proceeds to step S41.

[0169] In step S41, the operation planning unit 163 controls the output control unit 105 to display an image and sound to the driver via the information display unit 252 and the speaker 253 of the output unit 106, informing the driver that the line of sight is not appropriate and that the driver should decelerate to ensure safety.

[0170] In step S42, the action planning unit 163 notifies the action control unit 135 to decelerate by a predetermined percentage with respect to the current running speed. In the action control unit 135, the acceleration / deceleration control unit 172 of the vehicle control unit 211 controls the drivetrain control unit 107 to operate the drivetrain system 108 so as to decelerate by the predetermined percentage with respect to the notified current running speed. As a result, it becomes possible to decelerate by the predetermined percentage with respect to the current running speed.

[0171] Also, in step S40, if the autonomous driving level is not level 1, that is, if the autonomous driving level is level 0 and the driver controls everything, the process proceeds to step S43.

[0172] In step S43, the operation planning unit 163 controls the output control unit 105 to display information to the driver in the form of images and audio via the information display unit 252 and the speaker 253 of the output unit 106, urging the driver to improve the line of sight by moving in an appropriate direction since the line of sight is not appropriate.

[0173] In step S32, if the road currently being traveled does not have multiple lanes, or if in step S36 the lane currently being traveled is the lane with the smallest risk assessment value and no lane change is necessary, the process proceeds to step S41.

[0174] That is, in this case, the current traveling speed is reduced by a predetermined rate to ensure safety.

[0175] As a result of the above processing, if the line of sight is inappropriate and inattentive driving or drowsy driving is suspected, and the autonomous driving level is level 2, and the road being traveled on has multiple lanes and is not the lane with the smallest risk assessment value, the lane will be changed to the lane with the smallest risk assessment value.

[0176] By performing this processing, the risk assessment value of the driving lane is reduced, which can buy time before an accident caused by inattentive driving or drowsy driving occurs in situations where inattentive driving or drowsy driving is suspected, and can also reduce the level of damage if an accident does occur.

[0177] As a result, it is possible to improve safety even in situations where the driver's line of sight is inappropriate and inattentive driving or drowsy driving is suspected.

[0178] In addition, in situations where the direction of gaze is inappropriate and inattentive driving or drowsy driving is suspected, if the autonomous driving level is level 2, the road being traveled on is not a multi-lane road, or the vehicle is traveling on the lane with the smallest risk assessment value, and if the autonomous driving level is level 1, the vehicle will be slowed down by a predetermined percentage of its current driving speed.

[0179] By this processing, in a situation where the line of sight is inappropriate and inattentive driving or drowsy driving is suspected, it is possible to buy time before an accident due to inattentive driving or drowsy driving occurs, and to reduce the level of damage even if an accident does occur. Furthermore, if the line of sight is inappropriate and a situation where inattentive driving or drowsy driving is suspected continues, the speed is gradually reduced, making it possible to ensure even greater safety.

[0180] If the speed is gradually reduced and the line of sight continues to be inappropriate, the vehicle 11 may be eventually stopped. When stopping the vehicle, if the autonomous driving is at level 2 and the direction can be controlled, the vehicle may be controlled to stop at a safe position on the shoulder of the road.

[0181] As a result, it is possible to improve safety even in situations where the driver's line of sight is inappropriate and inattentive driving or drowsy driving is suspected.

[0182] Furthermore, in the safe driving control process, when changing lanes or decelerating, information is presented in advance to warn the driver that driving control such as a lane change or deceleration will be performed, and then driving control is performed. Therefore, if the driver wakes up just before a control operation by the safe driving control process is performed, it is possible to prevent the driver from making a mistake due to a sudden lane change or deceleration and performing an inadvertent driving operation.

[0183] Furthermore, even if the autonomous driving level is Level 0 in a situation where the driver's gaze direction is inappropriate and distracted driving or drowsy driving is suspected, although driving control such as lane changes or deceleration cannot be performed, information will continue to be presented in the form of images and audio to encourage the driver to gaze in the appropriate direction.

[0184] As a result, it is possible to improve safety even in situations where the driver's line of sight is inappropriate and inattentive driving or drowsy driving is suspected.

[0185] <<6. Example of execution by software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes are executed by software, the programs constituting the software are installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0186] 8 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0187] Connected to the input / output interface 1005 are an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected to the input / output interface 1005 is a drive 1010 that reads and writes data from / to a removable recording medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0188] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable recording medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0189] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0190] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable recording medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0191] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable recording medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0192] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0193] 8 realizes the function of the automatic driving control unit 112 in Fig. 5. Also, the storage unit 1008 in Fig. 8 realizes the function of the storage unit 111 in Fig. 5.

[0194] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0195] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0196] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0197] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0198] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0199] The present disclosure can also be configured as follows.

[0200] [Claim 1] an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an acquisition unit that acquires a detection result of an in-vehicle sensor that detects the line of sight of the driver of the vehicle; an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on the detection result; a feedback control unit that, when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction; If the in-vehicle environment recognition unit determines that the line of sight direction of the driver has not improved after the feedback control unit has provided the feedback, the operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to a safe lane. Information processing device. [Claim 2] The vehicle information detecting unit detects vehicle outside information, which is information about the surroundings of the vehicle. The vehicle interior environment recognition unit determines whether the line of sight direction of the driver is appropriate based on the detection result and the vehicle exterior information. The information processing device according to claim 1. [Claim 3] an action planning unit that plans the action of the vehicle to either reduce the traveling speed or change lanes to the safe lane when it is determined that the line of sight direction of the driver has not improved; The operation control unit controls the operation of the vehicle so as to realize the operation planned by the operation planning unit. The information processing device according to claim 2. [Claim 4] When it is determined that the line of sight direction of the driver will not improve, the operation planning unit calculates a risk assessment value for each of a plurality of lanes of the road on which the vehicle is currently traveling, and plans the operation of the vehicle to either reduce the traveling speed or change lanes to the safe lane based on the risk assessment value. The information processing device according to claim 3. [Claim 5] the operation planning unit, when it is determined that the line of sight direction of the driver will not improve, plans the operation of the vehicle based on the risk assessment value so as to change lanes to the safest lane among a plurality of lanes of the road on which the vehicle is currently traveling; The operation control unit controls the operation of the vehicle to change lanes to the safest lane among a plurality of lanes on the road on which the vehicle is currently traveling, according to the plan of the operation planning unit. The information processing device according to claim 4. [Claim 6] the operation planning unit plans a lane change to a lane that is the safest lane among a plurality of lanes of the road on which the vehicle is currently traveling, and that has the smallest risk evaluation value, based on the risk evaluation value; The operation control unit controls the operation of the vehicle so that the vehicle changes lanes to the lane that minimizes the risk assessment value, as planned by the operation planning unit. The information processing device according to claim 5. [Claim 7] When the action planning unit plans the action of changing lanes to the safe lane, the feedback control unit feeds back to the driver that the lane change to the safe lane has been planned; After the driver is fed back that a lane change to the safe lane is planned, the operation control unit controls the operation of the vehicle to change lanes to the safe lane. The information processing device according to claim 3. [Claim 8] the operation planning unit plans a reduction in the traveling speed when the currently traveling lane is the safe lane; The operation control unit controls the operation of the vehicle so as to decelerate the traveling speed in the lane in which the vehicle is currently traveling. The information processing device according to claim 3. [Claim 9] When the operation planning unit determines that the currently traveling lane is the safe lane and plans to decelerate the traveling speed, the feedback control unit feeds back to the driver that it is planned to decelerate the traveling speed; After the driver is fed back that the driving speed is planned to be reduced, the operation control unit controls the operation of the vehicle so as to reduce the driving speed. The information processing device according to claim 8. [Claim 10] When it is determined that the line of sight direction of the driver will not improve, the operation planning unit accesses a cloud server to acquire information on lanes of the road on which the vehicle is currently traveling, and plans the operation of the vehicle to either reduce the traveling speed or change lanes to the safe lane based on the acquired information on the lane of the road on which the vehicle is currently traveling. The information processing device according to claim 3. [Claim 11] When it is determined that the line of sight of the driver is not appropriate, the feedback control unit applies at least one of the following feedback to prompt the driver to turn the line of sight in an appropriate direction: displaying an image on a display unit; outputting sound from a speaker; generating an irritating odor; having the driver talk; vibrating a seat belt; vibrating a steering wheel; and vibrating a seat. The information processing device according to claim 1. [Claim 12] the detection result of the in-vehicle sensor for detecting the line of sight of the driver of the vehicle is an image of the vicinity of the face of the driver; The in-vehicle environment recognition unit identifies a position of the driver's eyes from an image of the vicinity of the driver's face, which is acquired by the acquisition unit as the detection result of the in-vehicle sensor, and detects the line of sight direction of the driver based on the identified position of the eyes. The information processing device according to claim 1. [Claim 13] An operation control process for controlling the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; performing an in-vehicle environment recognition process for determining whether the driver's line of sight is appropriate based on a detection result of an in-vehicle sensor for detecting the line of sight of the driver of the vehicle; a feedback control process for feeding back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction when the line of sight is determined to be inappropriate by the in-vehicle environment recognition process, If the in-vehicle environment recognition process determines that the driver's line of sight direction has not improved after the feedback control process has provided the feedback, the operation control process controls the operation of the vehicle to reduce the driving speed or change lanes to a safe lane. Information processing methods. [Claim 14] an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on a detection result of an in-vehicle sensor that detects the driver's line of sight of the vehicle; when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, causing the computer to function as a feedback control unit that feeds back information to the driver to prompt the driver to change the line of sight to an appropriate direction; If the in-vehicle environment recognition unit determines that the line of sight direction of the driver has not improved after the feedback control unit has provided the feedback, the operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to a safe lane. program. [Claim 15] an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an in-vehicle sensor for detecting the line of sight of a driver of the vehicle; an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on the detection result of the in-vehicle sensor; a feedback control unit that, when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction; If the in-vehicle environment recognition unit determines that the line of sight direction of the driver has not improved after the feedback control unit has provided the feedback, the operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to a safe lane. vehicle. [Explanation of symbols]

[0201] 91 Vehicle, 100 Vehicle control system, 102 Data acquisition unit, 112 Autonomous driving control unit, 133 Situation analysis unit, 134 Planning unit, 135 Operation control unit, 153 Situation recognition unit, 154 Situation prediction unit, 172 Acceleration / deceleration control unit, 173 Direction control unit, 201 In-vehicle environment recognition unit, 202 Feedback control unit, 203 E-call, 211 Vehicle control unit, 231 In-vehicle sensor, 251 Pungent odor generation unit, 252 Information display unit, 253 Speaker

Claims

1. an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an acquisition unit that acquires a detection result of an in-vehicle sensor that detects the line of sight of the driver of the vehicle; an in-vehicle environment recognition unit that determines whether the driver's line of sight direction is appropriate based on the detection result; a feedback control unit that, when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction; and an action planning unit that plans the action of the vehicle, which is either to reduce the traveling speed or to change lanes to a safe lane; If the in-vehicle environment recognition unit determines that the line of sight of the driver has not improved after the feedback control unit has provided the feedback, the operation planning unit calculates a risk assessment value for each of a plurality of lanes of the road on which the vehicle is currently traveling, based on the ratio of parked vehicles present, the ratio of vehicles entering and leaving nearby facilities, congestion prediction, and overtaking lane information, and plans the operation of the vehicle based on the risk assessment value. The operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to the safe lane in accordance with the plan planned by the operation planning unit. Information processing device.

2. The operation planning unit calculates the risk assessment value based on the product sum of the ratio of the parked vehicles present, the ratio of the vehicles entering and leaving the surrounding facilities, the congestion forecast, and the overtaking lane information, and the weights set for each of the ratios, for each of the multiple lanes of the road currently being traveled. The information processing device according to claim 1 .

3. The vehicle information detecting unit detects vehicle outside information, which is information about the surroundings of the vehicle. The vehicle interior environment recognition unit determines whether the line of sight direction of the driver is appropriate based on the detection result and the vehicle exterior information. The information processing device according to claim 1 .

4. the operation planning unit, when it is determined that the line of sight direction of the driver will not improve, plans the operation of the vehicle based on the risk assessment value so as to change lanes to the safest lane among a plurality of lanes of the road on which the vehicle is currently traveling; The operation control unit controls the operation of the vehicle to change lanes to the safest lane among a plurality of lanes on the road on which the vehicle is currently traveling, according to the plan of the operation planning unit. The information processing device according to claim 1 .

5. the operation planning unit plans a lane change to a lane that is the safest lane among a plurality of lanes of the road on which the vehicle is currently traveling, and that has the smallest risk evaluation value, based on the risk evaluation value; The operation control unit controls the operation of the vehicle so that the vehicle changes lanes to the lane that minimizes the risk assessment value, as planned by the operation planning unit. The information processing device according to claim 4 .

6. When the action planning unit plans the action of changing lanes to the safe lane, the feedback control unit feeds back to the driver that the lane change to the safe lane has been planned; After the driver is fed back that a lane change to the safe lane is planned, the operation control unit controls the operation of the vehicle to change lanes to the safe lane. The information processing device according to claim 1 .

7. the operation planning unit plans a reduction in the traveling speed when the currently traveling lane is the safe lane; The operation control unit controls the operation of the vehicle so as to decelerate the traveling speed in the lane in which the vehicle is currently traveling. The information processing device according to claim 1 .

8. When the operation planning unit determines that the currently traveling lane is the safe lane and plans to decelerate the traveling speed, the feedback control unit feeds back to the driver that it is planned to decelerate the traveling speed; After the driver is fed back that the driving speed is planned to be reduced, the operation control unit controls the operation of the vehicle so as to reduce the driving speed. The information processing device according to claim 7 .

9. When it is determined that the line of sight direction of the driver will not improve, the operation planning unit accesses a cloud server to acquire information on lanes of the road on which the vehicle is currently traveling, and plans the operation of the vehicle to either reduce the traveling speed or change lanes to the safe lane based on the acquired information on the lane of the road on which the vehicle is currently traveling. The information processing device according to claim 1 .

10. When it is determined that the line of sight of the driver is not appropriate, the feedback control unit applies at least one of the following feedback to prompt the driver to turn the line of sight in an appropriate direction: displaying an image on a display unit; outputting sound from a speaker; generating an irritating odor; having the driver talk; vibrating a seat belt; vibrating a steering wheel; and vibrating a seat. The information processing device according to claim 1 .

11. the detection result of the in-vehicle sensor for detecting the line of sight of the driver of the vehicle is an image of the vicinity of the face of the driver, The in-vehicle environment recognition unit identifies a position of the driver's eyes from an image of the vicinity of the driver's face, which is acquired by the acquisition unit as the detection result of the in-vehicle sensor, and detects the line of sight direction of the driver based on the identified position of the eyes. The information processing device according to claim 1 .

12. An operation control process for controlling the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; performing an in-vehicle environment recognition process for determining whether the driver's line of sight is appropriate based on a detection result of an in-vehicle sensor for detecting the driver's line of sight of the vehicle; a feedback control process for feeding back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction when the line of sight is determined to be inappropriate by the in-vehicle environment recognition process; and planning the vehicle's movement to either reduce its speed or change lanes to a safe lane; If it is determined by the in-vehicle environment recognition process that the line of sight of the driver has not improved after the feedback is provided by the feedback control process, the operation planning process calculates a risk assessment value for each of a plurality of lanes of the road on which the vehicle is currently traveling, based on the proportion of parked vehicles present, the proportion of vehicles entering and leaving surrounding facilities, congestion prediction, and overtaking lane information, and plans the operation of the vehicle based on the risk assessment value. The operation control process controls the operation of the vehicle to reduce the traveling speed or change lanes to the safe lane in accordance with the plan planned by the operation planning process. Information processing methods.

13. an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on a detection result of an in-vehicle sensor that detects the driver's line of sight; a feedback control unit that, when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction; and causing the computer to function as an action planner that plans the action of the vehicle, either to reduce the driving speed or to change lanes to a safe lane; If the in-vehicle environment recognition unit determines that the line of sight of the driver has not improved after the feedback control unit has provided the feedback, the operation planning unit calculates a risk assessment value for each of a plurality of lanes of the road on which the vehicle is currently traveling, based on the ratio of parked vehicles present, the ratio of vehicles entering and leaving nearby facilities, congestion prediction, and overtaking lane information, and plans the operation of the vehicle based on the risk assessment value. The operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to the safe lane in accordance with the plan planned by the operation planning unit. program.

14. an operation control unit that controls the operation of the vehicle by coordinating driving operations related to acceleration and deceleration and driving operations related to steering; an in-vehicle sensor for detecting the line of sight of a driver of the vehicle; an in-vehicle environment recognition unit that determines whether the driver's line of sight is appropriate based on the detection result of the in-vehicle sensor; a feedback control unit that, when the in-vehicle environment recognition unit determines that the line of sight is inappropriate, feeds back information to the driver to prompt the driver to adjust the line of sight to an appropriate direction; and an action planning unit that plans the action of the vehicle, which is either to reduce the traveling speed or to change lanes to a safe lane; If the in-vehicle environment recognition unit determines that the line of sight of the driver has not improved after the feedback control unit has provided the feedback, the operation planning unit calculates a risk assessment value for each of a plurality of lanes of the road on which the vehicle is currently traveling, based on the ratio of parked vehicles present, the ratio of vehicles entering and leaving nearby facilities, congestion prediction, and overtaking lane information, and plans the operation of the vehicle based on the risk assessment value. The operation control unit controls the operation of the vehicle to reduce the traveling speed or change lanes to the safe lane in accordance with the plan planned by the operation planning unit. vehicle.

Images