Travel control device and travel control method

Abstract

The present disclosure relates to a driving control device and a driving control method. The driving control device includes: a detection unit for detecting a perimeter confirmation action performed by a driver of a vehicle to confirm the perimeter of the vehicle from an image showing the driver's face; a recording unit for recording the time at which the perimeter confirmation action is detected; a proposal unit for notifying the driver of a lane change proposal to change the lane of the vehicle to an adjacent lane; and an execution unit for causing the vehicle to execute a lane change when the perimeter confirmation action is detected during a pre-confirmation period prior to the time at which the lane change proposal is notified, regardless of whether the perimeter confirmation action is detected during a post-confirmation period subsequent to the time at which the lane change proposal is notified.

Description

Technical Field

[0001] The present disclosure relates to a travel control device and a travel control method for automatically controlling the travel of a vehicle. Background Art

[0002] There is known a driving control device that automatically controls the driving of a vehicle such as lane change (LC) based on a peripheral image generated by a camera mounted on the vehicle.

[0003] Patent Document 1 describes a travel control device that proposes LC to a driver and starts LC on the condition that it is determined that the driver has taken an action to check the safety situation around the vehicle (surroundings checking action).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-102519 Summary of the Invention

[0007] When an LC proposal is made after the surrounding confirmation action is performed, it is troublesome for the driver to request the surrounding confirmation action again during the execution of the LC.

[0008] An object of the present disclosure is to provide a travel control device capable of controlling the travel of a vehicle in a manner that does not cause trouble to the driver.

[0009] The driving control device disclosed in the present invention comprises: a detection unit that detects a periphery confirmation action of the driver confirming the periphery of the vehicle from an image showing the face of the driver of the vehicle; a recording unit that records the time when the periphery confirmation action is detected; a proposal unit that notifies the driver of a lane change proposal to change the lane in which the vehicle is traveling to an adjacent lane; and an execution unit that causes the vehicle to execute a lane change when the periphery confirmation action is detected during a pre-confirmation period that is earlier than the time when the lane change proposal is notified, regardless of whether the periphery confirmation action is detected during a post-confirmation period that is later than the time when the lane change proposal is notified.

[0010] In the travel control device of the present disclosure, preferably, when no periphery checking action is detected during the pre-check period, the execution unit causes the vehicle to execute the lane change on the condition that a periphery checking action is detected during the post-proposal check period.

[0011] In the driving control device of the present invention, the detection unit preferably detects the direction of the surroundings of the vehicle confirmed by the driver in the surrounding confirmation action, and the recording unit also records the direction in the surrounding confirmation action. When the recorded direction in the surrounding confirmation action is detected during the prior confirmation period and the surrounding confirmation action corresponds to the direction from the driving lane toward the lane of the change target, the execution unit causes the vehicle to execute a lane change.

[0012] In the travel control device of the present disclosure, preferably, when a periphery confirmation action is detected during the pre-confirmation period, the suggestion unit notifies the driver of the execution of the lane change regardless of whether a periphery confirmation action is detected during the post-proposition confirmation period.

[0013] The driving control method disclosed herein includes: detecting a periphery confirmation action of the driver to confirm the periphery of the vehicle from an image presenting the face of the driver of the vehicle, recording the time when the periphery confirmation action is detected, notifying the driver of a lane change proposal to change the lane in which the vehicle is traveling to an adjacent lane, and when the periphery confirmation action is detected during a pre-confirmation period earlier than the time of notifying the lane change proposal, causing the vehicle to execute a lane change regardless of whether the periphery confirmation action is detected during a post-proposal confirmation period later than the time of notifying the lane change proposal.

[0014] According to the travel control device of the present disclosure, it is possible to control the travel of a vehicle in a manner that does not cause trouble to the driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the structure of a vehicle equipped with a travel control device.

[0016] Figure 2 This is a hardware diagram of the travel control device.

[0017] Figure 3 This is a functional block diagram of a processor included in the travel control device.

[0018] Figure 4 It is a schematic diagram explaining the outline of driving control.

[0019] Figure 5 This is a flowchart of the first travel control process.

[0020] Figure 6 This is a flowchart of the second travel control process.

[0021] Explanation of symbols

[0022] 1: Vehicle; 4: Travel control device; 431: Detection unit; 432: Recording unit; 433: Proposal unit; 434: Execution unit. DETAILED DESCRIPTION

[0023] The following describes in detail a driving control device capable of controlling the driving of a vehicle in a manner that is not burdensome to the driver, with reference to the accompanying drawings. The driving control device detects, from an image showing the driver's face, a perimeter confirmation action performed by the driver to confirm the perimeter of the vehicle. Furthermore, the driving control device records the time at which the perimeter confirmation action is detected. Next, the driving control device notifies the driver of a lane change proposal to change the lane in which the vehicle is traveling to an adjacent lane. Then, if the perimeter confirmation action is detected during a pre-confirmation period prior to the time at which the lane change proposal is notified, the driving control device causes the vehicle to execute a lane change, regardless of whether the perimeter confirmation action is detected during a post-confirmation period subsequent to the time at which the lane change proposal is notified.

[0024] Figure 1 This is a schematic diagram of the structure of a vehicle equipped with a travel control device.

[0025] Vehicle 1 includes perimeter cameras 2, driver monitoring camera 3, and travel control device 4. Perimeter cameras 2, driver monitoring camera 3, and travel control device 4 are communicatively connected via an in-vehicle network conforming to a standard such as a controller area network.

[0026] The perimeter camera 2 is an example of a perimeter imaging unit that generates a perimeter image representing the vehicle's surroundings. The perimeter camera 2 includes a two-dimensional detector, such as a CCD or C-MOS, consisting of an array of photoelectric conversion elements sensitive to visible light, and an imaging optical system that forms an image of the target area on the two-dimensional detector. The perimeter camera 2 is positioned, for example, in the upper front portion of the vehicle interior, facing forward. It captures the surroundings of the vehicle 1 through the windshield at predetermined intervals (e.g., 1 / 30 to 1 / 10 second) and outputs a perimeter image representing the surrounding conditions.

[0027] The driver monitoring camera 3 is an example of a driver imaging unit for generating a facial image showing the facial area of ​​the driver of the vehicle. The driver monitoring camera 3 includes: a two-dimensional detector such as a CCD or C-MOS, which is composed of an array of photoelectric conversion elements that are sensitive to infrared light; and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. In addition, the driver monitoring camera 3 includes a light source that emits infrared light. The driver monitoring camera 3 is installed at the front of the vehicle interior, for example, facing the face of the driver sitting in the driver's seat. The driver monitoring camera 3 irradiates the driver with infrared light at a predetermined shooting period (for example, 1 / 30 second to 1 / 10 second) and outputs an image of the driver's face.

[0028] The driving control device 4 is an ECU (Electronic Control Unit) equipped with a communication interface, memory, and a processor. The driving control device 4 outputs control signals to the vehicle 1's driving mechanisms (not shown), such as the engine, brakes, and steering, to ensure compliance with prescribed safety standards, thereby executing automated driving control for the vehicle 1. Furthermore, the driving control device 4 detects the driver's surrounding verification actions from facial images received from the driver monitoring camera 3 via the communication interface and records the time at which these actions are detected. The driving control device 4 then controls lane changes in the vehicle 1 based on the time at which these actions are detected.

[0029] Figure 2 4 is a hardware diagram of the travel control device 4. The travel control device 4 includes a communication interface 41, a memory 42, and a processor 43.

[0030] The communication interface 41 is an example of a communication unit and includes a communication interface circuit for connecting the travel control device 4 to the in-vehicle network. The communication interface 41 supplies received data to the processor 43. The communication interface 41 also outputs data supplied from the processor 43 to the outside.

[0031] The memory 42 is an example of a storage unit and includes both volatile semiconductor memory and nonvolatile semiconductor memory. The memory 42 stores various data used for processing by the processor 43, such as the conditions for detecting the surrounding confirmation action from a facial image, the parameter group that defines the operation of the neural network as an object recognizer that detects objects from surrounding images, and the time when the surrounding confirmation action is detected. The memory 42 also stores various application programs, such as a travel control program that executes travel control processing.

[0032] The processor 43 is an example of a control unit and includes one or more processors and their peripheral circuits. The processor 43 may also include other arithmetic circuits such as a logical operation unit, a numerical operation unit, or a graphics processing unit.

[0033] Figure 3 This is a functional block diagram of the processor 43 included in the travel control device 4 .

[0034] The processor 43 of the travel control device 4 includes, as functional blocks, a detection unit 431, a recording unit 432, a suggestion unit 433, and an execution unit 434. These components of the processor 43 are functional modules implemented by programs executed on the processor 43. Alternatively, these components of the processor 43 may be implemented in the travel control device 4 as independent integrated circuits, microprocessors, or firmware.

[0035] The detection unit 431 detects the driver's surrounding confirmation actions, which are performed by checking the surroundings of the vehicle, from the facial image received from the driver monitoring camera 3 via the communication interface. For example, the detection unit 431 detects the pupil and corneal reflection images of the light source by performing template matching between a template representing the pupil and corneal reflection images of the light source and the facial image, and detects the gaze direction based on their positional relationship. The gaze direction is represented by the horizontal angle between the direction of travel of the vehicle 1 and the driver's gaze. The detection unit 431 then detects the driver's surrounding confirmation actions when the driver's gaze is directed toward the surroundings of the vehicle 1 (e.g., at an angle of 30 degrees or more relative to the direction of travel).

[0036] Detection unit 431 may also detect the driver's surrounding confirmation action when the angle between the driver's facial orientation detected from the facial image and the horizontal direction of vehicle 1 is greater than a predetermined angle threshold. In this case, detection unit 431 inputs the facial image into a facial part recognizer trained to detect predetermined facial parts such as the outer corners of the eyes, inner corners of the eyes, and corners of the mouth, thereby determining the positions of the predetermined parts included in the facial image. Furthermore, detection unit 431 compares the positions of the predetermined parts detected from the facial image with a standard 3D model of the face. Detection unit 431 then detects the facial orientation angle in the 3D model as the facial orientation angle in the facial image, based on the best fit between the positions of each part and the positions of the part detected from the facial image.

[0037] The facial part recognition system can employ, for example, a convolutional neural network (CNN) having multiple convolutional layers connected in series from the input side to the output side. Facial images are pre-trained as training data and fed into the CNN for learning, allowing the CNN to function as a facial part recognition system that detects specific facial parts from facial images.

[0038] The detection unit 431 detects a gaze direction or a facial orientation angle as a direction of the surroundings of the vehicle 1 that the driver confirms during the surrounding confirmation action.

[0039] The recording unit 432 records the time at which the surrounding confirmation action is detected in the memory 42. The recording unit 432 may also associate data indicating the detection with the time of detection and store it in the memory 42 for a predetermined period of time. Furthermore, the recording unit 432 may associate the time of capture and the presence or absence of the surrounding confirmation action for each of a plurality of facial images received in time series from the driver monitoring camera 3 and record them in the memory 42. Furthermore, the recording unit 432 may also record the direction during the surrounding confirmation action.

[0040] Proposal unit 433 notifies the driver of a lane change proposal to change the lane of vehicle 1 to an adjacent lane. Proposal unit 433 creates a driving route for the lane change and, for example, displays an image presenting the lane change proposal on an instrument display (not shown) connected via communication interface 41. The image presenting the lane change proposal may include, for example, a string such as "Scheduled Lane Change" or an image showing a driving route from the current lane to the adjacent lane. Proposal unit 433 may also, for example, cause a speaker (not shown) connected via communication interface 41 to play a sound presenting the lane change proposal.

[0041] Proposal unit 433 determines whether a lane change is necessary and, if so, notifies the driver of a lane change proposal. For example, proposal unit 433 detects lane markings on the road on which vehicle 1 is traveling and other vehicles traveling around vehicle 1 from surrounding images received from surrounding camera 2 via communication interface 41. Proposal unit 433, for example, notifies the driver of a lane change proposal to an adjacent lane if the gap to another vehicle ahead of vehicle 1 in the lane in which vehicle 1 is traveling decreases and becomes smaller than a gap threshold.

[0042] Furthermore, the suggestion unit 433 notifies the driver of a lane change suggestion before a position where a lane change is required in order to travel along a predetermined travel route (for example, branching to a side road).

[0043] Proposing unit 433 may further inform the driver whether the proposed lane change has been executed by executing unit 434. For example, proposing unit 433 may display an image containing the text strings "Lane change executed" or "Lane change canceled" on the instrument panel. Proposing unit 433 may also play a voice message such as "Lane change executed" or "Lane change canceled" from a speaker.

[0044] When a surrounding confirmation action is detected during the pre-confirmation period, the execution unit 434 causes the vehicle 1 to execute a lane change, regardless of whether a surrounding confirmation action is detected during the post-proposal confirmation period. The pre-confirmation period is a period of a prescribed time (e.g., 5 seconds) before the time the lane change is proposed. The pre-confirmation period may also be a period from the time the lane change is proposed to the prescribed time before. The post-proposal confirmation period is a period of a prescribed time (e.g., 5 seconds) after the time the lane change is proposed. The post-proposal confirmation period may also be a period from the time the lane change is proposed to the prescribed time after. In addition, the length of the pre-confirmation period and the length of the post-proposal confirmation period may be different.

[0045] The execution unit 434 outputs a control signal to a driving mechanism (not shown) of the vehicle 1 via the communication interface 41 so that the vehicle 1 travels according to the driving path for lane change. The driving mechanism includes, for example, an engine that supplies power to the vehicle 1, brakes that reduce the speed of the vehicle 1, and a steering mechanism that steers the vehicle 1.

[0046] When no periphery confirmation action is detected during the pre-confirmation period, the execution unit 434 may cause the vehicle 1 to execute a lane change on the condition that a periphery confirmation action is detected during the post-proposal confirmation period.

[0047] Execution unit 434 may also determine whether the direction detected during the perimeter confirmation operation corresponds to the direction from the vehicle 1's currently traveling lane toward the lane to which the vehicle 1 is changing lanes. In this case, execution unit 434 causes vehicle 1 to execute a lane change if the angle formed between the direction during the perimeter confirmation operation and the direction from the vehicle 1's currently traveling lane toward the lane to which the vehicle 1 is changing lanes is less than a predetermined angle threshold. Furthermore, execution unit 434 does not cause vehicle 1 to execute a lane change if the direction during the perimeter confirmation operation does not correspond to the direction from the vehicle 1's currently traveling lane toward the lane to which the vehicle 1 is changing lanes.

[0048] Figure 4 It is a schematic diagram explaining the outline of driving control.

[0049] exist Figure 4 The proposal unit 433 notifies the lane change proposal at time t2. Time t1 is included in the pre-confirmation period P1 before time t2, and time t3 is included in the post-proposal confirmation period P2 after time t2.

[0050] At time t1, vehicle 1 and other vehicle 100 are traveling in lane L1. At time t1, driver D of vehicle 1 directs his gaze in direction D1 from lane L1 toward adjacent lane L2. Detection unit 431 detects the driver's perimeter checking action during pre-check period P1.

[0051] At time t2, vehicle 1 and other vehicle 100 are traveling in lane L1. At time t2, driver D of vehicle 1 directs his gaze in direction D2 toward the front of lane L1. Detection unit 431 does not detect the driver's surrounding confirmation action during post-proposal confirmation period P2.

[0052] Since the execution unit 434 detects the periphery confirmation action during the pre-confirmation period P1 , it causes the vehicle 1 to execute a lane change and travel to the future position 1 ′ regardless of whether the periphery confirmation action is detected during the post-proposal confirmation period.

[0053] Figure 52 is a flowchart of the first travel control process. The travel control device 4 repeatedly executes the first travel control process at predetermined time intervals (for example, every 1 / 10 second) while the vehicle 1 is traveling.

[0054] First, the detection unit 431 of the travel control device 4 detects a periphery confirmation action from a facial image generated by the driver monitoring camera 3 (step S11 ).

[0055] If the surrounding confirmation action is detected (step S11: Y), the recording unit 432 of the driving control device 4 records the time when the surrounding confirmation action was detected (step S12), and the first driving control process ends. If the surrounding confirmation action is not detected (step S11: N), the detection unit 431 of the driving control device 4 does not record the time when the surrounding confirmation action was detected, and the first driving control process ends.

[0056] Figure 6 2 is a flowchart of the second travel control process. The travel control device 4 repeatedly executes the second travel control process at predetermined time intervals (for example, every 1 / 10 second) while the vehicle 1 is traveling under automatic driving control.

[0057] The suggestion unit 433 of the travel control device 4 determines whether the situation requires a lane change (step S21 ). If it is determined that the situation does not require a lane change (step S21 : N), the second travel control process ends.

[0058] When it is determined that the situation requires a lane change (step S21 : Y), the proposal unit 433 notifies a lane change proposal (step S22 ).

[0059] Next, execution unit 434 determines whether a surrounding area confirmation action was detected during the pre-confirmation period (step S23). If it is determined that no surrounding area confirmation action was detected during the pre-confirmation period (step S23: N), execution unit 434 determines whether a surrounding area confirmation action was detected during the post-proposal confirmation period (step S24). If it is determined that no surrounding area confirmation action was detected during the post-proposal confirmation period (step S24: N), the second driving control process ends.

[0060] If it is determined that the surrounding confirmation action is detected during the pre-confirmation period (step S23: Y) or if it is determined that the surrounding confirmation action is detected during the post-proposal confirmation period (step S24: Y), the execution unit 434 executes the lane change (step S25) and ends the second driving control process.

[0061] If it is determined that a perimeter confirmation action is detected during the pre-confirmation period (step S23: Y), the execution unit 434 may notify the driver of the execution of the lane change before executing the lane change (step S25), regardless of whether a perimeter confirmation action is detected during the post-proposal confirmation period.

[0062] By executing the first and second travel control processes in this manner, the travel control device 4 can control the travel of the vehicle without causing any trouble to the driver.

[0063] According to a modified example, execution unit 434 detects changes in surrounding conditions, such as those caused by the detection of a new vehicle after a lane change proposal, based on surrounding images received from perimeter camera 2. If changes in surrounding conditions are detected after a lane change proposal, execution unit 434 does not execute the lane change, even if perimeter confirmation actions were detected during the pre-confirmation period. In this case, proposal unit 433 may notify the driver to perform perimeter confirmation actions.

[0064] According to a different variation, when notifying a lane change proposal under a condition where safety does not change with the passage of time, the proposal unit 433 sets the pre-confirmation period to be longer than when notifying a lane change proposal under a condition where safety does change with the passage of time. For example, a lane change under a condition where safety does not change with the passage of time is a lane change to a side lane where there is no possibility of a following vehicle approaching. For example, a lane change under a condition where safety does change with the passage of time is a lane change to a passing lane where a following vehicle may approach.

[0065] Furthermore, according to another variation, the proposal unit 433 detects the number of other vehicles in the surrounding area based on the surrounding images received from the surrounding camera 2. When the proposal unit 433 issues a lane change proposal when the number of other vehicles is large, the prior confirmation period is set shorter than when the proposal is issued when the number of other vehicles is small.

[0066] It should be understood that those skilled in the art can add various changes, substitutions, and modifications without departing from the spirit and scope of the present invention.

Claims

1. A driving control device comprising: a detection unit that detects, from an image showing a face of a driver of a vehicle, a periphery confirmation action of the driver confirming the periphery of the vehicle; a recording unit for recording a time when the peripheral confirmation action is detected; a proposing unit for notifying the driver of a lane change proposal to change the lane in which the vehicle is traveling to an adjacent lane; as well as The execution unit causes the vehicle to execute the lane change when the surrounding confirmation action is detected during a pre-confirmation period before the time of notifying the lane change proposal, regardless of whether the surrounding confirmation action is detected during a post-proposal confirmation period after the time of notifying the lane change proposal.

2. The travel control device according to claim 1, wherein: When the periphery confirmation action is not detected during the pre-confirmation period, the execution unit causes the vehicle to execute the lane change on the condition that the periphery confirmation action is detected during the post-proposal confirmation period.

3. The travel control device according to claim 1 or 2, wherein: The detection unit detects a direction of the surroundings of the vehicle confirmed by the driver in the surrounding confirmation action. The recording unit further records the direction in the surrounding confirmation action. The execution unit causes the vehicle to execute a lane change when a periphery confirmation action is detected during the pre-confirmation period in which the direction recorded in the periphery confirmation action corresponds to a direction from the traveling lane toward a lane to be changed.

4. The travel control device according to claim 1 or 2, wherein: When the periphery confirmation action is detected during the pre-confirmation period, the suggestion unit notifies the driver that a lane change is to be executed regardless of whether the periphery confirmation action is detected during the post-proposition confirmation period.

5. The travel control device according to claim 3, wherein: When the periphery confirmation action is detected during the pre-confirmation period, the suggestion unit notifies the driver that a lane change is to be executed regardless of whether the periphery confirmation action is detected during the post-proposition confirmation period.

6. A driving control method comprising: detecting, from an image showing a face of a driver of a vehicle, a periphery confirmation action of the driver confirming the periphery of the vehicle, Record the time when the peripheral confirmation action is detected, a lane change proposal notifying the driver to change the lane in which the vehicle is traveling to an adjacent lane, When the surrounding confirmation action is detected during the pre-confirmation period before the time of notifying the lane change proposal, the vehicle is caused to execute the lane change regardless of whether the surrounding confirmation action is detected during the post-proposal confirmation period after the time of notifying the lane change proposal.

Images