Travel control device, travel control method, and travel control program

Abstract

To provide a travel controller which can perform correct travel propriety determination in a case where travel propriety determination may be difficult depending on a controller.SOLUTION: A CPU suspends travel to temporarily stop a vehicle when a condition in which travel with automatic driving is possible is not satisfied in such a case where there is an obstacle on the front side in the travel direction of an own vehicle. The CPU operates a head-up display to display a marking MK that surrounds an object being a cause of non-satisfaction of the travel possible condition when a prescribed time elapses after the temporary stop. The CPU monitors the sight line input of a driver, determines that the driver have no intention of traveling when the sight line is oriented to the marking MK, but determines that the travel is possible and starts the vehicle when the sight line is oriented to a traffic lane on which travel is possible.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a travel control device, a travel control method, and a travel control program.

Background Art

[0002] Patent Document 1 below describes a travel control device that detects a moving object that may approach the host vehicle when the host vehicle approaches a crosswalk. Specifically, this device detects the road configuration approaching the target crosswalk, estimates the movement route of a moving object crossing the target crosswalk based on the road configuration, and detects a moving object within the area including the estimated movement route as a detection area.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the case of the above device, for example, it is not possible to determine the walking intention of a person standing still in front of a crosswalk, such as a schoolchild guardian, so there is a risk that unnecessary stopping will continue.

Means for Solving the Problems

[0005] Hereinafter, means for solving the above problems and their effects will be described. An operation acquisition process (S66) for acquiring the operation of the driver based on the image data of the driver of the vehicle, a travelability information acquisition process (S62) for acquiring information indicating that the vehicle cannot be driven automatically, and when acquiring information indicating that the vehicle cannot be driven, an inquiry process (S64) for operating a human interface to inquire the driver whether to instruct the vehicle to travel, a determination process (S70, S82) for determining whether the driver instructs the vehicle to travel based on the operation acquired by the operation acquisition process according to the inquiry process, and an automatic driving permission process (S72, S86) for operating the drive system of the vehicle to make the vehicle travel when it is determined by the determination process that the driver instructs the vehicle to travel. The running control device (30) executes these processes.

[0006] In the above configuration, when acquiring information indicating that the vehicle cannot be driven automatically, the human interface is operated to inquire the driver whether to instruct the vehicle to travel. Then, based on the result of the inquiry, it is determined whether the driver instructs the vehicle to travel according to the operation of the driver based on the image data. If it is determined that the driver instructs the vehicle to travel, the vehicle is permitted to travel automatically. In this way, by inquiring the driver's intention, when it is difficult for the control device to determine whether the vehicle can travel, the driver's judgment can assist the control device in the process of determining whether the vehicle can travel. Moreover, since it is determined whether the driver instructs the vehicle to travel according to the operation based on the image data, the driver does not need to convey the intention by operating a manual input device, and the intention conveyance can be simplified.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Mode for Carrying Out the Invention

[0008] <First Embodiment> Hereinafter, the first embodiment will be described with reference to the drawings. FIG. 1 shows an apparatus mounted on a vehicle VC in the present embodiment. As shown in FIG. 1, the optical sensor 12 irradiates laser light such as near-infrared light. Further, based on receiving the reflected light of the laser light, the optical sensor 12 generates ranging point data indicating a distance variable, which is a variable indicating the distance between the object that reflected the laser light and the vehicle, a direction variable, which is a variable indicating the irradiation direction of the laser light, and an intensity variable, which is a variable indicating the reflection intensity of the reflected object. This can be realized, for example, by the TOF (Time of Flight) method. However, it is not limited to the TOF method, and ranging point data may be generated by the FMCW (Frequency Modulated Continuous Wave) method. In that case, a speed variable, which is a variable indicating the relative speed with respect to the object that reflected the laser light, can be included in the ranging point data.

[0009] The optical sensor 12 periodically scans the irradiation direction of the laser light in the horizontal and vertical directions and outputs ranging point group data Drpc, which is a set of the obtained ranging point data. The LIDAR ECU 10 executes recognition processing of the object that reflected the laser light based on the ranging point group data Drpc. The recognition processing may include, for example, clustering processing of the ranging point group data Drpc and extracting feature amounts of a set of ranging point data identified as one object by the clustering processing, and inputting the extracted feature amounts into an identification model that determines whether it is a predetermined object. Alternatively, it may be a process of directly inputting the ranging point group data Drpc into a deep learning model to recognize an object.

[0010] The external camera 22 outputs external image data Dio, which is image data outside the vehicle VC. The image ECU 20 executes recognition processing of objects around the vehicle based on the external image data Dio, which is data regarding the image captured by the external camera 22.

[0011] The ADAS ECU 30 executes processing for controlling the running of the vehicle VC. When executing the processing for controlling the running, the ADAS ECU 30 receives the recognition results by the LIDAR ECU 10 and the image ECU 20 respectively via the local network 40. Also, when executing the processing for controlling the running, the ADAS ECU 30 refers to the position data Dgps of the global positioning system (GPS 42) and the map data 44 via the local network 40. Further, when executing the processing for controlling the running, the ADAS ECU 30 refers to the in-vehicle image data Dii, which is the image data inside the vehicle VC by the in-vehicle camera 46. Note that the in-vehicle camera 46 is mainly a device for imaging the driver. Also, when executing the processing for controlling the running, the ADAS ECU 30 operates the head-up display (HUD 48), the speaker 49, the drive system 50, the braking system 51, the steering system 52, and the external display device 54 while referring to the operation state of the input device 47. Here, the input device 47 is a means for manual operation by the driver to convey intentions. The HUD 48 is a device for providing visual information to the driver, and the external display device 54 is a device for providing visual information outside the vehicle. Also, the drive system 50 may be configured by at least one of, for example, an internal combustion engine and a rotary electric machine.

[0012] Specifically, ADASECU30 includes a CPU 32, a storage device 34, and a peripheral circuit 36, which are communicable via a local network 38. Here, the peripheral circuit 36 includes a circuit that generates a clock signal for defining internal operations, a power supply circuit, a reset circuit, and the like. By executing a program stored in the storage device 34 by the CPU 32, ADASECU30 executes an automatic driving process and a driver monitoring process, thereby constituting a so-called driver monitoring system (DMS). ADASECU30 executes each of the above processes by the CPU 32 executing a travel control program Pad and a DMS program Pdms stored in the storage device 34. In this embodiment, the DMS is constituted by ADASECU30 and an in-vehicle camera 46.

[0013] FIG. 2 shows a part of the processing procedure of the automatic driving process according to this embodiment. The process shown in FIG. 2 is realized by the CPU 32 repeatedly executing a travel control program Pad stored in the storage device 34, for example, at a predetermined cycle. Hereinafter, the step numbers of each process are represented by numbers with "S" added at the beginning.

[0014] In the series of processes shown in FIG. 2, the CPU 32 first determines whether there is an input of a destination by the driver manually operating the input device 47 (S10). Then, when the CPU 32 determines that there is an input (S10: YES), it sets a route from the current location to the destination (S12). That is, the CPU 32 sets a travel route based on the link information of the lanes to the destination included in the map data 44. When there are a plurality of lanes in the same direction, the CPU 32 selects a more desirable lane as the target route. For example, when the CPU 32 makes a left turn at the next intersection, it selects the left lane as part of the travel route.

[0015] Next, the CPU 32 determines whether the operating state of the input device 47 by the driver is in a state where automatic driving is selected (S14). And when the CPU 32 determines that automatic driving is selected (S14: YES), it substitutes "1" into the automatic driving mode flag F (S16).

[0016] Note that when the process of S16 is completed, or when a negative determination is made in the processes of S10 and S14, the CPU 32 temporarily ends the series of processes shown in FIG. 2. FIG. 3 shows a part of the procedures of the driver monitoring process and the automatic driving process. The process shown in FIG. 3 is realized by the CPU 32 repeatedly executing the driving control program Pad and the DMS program Pdms stored in the storage device 34, for example, at a predetermined cycle. Note that the cycle in which the process of FIG. 3 is executed is shorter than the cycle in which the process of FIG. 2 is executed.

[0017] In the series of processes shown in FIG. 3, first, the CPU 32 determines whether the automatic driving mode flag F is "1" (S20). When the CPU 32 determines that it is "1" (S20: YES), it acquires the face information of the driver based on the in-vehicle image data Dii output by the in-vehicle camera 46 (S22). Then, the CPU 32 determines whether the driver is making a side glance (S24). When the CPU 32 determines that the driver is making a side glance (S24: YES), it increments the counter C (S26). Next, the CPU 32 determines whether the counter C is equal to or greater than a predetermined value Cth (S28). And when the CPU 32 determines that it is equal to or greater than the predetermined value Cth (S28: YES), it operates the speaker 49 shown in FIG. 1 to emit a warning sound, thereby alerting the driver to be prepared to drive the vehicle manually at any time (S30).

[0018] On the other hand, when the CPU 32 makes a negative determination in the process of S24, it initializes the counter C (S32). When the CPU 32 completes the processes of S30 and S32, or makes a negative determination in the process of S28, it determines whether the driver is in a state where driving is impossible, such as a state where the driver is unconscious (S34). When the CPU 32 determines that the driver is in a state where driving is impossible (S34: YES), it operates the braking system 51 to decelerate the vehicle and move it sideways to stop on the road shoulder or the like (S36).

[0019] Incidentally, when the process of S36 is completed, or when a negative determination is made in the processes of S20 and S34, the CPU 32 temporarily ends the series of processes shown in FIG. 3. The processes of S20 to S34 are processes realized by the CPU 32 executing the DMS program Pdms, and the process of S36 is a process realized by the CPU 32 executing the driving control program Pad.

[0020] FIG. 4 shows a part of the procedure of the automatic driving process. The process shown in FIG. 4 is realized by the CPU 32 repeatedly executing the driving control program Pad stored in the storage device 34, for example, at a predetermined cycle. The cycle in which the process of FIG. 4 is executed is shorter than the cycle in which the process of FIG. 2 is executed.

[0021] In the series of processes shown in FIG. 4, first, the CPU 32 determines whether the automatic driving mode flag F is "1" (S40). When the CPU 32 determines that the automatic driving mode flag F is "1" (S40: YES), it refers to the map data 44 regarding the vicinity of the current location based on the position data Dgps (S42). Next, the CPU 32 acquires the outputs of the LIDAR ECU 10 and the image ECU 20 (S44).

[0022] The CPU 32 determines whether the vehicle has shifted from a state where it can travel by autonomous driving to a state where it cannot travel based on the map data and the above output (S46). This process is a process in which the CPU 32 determines whether the situation of the vehicle has shifted from a state that satisfies the travelable condition defined by the travelable condition data Drp stored in the storage device 34 shown in FIG. 1 to a state that does not satisfy the condition based on the map data and the above output. The travelable condition is a condition under which the vehicle can be driven by autonomous driving. The travelable condition is, for example, that there are no obstacles in the traveling direction of the vehicle. Specifically, when there is a traffic signal, in addition to there being no obstacles in the traveling direction of the vehicle, a condition that it is a green signal is included. Also, when there is a crosswalk and there is no traffic signal, in addition to there being no obstacles in the traveling direction of the vehicle, a condition that there are no people near the crosswalk is included.

[0023] When the CPU 32 determines that it has shifted to an impossible state (S46: YES), it executes a stop process by operating the braking system 51 (S48). Then, the CPU 32 identifies the object that caused the impossible travel and stores it in the storage device 34 (S50). For example, when an obstacle is detected in the traveling direction based on the output of the LIDAR ECU 10 and the output of the image ECU 20, the CPU 32 stores the position coordinates of the detected obstacle in the storage device 34.

[0024] Note that when the CPU 32 completes the process of S50 or makes a negative determination in the processes of S40 and S46, the series of processes shown in FIG. 4 is temporarily terminated. FIG. 5 shows the procedure of a part of the autonomous driving process. The process shown in FIG. 5 is realized by the CPU 32 repeatedly executing the travel control program Pad stored in the storage device 34, for example, at a predetermined cycle. Note that the cycle at which the process of FIG. 5 is executed is shorter than the cycle at which the process of FIG. 2 is executed.

[0025] In the series of processes shown in FIG. 5, the CPU 32 first determines whether or not the automatic driving mode flag F is "1" (S60). If the CPU 32 determines that it is "1" (S60: YES), after the temporary stop by the process of S48, it determines whether or not a certain period of time has elapsed (S62). Here, the certain period of time may be set based on, for example, the time when it is assumed that the vehicle will not resume running even if waiting any longer because the driving enable condition will not return to a state where it is satisfied.

[0026] If the CPU 32 determines that the certain period of time has elapsed (S62: YES), by operating the HUD 48, it performs a marking display so as to surround the periphery of the object that became the factor for the impossible driving, which was memorized in the process of S50 (S64).

[0027] FIG. 6(a) exemplifies the marking MK in the case where the fact that the vehicle VC(2) is stopped in front of the own vehicle is the factor for the impossible driving. The situation shown in FIG. 6(a) is the situation shown in FIG. 6(b). That is, when the vehicle VC(1) which is the own vehicle is running in the leftmost lane TR1 among the lanes TR1, TR2, and TR3 in the same traveling direction, the vehicle VC(2) is stopped in front of it, so that the driving enable condition is not satisfied and the vehicle VC(1) has stopped. Here, it is assumed that the vehicle VC(1) will turn left at the next intersection, and therefore, the lane TR1 is selected as the traveling route. Also, the vehicle VC(2) is stopped considerably behind the intersection, and there are no other vehicles in the area up to the intersection in front of it.

[0028] Returning to FIG. 5, the CPU 32 monitors the input of the driver's line of sight based on the in-vehicle image data Dii captured by the in-vehicle camera 46 (S66). Here, it is assumed that the driver has been made aware in advance that it is possible to convey the driver's intention by the line of sight. Specifically, it is well known in advance that when the line of sight is directed at an object marked with a line of sight, it indicates that there is no intention to start, and when the line of sight is directed at a lane in which the vehicle VC(1) can travel, it indicates that there is an intention to start. That is, in the case of the example shown in FIG. 6, when the driver directs the line of sight at the lane TR2, it indicates that it is the driver's intention to drive while avoiding the vehicle VC(2).

[0029] Then, the CPU 32 determines whether the situation in which the vehicle VC(1) is placed matches the conditions under which normal start is possible as defined in the normal start definition data Ddn stored in the storage device 34 shown in FIG. 1 (S68). The conditions under which normal start is possible are set to conditions that are assumed to have an extremely low possibility of falling into a dangerous situation during the transition from normal start to normal driving when the driver indicates an intention to start. For example, in the case of the example shown in FIG. 6, when starting while changing lanes to the lane TR2, if there is no vehicle traveling in the lane TR2 or no vehicle changing lanes to the lane TR2, it is considered that there is no particular problem. Therefore, in the present embodiment, in the case of changing lanes, the condition that there is no vehicle or the like traveling in the vicinity of the vehicle VC(1) and there is no obstacle in the lane to be changed is included in the conditions under which normal start is possible.

[0030] When the CPU 32 determines that the conditions for normal start are met (S68: YES), it determines whether the driver has instructed the vehicle to travel by the line of sight input, in other words, whether the start has been permitted (S70). Then, when the CPU 32 determines that the vehicle has been instructed to travel (S70: YES), it starts the vehicle by operating the drive system 50 and shifts to normal driving (S72).

[0031] For example, in the example shown in FIG. 6, when the driver looks at lane TR2, the CPU 32 starts the vehicle VC(1) by operating the drive system 50 and changes the lane in which the vehicle is traveling to lane TR2 by operating the steering system 52.

[0032] Also, in the case of the example shown in FIG. 7, after a marking MK is made on the red flag FR held by the person HM, when the driver looks at the right lane, the CPU 32 starts the vehicle VC(1) and changes the lane to the right lane.

[0033] Also, in the case of the example shown in FIG. 8, after a marking MK is made around the falling object FO in the traveling direction, when the driver looks at the left lane where there is no falling object FO, the CPU 32 starts the vehicle VC(1) and changes the lane to the left lane.

[0034] Returning to FIG. 5, when the CPU 32 makes a negative determination in the process of S68, it proceeds to the process of S74. Note that when the CPU 32 completes the processes of S72 and S74, or when it makes a negative determination in the processes of S60, S62, and S70, it temporarily ends the series of processes shown in FIG. 5.

[0035] FIG. 9 shows an example of the process of S74. In the series of processes shown in FIG. 9, the CPU 32 first determines whether there is a pedestrian with an unclear intention to cross in front of a crosswalk without a traffic signal (S80). Here, based on the output of the LIDAR ECU 10 and the output of the image ECU 20, when a person is recognized in front of the crosswalk, the CPU 32 may determine that there is a pedestrian with an unclear intention to cross. When the CPU 32 determines that there is a pedestrian with an unclear intention to cross (S80: YES), it operates the external display device 54 shown in FIG. 1 to display visual information indicating that it is waiting to cross (S81).

[0036] This scene will be described with reference to FIGS. 10 and 11. FIG. 10 shows an example of the display of the HUD 48 in the situation where the process of S80 is performed. In the example shown in FIG. 10, two persons are standing in front of a crosswalk, and markings MK are made so as to surround them.

[0037] FIG. 11 exemplifies the process of S81. In the example shown in FIG. 11, an example of displaying "Waiting for crosswalk" on the out-vehicle display device 54 is shown. Returning to FIG. 9, the CPU 32 determines whether or not the driver has instructed to drive based on the monitoring result of the line-of-sight input by the process of S66 (S82). When the CPU 32 determines that the driver has instructed to drive (S82: YES), it operates the out-vehicle display device 54 shown in FIG. 1 to display an indication of the intention to start (S84).

[0038] Then, the CPU 32 starts by operating the drive system 50 and performs a very slow driving (S86). Here, the very slow driving is, for example, driving at a speed of "10 km / h" or less, and preferably driving at a speed of "5 km / h" or less. After starting, the CPU 32 checks whether or not a pedestrian whose intention was unclear crosses the crosswalk based on the output of the LIDAR ECU 10 and the output of the image ECU 20 (S88). When the CPU 32 determines that the pedestrian does not cross (S88: YES), it shifts to normal driving (S90). On the other hand, when the CPU 32 determines that the pedestrian crosses (S88: NO), it operates the braking system 51 to restart the vehicle (S92).

[0039] Note that when the CPU 32 completes the processes of S90 and S92, or when a negative determination is made in the processes of S80 and S82, the series of processes shown in FIG. 9 is temporarily terminated. FIG. 12 shows another example of the process of S74. In FIG. 12, for the processes corresponding to the processes exemplified in FIG. 9, the same step numbers are given for convenience and the description thereof is omitted.

[0040] In the series of processes shown in FIG. 12, the CPU 32 first determines whether or not there is a blind spot at an intersection without a traffic signal (S100). FIG. 13 shows an example of a blind spot.

[0041] In the example shown in FIG. 13, an example is shown where a marking MK is made on an object OS because the right side of the lanes intersecting the lane being traveled is blocked by the object OS. Here, at an intersection without a traffic signal, not only must there be no obstacle in the forward direction of travel, but also there must be no vehicle traveling from the intersecting lane into the lane being traveled for it to be a travelable condition. Therefore, if the presence or absence of a vehicle traveling from the right side into the lane being traveled cannot be confirmed by the object OS, it does not satisfy the travelable condition, which can be a factor for a positive determination in the process of S46 in FIG. 4.

[0042] Returning to FIG. 12, when the CPU 32 determines that there is a blind spot (S100: YES), it determines whether or not the driver has been instructed to start based on the monitoring result of the line-of-sight input by the process of S66 (S82). When the CPU 32 determines that the start has been instructed (S82: YES), it proceeds to the process of S86. Next, the CPU 32 determines whether or not the blind spot has been eliminated (S102). Then, when the CPU 32 determines that the blind spot has been eliminated (S102: YES), it determines whether or not the safety check such as the presence or absence of a vehicle in the area where the field of view has opened has been completed (S104). Then, when the CPU 32 can perform the safety check (S104: YES), it proceeds to the process of S90, while when the safety check cannot be performed (S104: NO), it proceeds to the process of S92.

[0043] Note that when the processes of S90 and S92 are completed, or when a negative determination is made in the processes of S100, S82, and S102, the CPU 32 temporarily ends the series of processes shown in FIG. 12.

[0044] Here, the operations and effects of the present embodiment will be described. When the CPU 32 fails to meet the travelable conditions in the autonomous driving mode, it executes a stop process. After that, when the stop time exceeds a certain period, it operates the HUD 48 to mark and display the object that caused the stop, thereby asking the driver about the feasibility of starting. On the other hand, when the driver looks towards a travelable lane, the vehicle is started on the assumption that the driver has instructed the vehicle to travel. This can assist the ADASECU 30 in determining the feasibility of travel based on the driver's judgment when it is difficult for the ADASECU 30 to make such a determination.

[0045] According to the present embodiment described above, the following operations and effects can be obtained. (1) The inquiry about whether to instruct travel is made by marking and displaying the object that caused the stop. This can inform the driver of the reason for the vehicle's stop, so the driver can more accurately judge the feasibility of travel compared to the case of asking whether to instruct travel without informing the reason for the stop.

[0046] (2) It is determined whether the driver instructs travel based on the driver's line of sight. This can simplify the tasks that the driver has to perform compared to the case of using the input device 47 that is the target of manual operation. That is, when using the input device 47 that is the target of manual operation, after visually confirming safety, the input device 47 that is the target of manual operation is operated to inform that travel is instructed. In contrast, in this embodiment, after confirming the marking MK, the line of sight is moved to instruct travel, so the communication of intent can be simplified.

[0047] (3) When the conditions defined by the normal start definition data Ddn are not met, the vehicle travels at the slowest speed until safety can be confirmed after starting. This can enhance safety compared to the case of immediately returning to normal travel after starting.

[0048] When temporarily stopped at a crosswalk without a traffic signal, the vehicle exterior display device 54 displayed information indicating that the vehicle was waiting to cross. This enables a person in front of the crosswalk to understand the vehicle's intention.

[0049] When starting from a temporary stop at a crosswalk without a traffic signal, prior to starting, the vehicle exterior display device 54 displayed information indicating that the vehicle was about to drive. This can alert pedestrians and thus enhance safety.

[0050] The inquiry process was realized by diverting the functions of an existing driver monitoring system. This enables the execution of the inquiry process without adding new hardware means.

[0051] <Second Embodiment> Hereinafter, the second embodiment will be described with reference to the drawings, focusing on the differences from the first embodiment.

[0052] FIG. 14 shows the device mounted on the vehicle VC in this embodiment. In FIG. 14, members corresponding to the members shown in FIG. 1 are given the same reference numerals for convenience. As shown in FIG. 14, the vehicle VC according to this embodiment includes a DMSECU 60 independently of the ADASECU 30. The DMSECU 60 includes a CPU 62, a storage device 64, and a peripheral circuit 66, and they are communicable with each other via a local network 68. The storage device 64 stores a DMS program Pdms. Then, the CPU 62 executes the DMS program Pdms stored in the storage device 64 to execute the processes of S20 to S34 shown in FIG. 3.

[0053] The memory device 34 of ADASECU30 does not store the DMS program Pdms. Also, the memory device 34 does not store the normal start definition data Ddn. The memory device 34 stores the driving prohibition condition data Drf and the status list data Dl. The driving prohibition condition data Drf is data that defines the driving prohibition conditions that are the conditions for prohibiting the vehicle VC from being driven by automatic driving. The driving prohibition conditions are, for example, conditions such as the signal being red in front of an intersection.

[0054] When the CPU 32 does not satisfy both the driving enable conditions and the driving prohibition conditions, it asks the driver whether to instruct the vehicle VC to drive. To execute this process, the CPU 32 uses the status list data Dl. The status list data Dl defines the status of each object considered in determining whether it is possible to drive the vehicle VC. The CPU 32 determines whether the vehicle can ultimately be driven based on the status of each object.

[0055] Hereinafter, among the automatic driving processes, the process of executing the stop process based on the status, the process of updating the status based on the line of sight, and the process regarding the resumption of automatic driving will be described in this order.

[0056] "Process of Executing the Stop Process Based on the Status" FIG. 15 shows some steps of the automatic driving process. The process shown in FIG. 15 is realized by the CPU 32 repeatedly executing the driving control program Pad stored in the memory device 34, for example, at a predetermined cycle. Among the processes shown in FIG. 15, for the processes corresponding to the processes shown in FIG. 4, for convenience, the same step numbers are assigned.

[0057] In the series of processes shown in FIG. 15, when the CPU 32 completes the process of S44, based on the outputs of the LIDAR ECU 10 and the image ECU 20, it extracts the target objects OBJ(1) to OBJ(n) and updates the status list data Dl (S101). That is, the CPU 32 sets the target objects OBJ whose status is defined by the status list data Dl to the target objects OBJ(1) to OBJ(n) extracted by the process of S101.

[0058] The target object is an object considered in determining whether the vehicle VC can be driven. The CPU 32 determines for each of the target objects OBJ(1) to OBJ(n) whether the vehicle VC cannot be driven due to that object. The target object OBJ includes traffic indicators such as traffic lights. Also, the target object OBJ includes an object located in any one of three regions: the lane on which the vehicle VC travels, the lane adjacent to the same lane, and the vicinity of the same lane. Note that there may be no target object OBJ. This is defined by "n = 0" above.

[0059] Next, the CPU 32 determines whether all of the target objects OBJ(1) to OBJ(n) extracted by S101 have been selected by the process of S105 described later (S103). Here, the CPU 32 also makes an affirmative determination even when there is no target object OBJ.

[0060] When the CPU 32 determines that there is a target object OBJ that has not yet been selected (S103: NO), it selects one target object OBJ(i) from the target objects OBJ(1) to OBJ(n) extracted in S101 (S105). Then, the CPU 32 determines whether the target object OBJ(i) satisfies the travel possible condition (S106). For example, when the target object OBJ(i) is displaced in a direction away from the travel lane of the vehicle VC, the CPU 32 determines that the travel possible condition is satisfied. Furthermore, even when the target object OBJ(i) is displaced toward the travel lane of the vehicle VC, the CPU 32 determines that the travel possible condition is satisfied when it is separated from the vehicle VC by a predetermined distance or more. When the CPU 32 determines that the travel possible condition is satisfied (S106: YES), it sets the status of the target object OBJ(i) defined by the status list data Dl to "determinable" (S108).

[0061] On the other hand, when the CPU 32 determines that the driving enabling condition is not satisfied (S106: NO), it determines whether or not the driving prohibition condition defined by the driving prohibition condition data Drf is satisfied (S110). When the CPU 32 determines that the driving prohibition condition is satisfied (S110: YES), it proceeds to the process of S108. Note that, for example, when there is a stopped vehicle ahead of the vehicle VC in the driving lane, it is desirable not to include this in the driving prohibition condition. By not including it, it becomes possible for the driver to instruct driving by the process described below in the situation illustrated in FIG. 6.

[0062] As described above, when the driving enabling condition or the driving prohibition condition is satisfied, the CPU 32 sets the status to "determinable" because the ADASECU 30 can determine on its own whether or not to drive by autonomous driving.

[0063] On the other hand, when it is determined that the travel prohibition condition is not satisfied (S110: NO), the CPU 32 determines whether or not the logical product of the following condition (A) and condition (B) is true (S112). Condition (A): This condition means that the status of the target object OBJ(i) is set to "permitted to drive" or "object gaze" by the process described below.

[0064] Condition (a): A condition that the target object OBJ(i) has not been displaced since the previous execution timing of the series of processes in FIG. 15. When the CPU 32 determines that the above logical product is false (S112: NO), it sets the status to "indeterminable" (S114).

[0065] When the processes of S108 and S114 are completed and when an affirmative determination is made in the process of S112, the CPU 32 returns to the process of S103. On the other hand, when an affirmative determination is made in the process of S103, the CPU 32 determines whether or not the vehicle has shifted from a state where running by automatic driving is possible to a state where it is impossible (S46a). Here, the state where running is possible is a state in which each of the statuses of the target object OBJ is "determinable" and satisfies the driving permission condition or is "driving permitted". On the other hand, the impossible state shifted from the state where running is possible is a state in which at least one of the target objects OBJ has a status of "determinable" and satisfies the driving prohibition condition and a status of "indeterminable".

[0066] When the CPU 32 determines that it has shifted to an impossible state (S46a: YES), it proceeds to the process of S48. When the process of S48 is completed and when a negative determination is made in the processes of S40 and S46a, the CPU 32 temporarily ends the series of processes shown in FIG. 15.

[0067] "Process of updating status based on line of sight" FIG. 16 shows a part of the procedure of the automatic driving process. The process shown in FIG. 16 is realized by the CPU 32 repeatedly executing the driving control program Pad stored in the storage device 34, for example, at a predetermined cycle. It is desirable that the cycle in which the process of FIG. 16 is executed is a cycle equal to or less than the cycle in which the process of FIG. 15 is executed.

[0068] In the series of processes shown in FIG. 16, the CPU 32 first determines whether the automatic driving mode flag F is "1" (S120). When the CPU 32 determines that it is "1" (S120: YES), it determines whether there is a target object OBJ whose status is "undeterminable" (S122). When there is a target object OBJ whose status is "undeterminable" (S122: YES), the CPU 32 extracts the target object OBJ (S124). Then, the CPU 32 generates a marking group, which is a group for collectively marking one of the target objects OBJ (S126). For example, when there are multiple people standing in front of a crosswalk, the CPU 32 sets the target objects OBJ corresponding to those people as one marking group. Note that a marking group may be composed of one target object OBJ.

[0069] Next, the CPU 32 selects one of the marking groups generated by the process of S126 for marking display (S128). The CPU 32 selects the group with the closest distance from the vehicle VC among the groups that have not yet been marked for display (S130). Then, the CPU 32 receives from the DMS ECU 60 the monitoring result of the driver's line-of-sight input based on the in-vehicle image data Dii (S66a). Then, the CPU 62 determines whether the driver's line of sight is directed at the marking based on the received monitoring result (S132). In other words, the CPU 32 determines whether the driver is looking at the marking display. When the CPU 32 determines that the driver is looking at it (S132: YES), it updates the status of all the target objects OBJ belonging to that group to "object watched" (S134).

[0070] Note that when the CPU 32 completes the process of S134, and when a negative determination is made in the processes of S120, S122, and S132, the series of processes shown in FIG. 16 is temporarily terminated. A part of the procedure of the automatic driving process is shown in Fig. 17. The process shown in Fig. 17 is realized by the CPU 32 repeatedly executing the driving control program Pad stored in the storage device 34, for example, at a predetermined interval. Note that the interval at which the process in Fig. 17 is executed is preferably equal to or shorter than the interval at which the process in Fig. 15 is executed.

[0071] In the series of processes shown in Fig. 17, the CPU 32 first determines whether or not the automatic driving mode flag F is "1" (S140). If the CPU 32 determines that the flag F is "1" (S140: YES), the CPU 32 determines whether or not the logical product of the following conditions (C) and (D) is true (S142).

[0072] Condition (c): The condition is that the status specified by the status list data Dl does not include "undeterminable." Condition (d): The condition is that "object gaze" exists in the statuses defined by the status list data Dl.

[0073] When the CPU 32 determines that the logical product is true (S142: YES), the CPU 32 receives from the DMSECU 60 the monitoring result of the driver's gaze input based on the in-vehicle image data Dii (S66a). Then, based on the received monitoring result, the CPU 62 determines whether or not the driver's gaze is directed toward a drivable lane (S144). In other words, the CPU 32 determines whether or not the driver has instructed the vehicle VC to drive. When the CPU 32 determines that the driver has instructed the vehicle VC to drive (S144: YES), the CPU 32 updates the status of "object gaze" in the status list data Dl to "driving permitted" (S146).

[0074] It should be noted that the CPU 32 temporarily ends the series of processes shown in FIG. 17 when completing the process of S146 or when a negative determination is made in the processes of S140, S142, and S144.

[0075] "Processing for resuming automatic driving" FIG. 18 shows part of the procedures of the automatic driving process. The process shown in FIG. 18 is realized by the CPU 32 repeatedly executing the driving control program Pad stored in the storage device 34, for example, at a predetermined cycle. It is desirable that the cycle in which the process of FIG. 18 is executed is the same as the cycle in which the process of FIG. 15 is executed.

[0076] In the series of processes shown in FIG. 18, the CPU 32 first determines whether or not it is in the speed limit process described later (S150). When the CPU 32 determines that the speed limit process is not being executed (S150: NO), it determines whether or not it is in a temporary stop due to the process of S48 (S152). When the CPU 32 determines that it is in a temporary stop (S152: YES), it determines whether or not the logical product of the following condition (a) and condition (b) is true (S154).

[0077] Condition (a): This is a condition indicating that the driving prohibition condition is not satisfied. In other words, when there is a "determinable" status, this is a condition indicating that the status has been generated by a positive determination in the process of S106.

[0078] Condition (b): This is a condition indicating that the "driving permitted" status exists in the status defined by the status list data Dl. When the CPU 32 determines that the logical product is true (S154: YES), it executes the speed limit process (S156). Here, the speed limit process is a process of limiting the vehicle speed to be equal to or lower than the upper speed Lspd. When the distance L to the target object OBJ whose status is "driving permitted" is large, the CPU 32 sets the upper speed Lspd to a larger value than when the distance L is small.

[0079] Fig. 19 illustrates the relationship between the distance L and the upper limit speed Lspd. In the example shown in Fig. 19, when the distance L is less than or equal to the first distance L1, the upper limit speed Lspd is set to zero, and when the distance L is greater than the first distance L1 and less than or equal to the second distance L2, an example where the upper limit speed Lspd is set to the first limit value V1 is shown. Note that the first distance L1 is desirably a value of 1 m to 1.5 m or less. Also, for example, the second distance L2 may be set to about 1.5 to 2.5 m, and the first limit value V1 may be set to 1 to 1.5 km / h. When the upper limit speed Lspd is greater than zero, the vehicle VC is assumed to start running.

[0080] Returning to Fig. 18, when the CPU 32 determines that it is executing the speed limit process (S150: YES), it determines whether the status defined by the status list data Dl has the status of "travel permission" (S158). When the CPU 32 determines that there is no status of "travel permission" (S158: YES), it ends the speed limit process (S160).

[0081] Note that when the CPU 32 completes the processes of S156 and S160, and when a negative determination is made in the processes of S152, S154, and S158, the series of processes shown in Fig. 18 are temporarily ended.

[0082] Here, the operations and effects of the present embodiment will be described. Fig. 20(a) shows an example where there is a crosswalk in front of the driving lane of the vehicle VC, and four people are standing still in front of it. In the situation of Fig. 20(a), the CPU 32 designates each of the four people located in front of the crosswalk as the target objects OBJ(1) to OBJ(4). Also, the CPU 32 designates the two people located at a position considerably away from the driving lane as the target objects OBJ(5) and OBJ(6).

[0083] Then, as shown in Fig. 20(b), the CPU 32 writes the statuses of those target objects OBJ(1) to OBJ(6) into the status list data Dl. Specifically, the CPU 32 sets the statuses of the target objects OBJ(1) to OBJ(4) as "undeterminable" and the statuses of the target objects OBJ(5) and OBJ(6) as "determinable". That is, the target objects OBJ(5) and OBJ(6) satisfy the travelable conditions, so their statuses are set as "determinable".

[0084] Then, as shown in Fig. 20(c), the CPU 32 marks the target objects OBJ(1) to OBJ(4) as one marking group with the marking MK. Next, the CPU 32 monitors the driver's line of sight. When the CPU 32 determines that the line of sight is focused on the marking MK, as shown in Fig. 20(d), it updates the statuses of the target objects OBJ(1) to OBJ(4) to "object watched".

[0085] When the state shown in Fig. 20(d) is reached, the CPU 32 monitors whether the driver instructs to travel according to the driver's line of sight. That is, in the state shown in Fig. 20(d), there is no "undeterminable" in the status defined by the status list data Dl. Moreover, in the state shown in Fig. 20(d), the statuses that are "determinable" are in a state where it is determined that they are travelable. Therefore, if the driver instructs to travel, it monitors the driver's instruction to make the vehicle VC travel by automatic driving. Then, when the driver instructs to travel, the CPU 32 starts the vehicle VC under the speed limit process. And the CPU 32 continues the vehicle speed limit process for the period when the status is "travel permitted".

[0086] Fig. 21 shows an example of the transition of the upper speed limit Lspd by the vehicle speed limit process. When the vehicle VC starts at time t1 shown in FIG. 21 with a relatively large distance L between the vehicle VC and the target objects OBJ(1) to OBJ(4), the distance L between the vehicle VC and the target objects OBJ(1) to OBJ(4) gradually decreases. Therefore, the CPU 32 gradually reduces the upper limit speed Lspd. And at time t2 when the distance L between the vehicle VC and the target objects OBJ(1) to OBJ(4) becomes the minimum, the upper limit speed Lspd also becomes the minimum. After that, at time t3 when the target objects OBJ(1) to OBJ(4) are behind the vehicle VC, the CPU 32 determines in the process of S106 that the travelable condition is satisfied. And the CPU 32 sets the status of the target objects OBJ(1) to OBJ(4) to "determinable". Thereby, the speed limit process is released and the vehicle shifts to normal automatic driving.

[0087] As described above, according to this embodiment, after confirming that the target object OBJ surrounded by the marking MK has been watched, it is determined whether the driver has instructed the vehicle to travel. Therefore, for the target object OBJ that the CPU 32 cannot determine whether it is possible to travel by itself, after the driver surely recognizes it, it can be determined whether the driver instructs the vehicle to travel.

[0088] According to the embodiment described above, the following operations and effects can be obtained. (7) When the CPU 32 satisfies the travel prohibition condition, it sets the status to "determinable". Thereby, when the CPU 32 can determine that it cannot travel without inquiring the driver, it can avoid marking the target object OBJ. Therefore, it is possible to suppress giving the driver a complicated impression.

[0089] (8) The CPU 32 generates a marking group and displays the marking MK so as to surround the group collectively. Thereby, in the range where the driver can watch collectively at one time, the marking MK can be made simpler compared with the case where a plurality of markings MK are displayed.

[0090] (9) When there are a plurality of marking groups, the CPU 32 sequentially enclosed these groups with the marking MK. This can clearly convey to the driver the object to be focused on.

[0091] <Other Embodiments> Note that this embodiment can be implemented with the following modifications. This embodiment and the following modification examples can be implemented in combination with each other within a technically non - conflicting range.

[0092] "Regarding the operation acquisition process" · The operations acquired as the driver's operations in response to the inquiry process are not limited to eye movements, that is, line - of - sight. For example, a pointing operation or the like may be acceptable.

[0093] "Regarding the determination process of whether the vehicle can run by autonomous driving" · In the above - mentioned first embodiment, when the travelable condition defined by the travelable condition data Drp is not satisfied, it is determined that the vehicle cannot travel by autonomous driving, but it is not limited to this. For example, conditions for which travel by autonomous driving is impossible may be defined, and when the case corresponds to those conditions, it may be determined that the vehicle cannot travel by autonomous driving.

[0094] "Regarding the travelability information acquisition process" · In the above - mentioned first embodiment, when the vehicle is stopped for a certain period of time in the autonomous driving mode, it is assumed that information indicating that it cannot be determined that the vehicle can travel by autonomous driving, which triggers the inquiry process, is acquired, but it is not limited to this. For example, data defining conditions for prohibiting travel regardless of the driver's instruction to travel is stored in the storage device 34, and when the vehicle is stopped for a certain period of time in the autonomous driving mode and the factor that resulted in an affirmative determination in the process of S46 is due to a factor that does not satisfy the same conditions, it may be assumed that the above - mentioned information that triggers the inquiry process is acquired. In that case, rapid resumption of travel becomes possible. Incidentally, as the above - mentioned conditions, for example, there is a condition that the signal is red in front of an intersection.

[0095] Note that it is not essential for information indicating that it is impossible to determine that the vehicle can travel by autonomous driving to include information indicating that the vehicle has been stopped for a certain period of time in the autonomous driving mode. For example, the running permission information acquisition process for acquiring information indicating that it is impossible to determine that the vehicle can travel by autonomous driving may be the process of S46.

[0096] "Regarding the Prohibition Judgment Process" ·In the above second embodiment, it was determined whether or not the driving prohibition conditions were satisfied, but this may be deleted. In that case, the status of all target objects OBJ that do not satisfy the driving permission conditions may be set to "indeterminable".

[0097] "Regarding the Status" ·In the above second embodiment, both the case where the driving permission conditions are satisfied and the case where the driving prohibition conditions are satisfied are set to the same status of "determinable", but this is not the only case. For example, they may be set to different statuses such as "drivable" and "driving prohibited", respectively.

[0098] "Regarding the Inquiry Process" ·In the above first embodiment, the inquiry process was always executed when the stopped state continued for a certain period of time in the autonomous driving mode, but this is not the only case. For example, data defining conditions for prohibiting driving regardless of the driver's driving instruction may be stored in the storage device 34, and when the same conditions are satisfied, the inquiry process may not be executed even after a certain period of time has elapsed. Examples of such conditions include the condition that the signal is red in front of an intersection.

[0099] ·In the above second embodiment, marking display was performed when there was a status of "indeterminable", but when there are cases where the status has become "determinable" because the driving prohibition conditions are satisfied, marking display may not be performed. This can be achieved, for example, by adding a determination that the driving prohibition conditions are not satisfied in the process of S122.

[0100] ·As for the inquiry process, it is not limited to the process of marking the object that caused the pause process on the HUD 48. For example, in addition to marking and displaying to surround the object, as illustrated in FIG. 22, a message may be displayed. As in the example shown in FIG. 22, by visually displaying the information that the driver should do when instructing driving, the driver can surely grasp what to do. Also, for example, in addition to the marking process, the in-vehicle speaker may be operated to make an inquiry by voice. However, the marking process of the HUD 48 itself is not essential, and for example, only an inquiry by voice may be used.

[0101] "Regarding the pause process" ·In the above first embodiment, when a positive determination is made in the process of S46, the pause process is executed, but it is not limited to this. For example, when a positive determination is made in the process of S46, the vehicle is decelerated and the process of S64 is immediately executed. When the driver instructs driving, the pause process may not be executed. Also, for example, when a positive determination is made in the process of S46, the deceleration process is not performed and the process of S64 is immediately executed. When the driver instructs driving, the pause process may not be executed.

[0102] ·In the above second embodiment, when a positive determination is made in the process of S46a, the pause process is executed, but it is not limited to this. For example, if the factor that the driving becomes impossible is that the status of "indeterminable" appears, if the driver's driving instruction can be obtained quickly, the pause process may not be executed. FIG. 23 illustrates such a situation.

[0103] Figure 23(a) shows an example where people are standing in front of a crosswalk, and the target objects OBJ(1) and OBJ(2), which are those people, are enclosed by a marking MK. The status list data Dl at that time is shown in Figure 23(b). In this example, since the target object OBJ(3) is far from the vehicle VC, it is assumed to meet the travelable condition. When the driver gazes at the target objects OBJ(1) and OBJ(2) enclosed by the marking MK, their statuses are changed as shown in Figure 23(c). Then, afterwards, when the driver gazes ahead, the status is changed again and the vehicle starts.

[0104] Figure 23(d) shows an example of the status list data Dl immediately after starting. In Figure 23(d), as the distance between the vehicle VC and the target object OBJ(3) has decreased due to the start of the vehicle VC, an example where the status of the target object OBJ(3) is changed to "unable to determine" is shown. In that case, without executing the temporary stop process of the vehicle VC, as shown in Figure 23(e), the target object OBJ(3) is enclosed by the marking MK. And if the status of the target object OBJ(3) becomes "travel permitted" before reaching the front of the intersection, the vehicle VC can continue to travel without being temporarily stopped.

[0105] Note that Figure 23(b) shows an example where the status of the target object OBJ(3) is set to "determinable", but it is not limited to this. For example, the status of the status list data Dl itself may be set to "unable to determine", but when the distance is separated by a predetermined value or more, or when travel is instructed for the target objects OBJ(1) and OBJ(2), the vehicle may be set to start. This can be realized, for example, by deleting the above condition (c) in the process of S142.

[0106] "Regarding the determination process" ·The determination process is not limited to being based on whether the driver's line of sight is directed towards a lane where the vehicle can travel. For example, as described in the section "Regarding the operation acquisition process", when the operation acquisition process acquires a pointing gesture, it may also be a process based on whether the driver points to a lane where the vehicle can travel.

[0107] "Regarding the automatic driving permission process" ·In the above first embodiment, when the driver gives an instruction to drive in response to the inquiry process, the driving by automatic driving is resumed, but it is not limited to this. For example, data defining conditions for prohibiting driving regardless of the driver's driving instruction is stored in the storage device 34, and when the conditions are met, the vehicle may be kept in a stopped state. Examples of such conditions include a condition that the signal is red in front of an intersection.

[0108] ·As described in the section "Regarding the temporary stop process" above, when the temporary stop process is not performed and an instruction to drive by the driver is obtained, a process of continuing the driving by automatic driving without stopping may also be used as the automatic driving permission process. The continuation of the automatic driving here may be, for example, the slowest driving in the case of the scene where the process of S74 is executed in the above first embodiment, and normal driving in the case of the scene where the process of S72 is executed.

[0109] "Regarding the slow driving mode" · In the above first embodiment, the creep control, which is the automatic driving permission process in the creep driving mode, is illustrated in FIGS. 9 and 12, but is not limited thereto. For example, the creep control may be executed in a state where a part of a stopped vehicle enters the front of the driving lane. Generally, when executing the creep control in the case of a negative determination in the process of S68, it is complicated to generate all individual logics in advance regarding what specific cases there are. Therefore, when there is a negative determination in the process of S68, when the start permission is given by the line of sight, in principle, it is effective in avoiding complexity to adopt the logic of starting in the creep driving. In that case, when the factor for the temporary stop is eliminated and the conditions defined in the drivable condition data Drp shown in FIG. 1 are satisfied, the vehicle may shift to the normal driving. At this time, if a reason that does not satisfy the conditions defined in the drivable condition data Drp and is different from the reason at the time of the temporary stop newly occurs, the vehicle may stop temporarily again.

[0110] Note that it is not essential to provide the normal start definition data Ddn. For example, data defining the case of shifting to the creep driving may be provided, and when the case corresponds to the definition, the process may shift to the process of S72. However, it is not limited thereto. For example, at the time of starting after the temporary stop process, the creep driving process may be executed unconditionally.

[0111] "Regarding the speed limit process" · FIG. 21 shows an example of straight driving as a case where the vehicle VC travels while performing the speed limit process, but is not limited thereto. For example, as illustrated in FIG. 24, the vehicle may travel so as to increase the distance from the target object OBJ whose status is "driving permitted". FIG. 24 shows an example in which the vehicle VC travels on the right side of the lane to increase the distance L from the target objects OBJ(1) and OBJ(2) on the left side of the vehicle VC.

[0112] "Regarding the display on the display device" · In the above second embodiment, the processes of S81 and S84 were not mentioned in the situation illustrated in FIG. 20, but these may be included.

[0113] "Regarding the wake-up process" · In the above embodiment, the speaker 49 as a warning device was operated to alert the driver, but it is not limited to this. For example, when adding a condition that the driver has their hand on the steering wheel to the execution conditions of the automatic driving process, a device that vibrates the steering wheel may be used as the warning device, and the driver may be alerted by vibrating the steering wheel.

[0114] "Regarding the entity that determines whether the vehicle can travel" · In the above embodiment, the ADA SECU 30 as a driving control device determined whether the vehicle can travel by automatic driving, but it is not limited to this. For example, the travelable condition data Drp may be stored in a storage device outside the vehicle, and a computer outside the vehicle may determine whether the vehicle can travel by automatic driving. In that case, a computer not installed in the vehicle may transmit the determination result via a global network that enables communication between the computer and the vehicle. In that case, the processes of S46 and S46a will be processes based on the reception of the transmitted determination result.

[0115] Incidentally, in that case, the data transmitted from the vehicle side may be the object recognition result by the LIDAR ECU 10, the object recognition result by the image ECU 20, and the position data Dgps. However, it is not limited to this, and the ranging point cloud data Drpc, the outside-vehicle image data Dio, and the position data Dgps may be transmitted from the vehicle. In that case, the processes executed by the LIDAR ECU 10 and the image ECU 20 in the above embodiment will be executed by a computer not installed in the vehicle.

[0116] Incidentally, as a computer not installed in the vehicle, for example, a device that processes data from a plurality of vehicles may be used. However, instead of this, a driver's mobile terminal may also be used.

[0117] "Regarding the driving control device" ·In the above embodiment, the ADA SECU 30 as the driving control device receives the recognition result of an object obtained by performing clustering processing or the like on the distance measurement point cloud data Drpc by the LIDAR ECU 10, but it is not limited to this. For example, the ADA SECU 30 may receive the distance measurement point cloud data Drpc output by the optical sensor 12 and execute object recognition processing based on the distance measurement point cloud data Drpc by the ADA SECU 30. In other words, the ADA SECU 30 may execute the processing that the LIDAR ECU 10 executed in the above embodiment.

[0118] ·In the above embodiment, the ADA SECU 30 as the driving control device receives the recognition result of an object obtained by performing image recognition processing or the like on the outside vehicle image data Dio by the image ECU 20, but it is not limited to this. For example, the ADA SECU 30 may receive the outside vehicle image data Dio output by the outside vehicle camera 22 and execute object recognition processing based on the image data by the ADA SECU 30. In other words, the ADA SECU 30 may execute the processing that the image ECU 20 executed in the above embodiment.

[0119] ·In the above first embodiment, the ADA SECU 30 as the driving control device executes the driving control program Pad and the DMS program Pdms, but it is not limited to this. For example, as in the second embodiment, the main body that executes the DMS program Pdms may be a device different from the ADA SECU 30. In that case, the ADA SECU 30 may receive the driver's line-of-sight input in response to an inquiry from the different device.

[0120] ·Even when a DMS ECU 60 is provided separately from the ADA SECU 30, for example, the ADA SECU 30 may analyze the in-vehicle image data Dii output by the in-vehicle camera 46 and monitor the line-of-sight input. Even in that case, the in-vehicle camera 46 used by the DMS ECU 60 can be diverted for monitoring processing for the presence or absence of a driving instruction based on the line-of-sight input.

[0121] ·In the above-described second embodiment, the DMSECU 60 is provided separately from the ADASECU 30, but it is not limited thereto. For example, as in the above-described first embodiment, the ADASECU 30 and the DMSECU 60 may be integrated.

[0122] ·In FIG. 2, for convenience of explanation, the processes of S10 and S12 are assumed to be realized by the CPU 32 executing the travel control program Pad, but it is not limited thereto. For example, a navigation system may be provided separately from the ADASECU 30, and the processes of S10 and S12 may be realized by the navigation system.

[0123] ·The travel control device is not limited to one including the CPU 32 and the storage device 34 and executing software processing. For example, at least a part of what was software-processed in the above embodiment may be provided with a dedicated hardware circuit such as an ASIC that performs hardware processing. That is, the execution device may have any of the following configurations (a) to (c). (a) It includes a processing device that executes all of the above processes according to a program and a program storage device that stores the program. (b) It includes a processing device and a program storage device that execute part of the above processes according to a program, and a dedicated hardware circuit that executes the remaining processes. (c) It includes a dedicated hardware circuit that executes all of the above processes. Here, there may be a plurality of software execution devices including a processing device and a program storage device, and dedicated hardware circuits.

[0124] "Regarding the computer" · The computer is not limited to the one installed in the vehicle, such as the CPU 32 illustrated in FIG. 1. For example, although the processes of S64, S66, and S72 in FIG. 5 are executed by the computer installed in the vehicle, the processes of S62, S68, S70, etc. may be executed by a computer not installed in the vehicle. In that case, the computer installed in the vehicle and the computer not installed in the vehicle may cooperate while communicating with each other to execute the processes in FIG. 5. Specifically, for example, when the computer not installed in the vehicle makes an affirmative determination in the process of S70, it may notify the computer installed in the vehicle to that effect, and the computer installed in the vehicle may execute the process of S72, etc.

[0125] "Regarding the recognition process of an object outside the vehicle" · In the above embodiment, an example of recognizing an object based on the distance measurement point group data Drpc output by the optical sensor 12 and the out-of-vehicle image data Dio output by the out-of-vehicle camera 22 is shown, but it is not limited thereto. For example, the distance measurement data output by a radar device such as a millimeter wave may be taken into account. Also, the object recognition process may be executed based on the distance measurement point group data Drpc and the distance measurement data of the radar device without using the out-of-vehicle image data Dio. Further, for example, the object recognition process may be executed based on the out-of-vehicle image data Dio and the distance measurement data of the radar device without using the distance measurement point group data Drpc. Furthermore, it is not limited to the process based on at least two of the three types of data, i.e., the distance measurement point group data Drpc, the out-of-vehicle image data Dio, and the distance measurement data of the radar device. For example, it may be a process based on at least two of the four types of data including these three types and the data based on the reflected wave of ultrasonic waves. However, it is not essential to use so-called sensor fusion for recognizing an object based on the detection values of a plurality of sensors.

[0126] "Regarding the in-vehicle camera" · In the above embodiment, the in-vehicle camera 46 was not particularly mentioned, but it may be, for example, either a visible light camera or an infrared camera.

[0127] Here, when using a visible light camera, as a method for calculating the line of sight, a so-called model-based method of estimating the line of sight by fitting a face or eye model to the input image may be adopted. This can be done, for example, by previously storing in the storage device mapping data that defines a mapping that takes in-vehicle image data Dii as input and outputs facial feature amounts. In that case, the CPU 32 calculates the facial feature amounts by using the in-vehicle image data Dii as the input to the mapping. The facial feature amounts are the coordinate components in the image of a plurality of predetermined facial feature points. The facial feature points include not only the positions of the eyes but also points that are useful in calculating the head pose. The above mapping may be, for example, a convolutional neural network (CNN). Alternatively, a decision tree, support vector regression, or the like may be used. Then, the CPU 32 estimates the head pose that determines the position and direction of the head using a three-dimensional face model from the coordinates of each feature point of the face, which is the facial feature amount. Further, the CPU 32 estimates the center of the eyeball based on the head pose and the coordinates of a predetermined facial feature point. Then, the CPU 32 estimates the position of the iris center based on the eyeball model and the center of the eyeball. Then, the CPU 32 calculates the direction from the center of the eyeball to the center of the iris and uses this as the line of sight direction.

[0128] However, alternatively, the above mapping data may be, for example, data that defines a mapping that takes in-vehicle image data Dii as input and outputs the head pose and the position of the center of the eyeball. Also, for example, the above mapping data may be data that defines a mapping that takes in-vehicle image data Dii as input and outputs the position of the iris center and the position of the center of the eyeball.

[0129] Also, the model adopted in the model-based method is not limited to a model in which the direction from the center of the eyeball to the center of the iris is defined as the line of sight direction. For example, an eyeball model including the shape of the eyelid may be used.

[0130] The method for estimating the line-of-sight direction is not limited to the model-based method. For example, an appearance-based method using a learned model that takes in-vehicle image data Dii as input and outputs a fixation point may be used. Here, as the learned model, for example, a linear regression model, a Gaussian process regression model, a CNN, etc. may be used.

[0131] Also, when using an infrared camera, the reflection point on the cornea may be identified from the reflected light of near-infrared rays, and line-of-sight estimation may be performed based on this and the pupil center position.

Explanation of Signs

[0132] 10…LIDARECU 12…Light sensor 20…Image ECU 22…External camera 30…ADAS ECU 32…CPU 34…Memory device 36…Peripheral circuit 38, 40…Local network 42…Global positioning system 44…Map data 46…Internal camera 47…Input device 48…HUD 49…Speaker 50…Drive system 54…External display device

Claims

1. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle; A travel availability information acquisition process (S62; S103, S105, S106 - S114, S122) for acquiring information indicating that the vehicle cannot be determined to be able to travel by autonomous driving; When acquiring information indicating that the travel cannot be determined to be possible, an inquiry process (S64; S130) for operating a human interface to inquire whether to instruct the driver to drive the vehicle; A determination process (S70, S82; S132, S134, S142 - S146) for determining whether the driver instructs to drive based on the operation acquired by the operation acquisition process according to the inquiry process; When it is determined by the determination process that the driver instructs to drive, an autonomous driving permission process (S72, S86; S154 - S160) for allowing the vehicle to travel by operating the drive system of the vehicle by autonomous driving, and executes, The human interface is a head-up display (48), The inquiry process includes a process of displaying a graphic indicating the object that caused the factor that cannot be determined to be possible on the head-up display, The operation acquisition process includes a process of acquiring the line of sight of the driver, The determination process includes a process of determining that the driver instructs to drive when the line of sight moves away from the graphic and is directed to a lane where the vehicle can travel, a driving control device (30).

2. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle; A travel availability information acquisition process (S62; S103, S105, S106 - S114, S122) for acquiring information indicating that the vehicle cannot be determined to be able to travel by autonomous driving; When acquiring information indicating that the travel cannot be determined to be possible, an inquiry process (S64; S130) for operating a human interface to inquire whether to instruct the driver to drive the vehicle; A determination process (S70, S82; S132, S134, S142 - S146) for determining whether the driver instructs to drive based on the operation acquired by the operation acquisition process according to the inquiry process; When it is determined by the determination process that the driver instructs driving, an automatic driving permission process (S72, S86; S154 to S160) that permits the vehicle to travel by operating the drive system of the vehicle by automatic driving is executed. The automatic driving permission process includes a process (S72) of operating the drive system according to a normal driving mode and a process (S86) of operating the drive system according to a low-speed driving mode. The low-speed driving mode is a mode that is executed on the condition that it is determined by the determination process that the driver instructs driving, and is a mode that continues until a state in which it cannot be determined that the vehicle can travel by automatic driving independently of the determination by the determination process is resolved. A driving control device (30).

3. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle. A travel availability information acquisition process (S62; S103, S105, S106 to S114, S122) for acquiring information indicating that it cannot be determined that the vehicle can travel by automatic driving. When acquiring information indicating that it cannot be determined that the vehicle can travel, an inquiry process (S64; S130) for operating a human interface to inquire the driver whether to instruct the vehicle to travel. A determination process (S70, S82; S132, S134, S142 to S146) for determining whether the driver instructs driving based on the operation acquired by the operation acquisition process according to the inquiry process. When it is determined by the determination process that the driver instructs driving, an automatic driving permission process (S72, S86; S154 to S160) that permits the vehicle to travel by operating the drive system of the vehicle by automatic driving is executed. The automatic driving permission process includes a speed limit process (S156) of operating the drive system to drive the vehicle while limiting the vehicle speed by automatic driving until a state in which it cannot be determined that the vehicle can travel by automatic driving independently of the determination by the determination process is resolved. The speed limit process is a process of limiting the traveling speed of the vehicle to a smaller value when the distance between the object that has become a factor that cannot be determined to be possible and the vehicle is small than when it is large. A driving control device (30).

4. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle. A travelability information acquisition process (S62; S103, S105, S106 - S114, S122) for acquiring information indicating that the vehicle cannot be determined to be capable of traveling by autonomous driving, When acquiring information indicating that the travel is not possible, an inquiry process (S64; S130) for operating a human interface to inquire whether to instruct the driver to travel the vehicle, A determination process (S70, S82; S132, S134, S142 - S146) for determining whether the driver instructs travel based on the operation acquired by the operation acquisition process according to the inquiry process, When it is determined by the determination process that the driver instructs travel, an autonomous driving permission process (S72, S86; S154 - S160) for permitting the vehicle to travel by operating the drive system of the vehicle by autonomous driving, and executes, The inquiry process includes a process of inquiring whether to instruct the travel when a person exists in front of a crosswalk without a traffic signal in front of the traveling direction of the vehicle. A travel control device (30).

5. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle, A travelability information acquisition process (S62; S103, S105, S106 - S114, S122) for acquiring information indicating that the vehicle cannot be determined to be capable of traveling by autonomous driving, When acquiring information indicating that the travel is not possible, an inquiry process (S64; S130) for operating a human interface to inquire whether to instruct the driver to travel the vehicle, A determination process (S70, S82; S132, S134, S142 - S146) for determining whether the driver instructs travel based on the operation acquired by the operation acquisition process according to the inquiry process, When it is determined by the determination process that the driver instructs travel, an autonomous driving permission process (S72, S86; S154 - S160) for permitting the vehicle to travel by operating the drive system of the vehicle by autonomous driving, and executes, The inquiry process includes a process of inquiring whether to instruct the travel when there is an object obstructing the view of a lane intersecting the travel lane among the intersections in front of the vehicle. A travel control device (30).

6. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle; A travel availability information acquisition process (S62; S103, S105, S106 to S114, S122) for acquiring information indicating that the vehicle cannot be determined to be able to travel by automatic driving; When acquiring information indicating that the travel cannot be determined to be possible, an inquiry process (S64; S130) for operating a human interface to inquire the driver whether to instruct the vehicle to travel; A determination process (S70, S82; S132, S134, S142 to S146) for determining whether the driver instructs travel based on the operation acquired by the operation acquisition process according to the inquiry process; When it is determined by the determination process that the driver instructs travel, an automatic driving permission process (S72, S86; S154 to S160) for permitting the vehicle to travel by operating the drive system of the vehicle by automatic driving is executed, The inquiry process includes a process of inquiring whether to instruct the travel when an object exists in front of the lane in which the vehicle is traveling; The determination process includes a process of determining that the driver instructs travel when there are a plurality of lanes with the traveling direction of the vehicle as the traveling direction and the driver makes an operation of instructing a lane adjacent to the lane in which the driver is traveling. A travel control device (30).

7. An operation acquisition process (S66, S66a) for acquiring the operation of the driver based on the image data of the driver of the vehicle; A travel availability information acquisition process (S62; S103, S105, S106 to S114, S122) for acquiring information indicating that the vehicle cannot be determined to be able to travel by automatic driving; When acquiring information indicating that the travel cannot be determined to be possible, an inquiry process (S64; S130) for operating a human interface to inquire the driver whether to instruct the vehicle to travel; A determination process (S70, S82; S132, S134, S142 to S146) for determining whether the driver instructs travel based on the operation acquired by the operation acquisition process according to the inquiry process; When it is determined by the determination process that the driver instructs travel, an automatic driving permission process (S72, S86; S154 to S160) for permitting the vehicle to travel by operating the drive system of the vehicle by automatic driving is executed, In the vehicle, when it is determined based on the image data that the driver's operation is inappropriate for driving the vehicle, an arousal process is executed to operate a warning device to arouse the driver's attention. A driving control device (30).

8. The determination process is For the inquiry process, when it is detected that the line of sight is directed toward the figure, a detection history storage process (S132, 134) for storing that fact, A travel permission determination process (S142 to S146) for determining that when it is detected that the line of sight is directed toward the figure and the line of sight moves away from the figure and is directed toward a lane in which the vehicle can travel, the driver instructs travel. The driving control device according to claim 1, comprising:

9. The travelability information acquisition process is Including a prohibition determination process (S110) for determining whether or not the vehicle can travel under the conditions that the vehicle cannot travel by the automatic driving, The driving control device according to claim 1, which is a process of inquiring whether or not to instruct the driver to drive the vehicle when it is determined by the prohibition determination process that the vehicle does not meet the conditions for not being able to travel.

10. The vehicle includes a display device (52) visible from the outside of the vehicle, When it is determined that the vehicle cannot travel in a situation where a person exists in front of a crosswalk without a traffic signal in front of the vehicle in the traveling direction, and the vehicle is stopped, a process (S81) of displaying information indicating that the vehicle is waiting to cross on the display device. The driving control device according to claim 4, comprising:

11. The vehicle includes a display device (52) visible from the outside of the vehicle, When it is determined that the driver instructs the vehicle to travel in a situation where a person exists in front of a crosswalk without a traffic signal in front of the vehicle in the traveling direction, a process (S84) of displaying information prompting the driver to pay attention to travel on the display device. The driving control device according to claim 4, comprising:

12. A driving control method having steps of executing each of the travelability information acquisition process, the inquiry process, and the determination process in the driving control device according to any one of claims 1 to 11.

13. A driving control program (Pad) for causing a computer (32) to execute the travelability information acquisition process, the inquiry process, and the determination process in the driving control device according to any one of claims 1 to 11.

Images