Vehicle control device, vehicle control method, and vehicle control program
The vehicle remote instruction system optimizes sensor data transmission by limiting it to a defined range based on the external situation and route, addressing inefficiencies in existing systems by ensuring only necessary information is sent to remote commanders.
Patent Information
- Application Number
- JP2023073538
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2039-09-06
AI Technical Summary
Existing vehicle remote instruction systems transmit unnecessary sensor information to remote commanders, increasing data volume and inefficiency due to lack of consideration for the external situation of the automated driving vehicle.
A vehicle remote instruction system that limits the transmission of sensor information to a remote commander based on the external situation and route of the automated driving vehicle, using a transmission information limiting section to set a limited information range for sensor data transmission.
Reduces data volume transmitted to remote commanders while ensuring accurate remote instructions by only sending required sensor information, enhancing system efficiency and reducing unnecessary data transmission.
Smart Images

Figure 0007578154000001 
Figure 0007578154000002 
Figure 0007578154000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a vehicle remote instruction system in which a remote commander remotely instructs an autonomous vehicle regarding its travel. [Background technology]
[0002] For example, Patent Document 1 describes a vehicle remote instruction system that controls the traveling of an autonomous vehicle by remotely instructing the autonomous vehicle from a remote commander. In this vehicle remote instruction system, sensor information from an on-board sensor is transmitted from the autonomous vehicle to the remote commander, and the remote commander issues remote instructions based on the transmitted sensor information. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-180771 A Summary of the Invention [Problem to be solved by the invention]
[0004] Here, the sensor information required when the remote commander issues a remote instruction varies depending on the external situation (scene) of the autonomous vehicle. When sensor information is transmitted without considering the external situation of the autonomous vehicle, unnecessary sensor information is also transmitted, and the amount of transmitted data increases. For this reason, in this technical field, there is a demand for reducing the data amount of sensor information transmitted to the remote commander depending on the external situation of the autonomous vehicle. [Means for solving the problem]
[0005] One aspect of the present invention is a vehicle remote instruction system in which a remote commander issues remote instructions regarding the driving of an autonomous vehicle based on sensor information from an external sensor that detects the external environment of the autonomous vehicle, and the system is equipped with a transmission information limiting unit that sets the range of information to be transmitted to the remote commander from the sensor information detected by the external sensor to a limited information range based on the external situation obtained based on the external environment or map information, and the route of the autonomous vehicle.
[0006] In this vehicle remote instruction system, the transmission information limiting unit sets the range of information to be transmitted to the remote commander to a limited information range, so that only sensor information (necessary sensor information) corresponding to the external situation and course of the autonomous vehicle among the sensor information detected by the external sensors can be transmitted to the remote commander. This enables the vehicle remote instruction system to appropriately execute remote instructions by the remote commander while reducing the data volume of the sensor information transmitted from the autonomous vehicle to the remote commander.
[0007] In the vehicle remote control system, the external sensor may have a plurality of sensors that detect the external environment, and the transmission information limiting unit may determine which of the plurality of sensors will transmit sensor information to the remote commander based on the external situation and the route, and set the sensor information detected by the determined sensor as sensor information within the limited information range.
[0008] In this way, even if the vehicle remote control system has multiple sensors as external sensors, it can reduce the data volume of sensor information transmitted to the remote commander by determining the sensor from which to transmit sensor information depending on the external situation, and setting the sensor information detected by the determined sensor as sensor information within the limited information range.
[0009] In the vehicle remote instruction system, the transmission information limiting unit may extract a portion of the sensor information detected by the external sensor to be transmitted to the remote commander based on the external situation and the route, and set the extracted portion of the sensor information as sensor information within the limited information range.
[0010] In this way, the vehicle remote control system can reduce the data volume of the sensor information transmitted to the remote commander by extracting the portion to be transmitted from the sensor information of the external sensor and setting the extracted portion as sensor information within the limited information range.
[0011] The vehicle remote instruction system may further include an information presence / absence determination unit that determines whether necessary information that is preset in accordance with an external situation is included in the sensor information within the limited information range, and a notification suggestion unit that, when the information presence / absence determination unit determines that the necessary information is not included in the sensor information within the limited information range, notifies the remote commander or suggests to the remote commander to change the position and attitude of the autonomous vehicle or the detection area of the external sensor.
[0012] In this case, the vehicle remote instruction system can notify the remote commander that the necessary information is not included in the limited information range, or can suggest to the remote commander that the necessary information is not included in the limited information range and that the position and attitude of the autonomous vehicle or the detection area of the external sensor needs to be changed, thereby enabling the remote commander to take action based on the notification or suggestion from the vehicle remote instruction system. Effect of the Invention
[0013] According to the present invention, it is possible to reduce the data volume of sensor information transmitted to a remote commander in accordance with the external environment of an autonomous vehicle. [Brief description of the drawings]
[0014] [Figure 1] FIG. 1 is a diagram showing an example of an overall image of a vehicle remote instruction system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of an autonomous driving vehicle. [Diagram 3] FIG. 3 is a schematic diagram illustrating the setting of a limited information range when an autonomous vehicle makes a right turn at an intersection. [Figure 4]Fig. 4(a) is a schematic diagram for explaining the setting of a limited information range when an autonomous vehicle overtakes a stopped vehicle on the driving lane. Fig. 4(b) is a schematic diagram showing a change in the detection area when the position and attitude of the autonomous vehicle are changed. Fig. 4(c) is a schematic diagram showing a change in the detection area when the detection area of the external sensor is changed. [Diagram 5] FIG. 5 is a schematic diagram illustrating the setting of a limited information range when an autonomous vehicle 2 travels straight through an intersection where traffic lights are installed. [Figure 6] Fig. 6(a) is a schematic diagram for explaining the setting of the limited information range when an autonomous vehicle is driving straight through a narrow alley intersection where no traffic lights are installed. Fig. 6(b) is a schematic diagram showing the change in the detection area when the position and attitude of the autonomous vehicle are changed. [Figure 7] FIG. 7 is a schematic diagram illustrating the setting of a limited information range when a construction zone exists ahead of an autonomously driven vehicle. [Figure 8] FIG. 8 is a block diagram showing an example of a hardware configuration of the remote instruction server. [Figure 9] FIG. 9 is a block diagram showing an example of the configuration of a remote instruction device. [Figure 10] FIG. 10 is a diagram illustrating an example of the instruction input unit. [Figure 11] FIG. 11 is a flowchart showing the flow of processing performed by the autonomous driving ECU when a remote instruction target situation occurs. [Figure 12] Fig. 12(a) is a schematic diagram for explaining the setting of a limited information range when an autonomous vehicle turns right at an intersection. Fig. 12(b) is a schematic diagram showing a change in the detection area when the position and attitude of the autonomous vehicle are changed. Fig. 12(c) is a schematic diagram showing a change in the detection area when the detection area of an external sensor is changed. [Figure 13] FIG. 13 is a schematic diagram illustrating the setting of a limited information range when an autonomous vehicle makes a left turn at an intersection. [Figure 14] FIG. 14 is a schematic diagram illustrating the setting of a limited information range when an autonomous vehicle enters a roundabout. [Figure 15] FIG. 15 is a schematic diagram for explaining the setting of a limited information range when a fallen object (obstacle) is present on the driving lane ahead of the autonomously driven vehicle. [Figure 16] FIG. 16 is a schematic diagram illustrating the setting of a limited information range when an autonomous vehicle passes through a pedestrian crossing. [Figure 17] FIG. 17 is a schematic diagram illustrating the setting of a limited information range when an autonomously driven vehicle is stopped at a parking space. [Figure 18] FIG. 18 is a schematic diagram illustrating the setting of a limited information range when starting an autonomously driven vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same or corresponding elements are denoted by the same reference numerals, and duplicated description will be omitted.
[0016] Fig. 1 is a diagram showing an example of an overall image of a vehicle remote instruction system according to an embodiment. The vehicle remote instruction system 100 shown in Fig. 1 is a system in which a remote commander R issues remote instructions regarding the traveling of an autonomous vehicle 2 based on detection information from an external sensor 22 that detects the external environment of the autonomous vehicle 2. The remote instructions are instructions from the remote commander R regarding the traveling of the autonomous vehicle 2.
[0017] The remote instructions include an instruction to proceed the autonomous vehicle 2 and an instruction to stop the autonomous vehicle 2. The remote instructions may also include an instruction to change lanes for the autonomous vehicle 2. The remote instructions may also include an instruction to offset avoid an obstacle ahead, an instruction to overtake a leading vehicle, an instruction to make an emergency evacuation, and the like.
[0018] (Configuration of vehicle remote control system) As shown in Fig. 1, a vehicle remote instruction system 100 includes a remote instruction device 1 to which a remote commander R inputs remote instructions. The remote instruction device 1 is communicably connected to a plurality of autonomously driven vehicles 2 via a network N. The network N is a wireless communication network. Various types of information are sent to the remote instruction device 1 from the autonomously driven vehicles 2.
[0019] In the vehicle remote instruction system 100, for example, in response to a remote instruction request from an autonomous vehicle 2, a remote commander R is requested to input a remote instruction. The remote commander R inputs the remote instruction to a commander interface 3 of the remote instruction device 1. The remote instruction device 1 transmits the remote instruction to the autonomous vehicle 2 via a network N. The autonomous vehicle 2 travels autonomously according to the remote instruction.
[0020] The number of remote commanders R in the vehicle remote instruction system 100 may be one or more. The number of autonomously driven vehicles 2 that can communicate with the vehicle remote instruction system 100 is not particularly limited. A configuration in which multiple remote commanders R take turns giving remote instructions to one autonomously driven vehicle 2, or a configuration in which one remote commander R gives remote instructions to two or more autonomously driven vehicles 2, may also be used.
[0021] (Configuration of autonomous vehicles) First, an example of the configuration of the autonomous vehicle 2 will be described. FIG. 2 is a block diagram showing an example of the configuration of the autonomous vehicle 2. As shown in FIG. 2, the autonomous vehicle 2 has an autonomous driving ECU 20 as an example. The autonomous driving ECU 20 is an electronic control unit having a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like. In the autonomous driving ECU 20, for example, a program recorded in the ROM is loaded into the RAM, and the program loaded into the RAM is executed by the CPU, thereby realizing various functions. The autonomous driving ECU 20 may be composed of a plurality of electronic units.
[0022] The autonomous driving ECU 20 is connected to a GPS (Global Positioning System) receiving unit 21, an external sensor 22, an internal sensor 23, a map database 24, a communication unit 25, and an actuator 26.
[0023] The GPS receiver 21 receives signals from three or more GPS satellites to measure the position of the autonomously driven vehicle 2 (for example, the latitude and longitude of the autonomously driven vehicle 2). The GPS receiver 21 transmits the measured position information of the autonomously driven vehicle 2 to the autonomous driving ECU 20.
[0024] The external sensor 22 is an on-board sensor that detects the external environment of the autonomous vehicle 2. The external sensor 22 transmits the detected detection information (sensor information) to the autonomous driving ECU 20. The external sensor 22 includes a plurality of sensors that detect the external environment. Specifically, the external sensor 22 includes at least a camera as a sensor. The camera is an imaging device that captures an image of the external environment of the autonomous driving vehicle 2. The camera is provided, for example, on the back side of the windshield of the autonomous driving vehicle 2, and captures an image in front of the vehicle. The camera transmits a captured image (sensor information) related to the external environment of the autonomous driving vehicle 2 to the autonomous driving ECU 20. The camera may be a monocular camera or a stereo camera. In addition, a plurality of cameras are provided, and capture images of all or at least a part of the surroundings of the autonomous driving vehicle 2, such as the front, left and right sides, and rear of the autonomous driving vehicle 2, in addition to the front of the autonomous driving vehicle 2.
[0025] The external sensor 22 may include a radar sensor as a sensor. The radar sensor is a detection device that detects objects around the autonomous vehicle 2 using radio waves (e.g., millimeter waves) or light. The radar sensor includes, for example, a radar (millimeter wave radar) or a lidar (LIDAR: Light Detection and Ranging) as a sensor. The radar sensor detects objects by transmitting radio waves or light to the periphery of the autonomous vehicle 2 and receiving the radio waves or light reflected by the object. The radar sensor transmits detected object information (sensor information) to the autonomous driving ECU 20. The object includes fixed objects such as guardrails and buildings, as well as moving objects such as pedestrians, bicycles, and other vehicles. A plurality of radar sensors are provided, and the entire or at least a part of the periphery of the autonomous vehicle 2 is the detection target. The external sensor 22 may also include a sonar sensor that irradiates sound waves around the autonomous vehicle 2 and detects the sound waves reflected by obstacles, etc.
[0026] The internal sensor 23 is an in-vehicle sensor that detects the traveling state of the autonomously driven vehicle 2. The internal sensor 23 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the autonomously driven vehicle 2. As the vehicle speed sensor, a wheel speed sensor that is provided on the wheels of the autonomously driven vehicle 2 or on a drive shaft that rotates integrally with the wheels and detects the rotational speed of each wheel can be used. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the autonomous driving ECU 20.
[0027] The acceleration sensor is a detector that detects the acceleration of the autonomously driven vehicle 2. The acceleration sensor includes, for example, a longitudinal acceleration sensor that detects the acceleration in the longitudinal direction of the autonomously driven vehicle 2. The acceleration sensor may include a lateral acceleration sensor that detects the lateral acceleration of the autonomously driven vehicle 2. The acceleration sensor transmits, for example, acceleration information of the autonomously driven vehicle 2 to the autonomous driving ECU 20. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the autonomously driven vehicle 2. For example, a gyro sensor can be used as the yaw rate sensor. The yaw rate sensor transmits the detected yaw rate information of the autonomously driven vehicle 2 to the autonomous driving ECU 20.
[0028] In addition to the external sensor 22, the autonomous vehicle 2 may be provided with a remote instruction sensor for a remote commander. The remote instruction sensor may be provided so as to have a different angle of view from that of the external sensor 22. The remote instruction sensor may be provided, for example, on the roof of the autonomous vehicle 2 and configured to detect the surroundings from a high position. The remote instruction sensor may have a lower resolution and a smaller data capacity than the external sensor 22 that recognizes the external environment for autonomous driving. This makes it possible to reduce the data capacity when transmitting the detection result of the remote instruction sensor to the remote instruction server 10, thereby suppressing the occurrence of delays.
[0029] The map database 24 is a database that records map information. The map database 24 is formed, for example, in a recording device such as a hard disk drive (HDD) mounted on the autonomous vehicle 2. The map information includes road position information, road shape information (e.g., curvature information), and intersection and branch point position information. The map information may include traffic regulation information such as a legal maximum speed associated with the position information. The map information may include landmark information used to acquire the position information of the autonomous vehicle 2. Examples of landmarks that may be used include road signs, road markings, traffic lights, utility poles, and the like. The map database 24 may be configured in a server that can communicate with the autonomous vehicle 2.
[0030] The communication unit 25 is a communication device that controls wireless communication with the outside of the autonomous driving vehicle 2. The communication unit 25 transmits and receives various information to and from the remote instruction device 1 (remote instruction server 10) via the network N.
[0031] The actuator 26 is a device used to control the autonomous vehicle 2. The actuator 26 includes at least a drive actuator, a brake actuator, and a steering actuator. The drive actuator controls the amount of air supplied to the engine (throttle opening) in response to a control signal from the autonomous driving ECU 20, thereby controlling the driving force of the autonomous vehicle 2. When the autonomous vehicle 2 is a hybrid vehicle, in addition to the amount of air supplied to the engine, a control signal from the autonomous driving ECU 20 is input to a motor serving as a power source to control the driving force. When the autonomous vehicle 2 is an electric vehicle, a control signal from the autonomous driving ECU 20 is input to a motor serving as a power source to control the driving force. The motor serving as a power source in these cases constitutes the actuator 26.
[0032] The brake actuator controls a brake system in response to a control signal from the autonomous driving ECU 20, and controls the braking force applied to the wheels of the autonomous driving vehicle 2. As the brake system, for example, a hydraulic brake system can be used. The steering actuator controls the drive of an assist motor that controls the steering torque of the electric power steering system in response to a control signal from the autonomous driving ECU 20. In this way, the steering actuator controls the steering torque of the autonomous driving vehicle 2.
[0033] Next, a description will be given of an example of the functional configuration of the autonomous driving ECU 20. The autonomous driving ECU 20 has a vehicle position acquisition unit 31, an external environment recognition unit 32, a driving state recognition unit 33, a remote instruction determination unit 34, a transmission information restriction unit 35, an information presence / absence determination unit 36, a notification proposal unit 37, a sensor angle of view change unit 38, a driving state information transmission unit 39, a route generation unit 40, and an autonomous driving control unit 41.
[0034] The vehicle position acquisition unit 31 acquires position information (position on the map) of the autonomous vehicle 2 based on the position information of the GPS receiver 21 and the map information of the map database 24. The vehicle position acquisition unit 31 may acquire the position information of the autonomous vehicle 2 by SLAM [Simultaneous Localization and Mapping] technology using target information included in the map information of the map database 24 and the detection result of the external sensor 22. The vehicle position acquisition unit 31 may recognize the lateral position of the autonomous vehicle 2 relative to the lane (the position of the autonomous vehicle 2 in the lane width direction) from the positional relationship between the lane markings and the autonomous vehicle 2, and include it in the position information. The vehicle position acquisition unit 31 may acquire the position information of the autonomous vehicle 2 by other well-known methods.
[0035] The external environment recognition unit 32 recognizes the external environment of the autonomous vehicle 2 based on the detection result of the external sensor 22. The external environment includes the relative positions of surrounding objects with respect to the autonomous vehicle 2. The external environment may include the relative speed and movement direction of surrounding objects with respect to the autonomous vehicle 2. The external environment may include types of objects such as other vehicles, pedestrians, and bicycles. The types of objects can be identified by well-known methods such as pattern matching. The external environment may include the results of lane marking recognition (white line recognition) around the autonomous vehicle 2. The external environment may include the recognition results of the lighting state of a traffic light. For example, the external environment recognition unit 32 can recognize the lighting state of a traffic light in front of the autonomous vehicle 2 (whether the lighting state is for passing or for not passing, etc.) based on an image from the camera of the external sensor 22.
[0036] The driving state recognition unit 33 recognizes the driving state of the autonomously driven vehicle 2 based on the detection result of the internal sensor 23. The driving state includes the speed of the autonomously driven vehicle 2, the acceleration of the autonomously driven vehicle 2, and the yaw rate of the autonomously driven vehicle 2. Specifically, the driving state recognition unit 33 recognizes the speed of the autonomously driven vehicle 2 based on vehicle speed information from a vehicle speed sensor. The driving state recognition unit 33 recognizes the acceleration of the autonomously driven vehicle 2 based on vehicle speed information from an acceleration sensor. The driving state recognition unit 33 recognizes the orientation of the autonomously driven vehicle 2 based on yaw rate information from a yaw rate sensor.
[0037] The remote instruction determination unit 34 determines whether the autonomous vehicle 2 should request a remote instruction from the remote commander R (remote instruction device 1). The remote instruction determination unit 34 determines whether a remote instruction should be requested based on at least one of the position information of the autonomous vehicle 2 acquired by the vehicle position acquisition unit 31 and the map information in the map database 24, the external environment recognized by the external environment recognition unit 32, and the route generated by the route generation unit 40 described later.
[0038] The remote instruction determination unit 34 determines that a remote instruction should be requested, for example, when the autonomous vehicle 2 is in a remote instruction target situation. The remote instruction target situation is a situation that is preset as a situation in which the autonomous vehicle should request a remote instruction from the remote instruction device 1.
[0039] The remote instruction target situation may include, for example, at least one of the following situations: the autonomous vehicle 2 turns right or left at an intersection, entering an intersection with or without traffic lights, entering a roundabout, passing a crosswalk, there is a stopped vehicle or obstacle ahead, changing lanes to avoid a construction zone, a situation where a decision on offset avoidance for an obstacle ahead is required, a stopped autonomous vehicle starts moving, and a situation where the autonomous vehicle stops at a boarding point or destination. Note that, in a country or region where the vehicle drives on the right side of the road, the situation of turning left at an intersection can be used instead of the situation of turning right at an intersection.
[0040] The remote instruction determination unit 34 determines that a remote instruction should be requested, for example, when the autonomous vehicle 2 is about to enter an intersection or to turn right at an intersection. The remote instruction determination unit 34 may also determine that a remote instruction should be requested when an obstacle that needs to be avoided by offsetting is present ahead of the autonomous vehicle 2.
[0041] The remote instruction determination unit 34 can recognize, for example, from the position information, map information, and target route of the autonomous vehicle 2, that the autonomous vehicle 2 is in a situation to turn right at an intersection, that the autonomous vehicle 2 is in a situation to enter an intersection with traffic lights, or that the autonomous vehicle 2 is in a situation to start changing lanes.
[0042] When the remote instruction determination unit 34 determines that a remote instruction should be requested, it requests a remote instruction from the remote commander R to the remote instruction server 10. The request for a remote instruction includes, for example, identification information of the autonomously driven vehicle 2. The remote instruction determination unit 34 may request a remote instruction well in advance. The remote instruction determination unit 34 may determine that a remote instruction should be requested when the distance between the autonomously driven vehicle 2 and an intersection or the like that is the target of the remote instruction becomes equal to or shorter than a certain distance. The remaining time until arrival may be used instead of the distance.
[0043] When the remote instruction determination unit 34 determines that a remote instruction request is necessary (when a remote instruction target situation occurs), the transmission information limiting unit 35 sets driving situation information of the autonomously driven vehicle 2 to be transmitted to the remote instruction server 10. The driving situation information of the autonomously driven vehicle 2 is used when the remote commander R issues a remote instruction. The driving situation information of the autonomously driven vehicle 2 will be described in detail later, but the driving situation information includes detection information of the external sensor 22. In this embodiment, the transmission information limiting unit 35 sets a range of information to be transmitted to the remote instruction server 10 from the detection information of the external sensor 22 included in the driving situation information.
[0044] Specifically, the transmission information limiting unit 35 sets the range of information to be transmitted to the remote instruction server 10 (remote commander R) from among the detection information detected by the external sensor 22, based on the external situation of the autonomously driven vehicle 2 and the route of the autonomously driven vehicle 2, to a limited information range.
[0045] The external situation of the autonomous vehicle 2 can be obtained based on the external environment or map information. The transmission information limiting unit 35 uses the external environment obtained based on the detection information of the external sensor 22 (i.e., the external environment recognized by the external environment recognition unit 32) as the external environment of the autonomous vehicle 2. The map information used by the transmission information limiting unit 35 is the map information stored in the map database 24. The transmission information limiting unit 35 can use the route generated by the route generation unit 40 as the route. The route here is the route before the route was generated in response to the remote instruction.
[0046] The limited information range is a range that is more limited than the detection information of the external sensor 22 that is transmitted to the remote instruction server 10 when the range of information based on the external situation and the course is not set. By setting the range of the detection information to be transmitted to the limited information range in this way, the data volume of the detection information to be transmitted is smaller than when the range is not set based on the external situation and the course.
[0047] The transmission information limiting unit 35 sets, based on the external situation and the course, detection information of a sensor whose detection area includes an area to be confirmed when the remote commander R issues a remote instruction, as information of the limited information range.
[0048] (Determine the sensor that will send the detection information and set the limited information range) A method of determining which of the multiple sensors included in the external sensor 22 will transmit detection information to the remote commander R, and setting the detection information detected by the determined sensor as detection information within the limited information range will be described below.
[0049] First, an example of setting the limited information range using the external situation obtained based on map information will be described. Based on the map information and the route, the transmission information limiting unit 35 determines which of the multiple sensors in the external sensor 22 will transmit detection information to the remote instruction server 10. Then, the transmission information limiting unit 35 can set the detection information detected by the determined sensor as detection information within the limited information range.
[0050] As an example, a case will be described in which the autonomous vehicle 2 turns right at an intersection as shown in FIG. 3. In this case, when issuing a remote instruction, the remote commander R needs to confirm the absence of both a vehicle traveling straight in the oncoming lane and a pedestrian crossing the road to which the autonomous vehicle 2 is to turn right. For this reason, the transmission information limiting unit 35 determines that the autonomous vehicle 2 is in a situation to turn right at an intersection based on the map information, the position information of the autonomous vehicle 2 acquired by the vehicle position acquiring unit 31, and the course. Then, the transmission information limiting unit 35 selects (determines) a sensor whose detection area is the front and a sensor whose detection area is the right front (diagonally right front) from among the multiple sensors that the external sensor 22 has.
[0051] In addition, when there is a sensor whose detection area is both forward and to the right forward, the transmission information limiting unit 35 may select this sensor. That is, the transmission information limiting unit 35 selects one or more sensors whose detection area includes an area to be confirmed by the remote commander R (the same applies to the selection of sensors described below). Then, the transmission information limiting unit 35 can set the detection information detected by the selected sensor as the detection information within the limited information range. In addition, in FIG. 3, the hatched area drawn around the autonomously driven vehicle 2 indicates the detection area (angle of view) of each sensor of the external sensor 22 (the same applies to other figures).
[0052] In this way, in the above example, the transmission information limiting unit 35 does not include in the limited information range the detection information of a sensor whose detection area is behind the vehicle. In other words, the transmission information limiting unit 35 does not include in the limited information range the detection information of a sensor whose detection area does not include the area that the remote commander R should check based on the external situation obtained based on the map information.
[0053] In this case, since it is sufficient to detect vehicles traveling straight in the oncoming lane and pedestrians crossing the road after turning right, the transmission information limiting unit 35 may select any type of sensor (sensor whose detection information is to be included within the limited information range) from among the types of sensors listed as examples of sensors possessed by the external sensor 22, namely, camera, radar, and lidar.
[0054] Next, an example of setting the limited information range using the external situation obtained based on the external environment will be described. The transmission information limiting unit 35 determines a sensor that transmits detection information to the remote instruction server 10 from among the multiple sensors that the external sensor 22 has, based on the external environment and route of the autonomously driven vehicle 2 recognized by the external environment recognition unit 32. Then, the transmission information limiting unit 35 can set the detection information detected by the determined sensor as detection information within the limited information range.
[0055] As an example, as shown in FIG. 4(a), a case of overtaking a stopped vehicle X1 that is stopped on the driving lane of the autonomous vehicle 2 will be described. In this case, when issuing a remote instruction to overtake the stopped vehicle X1, the remote commander R needs to confirm the absence of both an oncoming vehicle traveling straight in the oncoming lane in front of the autonomous vehicle 2 (in the case of a road with one lane in each direction) and a vehicle attempting to overtake the autonomous vehicle 2 and the stopped vehicle X1 from behind the autonomous vehicle 2. For this reason, the transmission information limiting unit 35 recognizes that the autonomous vehicle 2 is in a situation to overtake the stopped vehicle X1 based on the information (external environment) of the stopped vehicle X1 detected by the external environment recognition unit 32 and the course. Then, the transmission information limiting unit 35 selects (determines) a sensor whose detection area is the front and a sensor whose detection area is the rear of the autonomous vehicle 2 from among the multiple sensors that the external sensor 22 has. The transmission information limiting unit 35 can set the detection information detected by the selected sensor as detection information within the limited information range.
[0056] In this way, the transmission information limiting unit 35 does not include in the limited information range detection information of a sensor whose detection area does not include the area that the remote commander R should check, based on the external situation obtained from the external environment.
[0057] In this case, since it is sufficient to detect vehicles in front of and behind the autonomous vehicle 2, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a lidar, as examples.
[0058] The transmission information limiting unit 35 may set the limited information range using an external situation obtained based on the external environment and map information. Specifically, as an example, as shown in Fig. 3, when the autonomous vehicle 2 turns right at an intersection, the limited information range may be set based on the presence or absence of an oncoming vehicle X waiting to turn right in the oncoming lane of the autonomous vehicle 2.
[0059] For example, when an oncoming vehicle X waiting to turn right is present, the oncoming straight lane (the oncoming straight lane) located behind the oncoming vehicle X (the rear side as seen from the autonomous vehicle 2) is blocked by the oncoming vehicle X, making it difficult for the external sensor 22 to detect it, and the recognition performance of other vehicles and the like is reduced. In such a case, the autonomous vehicle 2 may travel based on a remote instruction from the remote commander R. However, when there is no oncoming vehicle X waiting to turn right in the oncoming lane ahead of the autonomous vehicle 2, the autonomous vehicle 2 can automatically turn right by autonomous driving. For this reason, as an example, when turning right at an intersection, when the oncoming vehicle X waiting to turn right ahead of the autonomous vehicle 2 is not recognized by the external environment recognition unit 32, the transmission information limiting unit 35 may set only the detection information of the sensor whose detection area is the right front as the detection information within the limited information range, without including the detection information of the sensor whose detection area is the front among the multiple sensors that the external sensor 22 has, in the limited information range.
[0060] 3, if a pedestrian attempting to cross the road after a right turn has not been recognized by the external environment recognition unit 32, the transmission information limiting unit 35 may not include the detection information of the sensor whose detection area is the right front as the detection information within the limited information range. In other words, if there is no pedestrian attempting to cross the road after a right turn, the remote commander R does not need to check for pedestrians, and therefore the transmission information limiting unit 35 may not include the detection information of the sensor whose detection area is the right front as the detection information within the limited information range.
[0061] (Extract the part to be sent from the detection information and set the limited information range) A method of extracting a portion to be transmitted to the remote commander R from the detection information detected by the external sensor 22 and setting the extracted portion of the detection information as detection information within the limited information range will be described below.
[0062] First, an example of setting the limited information range using the external situation obtained based on map information will be described. The transmission information limiting unit 35 extracts a part to be transmitted to the remote instruction server 10 from the detection information detected by the external sensor 22 based on the map information and the route. The transmission information limiting unit 35 can then set the extracted part of the detection information as the detection information within the limited information range. Here, extracting a part to be transmitted to the remote instruction server 10 from the detection information means, for example, cutting out an unnecessary part of the detection information and leaving only the necessary part (the part to be transmitted). As an example, the transmission information limiting unit 35 can cut out a part of an image captured by a camera of the external sensor 22 and use it as the part to be transmitted.
[0063] As an example, as shown in FIG. 5, a case where the autonomous vehicle 2 goes straight through an intersection where a traffic light is installed will be described. In this case, as an example, the remote commander R needs to check the light information of the traffic light (color information of the lighted traffic light) when issuing a remote instruction. For this reason, the transmission information limiting unit 35 determines that the autonomous vehicle 2 goes straight through the intersection where a traffic light is installed based on the map information, the position information of the autonomous vehicle 2 acquired by the vehicle position acquisition unit 31, and the route. Then, the transmission information limiting unit 35 selects (determines) a sensor that includes the traffic light in its detection area from among the multiple sensors that the external sensor 22 has. Here, the transmission information limiting unit 35 selects a camera as the sensor so that the color of the lighted traffic light can be determined. Then, the transmission information limiting unit 35 can extract a portion (pixels) including the light projecting unit S of the traffic light from the captured image (detection information) of the selected camera, and set the extracted extracted image A as detection information within the limited information range. Note that the position (height from the road surface, etc.) of the light projecting unit S of the traffic light may be included in the map information, for example.
[0064] Next, an example of setting the limited information range using the external situation obtained based on the external environment will be described. The transmission information limiting unit 35 extracts a portion to be transmitted to the remote instruction server 10 from the detection information detected by the external sensor 22 based on the external environment and the route recognized by the external environment recognition unit 32. Then, the transmission information limiting unit 35 can set the extracted portion of the detection information as detection information within the limited information range.
[0065] As an example, as shown in FIG. 4(a), a case where the autonomous vehicle 2 overtakes a stopped vehicle X1 that is stopped on the driving lane of the autonomous vehicle 2 will be described. In addition, here, the external sensor 22 is equipped with a sensor whose detection area is the entire periphery of the autonomous vehicle 2, or a sensor whose detection area is a 180° range in front of the autonomous vehicle 2 and a sensor whose detection area is a 180° range behind the autonomous vehicle 2. In this case, the transmission information limiting unit 35 determines that the autonomous vehicle 2 is in a situation where it is overtaking the stopped vehicle X1 based on the information (external environment) of the stopped vehicle X1 detected by the external environment recognition unit 32 and the course. Then, the transmission information limiting unit 35 can extract a portion including a region in front of the autonomous vehicle 2 and a portion including a region behind the autonomous vehicle 2 from the detection information of the external sensor 22, and set the detection information of the extracted portion as detection information within the limited information range. Here, the transmission information limiting unit 35 does not extract detection information of a portion including a region to the left and right of the autonomous vehicle 2 as detection information within the limited information range.
[0066] The transmission information limiting unit 35 may set the limited information range using the external situation obtained based on the external environment and map information. Specifically, as an example, a case where the autonomous vehicle 2 turns right at an intersection will be described as shown in FIG. 3. Here, the external sensor 22 is equipped with a sensor whose detection area is a 180° range in front of the autonomous vehicle 2. In this case, the transmission information limiting unit 35 extracts a portion of the detection information to be transmitted to the remote commander based on the presence or absence of a vehicle such as an oncoming vehicle X waiting to turn right in the oncoming lane of the autonomous vehicle 2.
[0067] Specifically, as an example, when turning right at an intersection, if the external environment recognition unit 32 has not recognized a vehicle such as oncoming vehicle X in front of the autonomous vehicle 2, the transmission information limiting unit 35 may not extract the portion in front of the autonomous vehicle 2 (the portion of detection area L1 in FIG. 3 ) from the detection information of the sensor, which has a detection area of 180° in front of the autonomous vehicle 2, but may extract only the portion including the right front (the portion of detection area L2 in FIG. 3 ), and set the detection information of the extracted portion as detection information within the limited information range.
[0068] 3, if a pedestrian about to cross the road after turning right is not recognized by the external environment recognition unit 32, the transmission information limiting unit 35 does not need to include a portion of the detection information including the right front as detection information within the limited information range. Note that, if the presence or absence of vehicles such as oncoming vehicle X and pedestrians crossing the road after turning right is not taken into consideration, the transmission information limiting unit 35 can extract a portion including the area in front and to the right front of the autonomous vehicle 2 from the detection information of a sensor having a detection area of 180° in front of the autonomous vehicle 2, and set the detection information of the extracted portion as detection information within the limited information range.
[0069] The information presence / absence determination unit 36 determines whether or not the detected information within the limited information range set by the transmission information limiting unit 35 includes necessary information that has been set in advance based on the external situation of the autonomously driven vehicle 2. In other words, this necessary information is set in advance based on the external environment of the autonomously driven vehicle 2 recognized by the external environment recognition unit 32 or map information.
[0070] The necessary information set based on the map information may be information that is set in advance according to a location where the autonomous vehicle 2 travels, such as an intersection. The necessary information in this case may include, for example, a direction or location (lane, road, etc.) that the remote commander R should check, and an object (traffic light, crosswalk, etc.) that the remote commander R should check. The necessary information set based on the external environment may be, for example, an object recognized by the external environment recognition unit 32 and set in advance according to an object that the remote commander R should check. The necessary information in this case may include, for example, an object that the remote commander R should check and that is not included in the map information (a traffic guide or a guide board in a construction zone, etc.).
[0071] As an example of necessary information preset based on map information, a case will be described in which the autonomous vehicle 2 goes straight through a narrow alley intersection where no traffic lights are installed, as shown in Fig. 6(a). In this case, when issuing a remote instruction, the remote commander R needs to check the situation of road R2 intersecting road R1 on which the autonomous vehicle 2 is traveling (the situation on the left and right of the autonomous vehicle 2). For this reason, for example, a road (road R2) intersecting the travel lane of the autonomous vehicle 2 is preset as necessary information for such a situation.
[0072] In the example shown in FIG. 6(a), the transmission information limiting unit 35 sets the detection information of the sensor that detects the right side of the autonomous vehicle 2 and the detection information of the sensor that detects the left side of the autonomous vehicle 2, among the multiple sensors that the external sensor 22 has, as detection information within the limited information range. The information presence / absence determining unit 36 determines whether or not the detection information detected by the external sensor 22 includes the situation of the road R2 based on the map information, the position information of the autonomous vehicle 2 acquired by the vehicle position acquiring unit 31, and the course. In the example shown in FIG. 6(a), fences W1 and W2 are installed on both sides of the road R1 on which the autonomous vehicle 2 travels. The detection area of the sensor that detects the right side of the autonomous vehicle 2 is shielded by the fence W1, and the detection area of the sensor that detects the left side is shielded by the fence W2. As a result, the situation of the road R2 cannot be detected by the sensor that detects the right side and the sensor that detects the left side. For this reason, the information presence / absence determining unit 36 determines that the detection information within the limited information range does not include necessary information.
[0073] As an example of necessary information that is preset based on the external environment, a case will be described in which a construction section K exists ahead of the autonomous vehicle 2 and a traffic guide P is performing traffic guidance just before the construction section K, as shown in FIG. 7. In this case, when issuing a remote instruction, the remote commander R needs to check the guidance action of the traffic guide P. For this reason, for example, the traffic guide P is preset as necessary information in such a situation. In addition, for example, a guidance guide board installed around the construction section K may be preset as necessary information in such a situation.
[0074] 7, it is assumed that a traffic director P is recognized by the external environment recognition unit 32. In this case, the information presence / absence determination unit 36 determines whether or not the traffic director P is included in the detected information within the limited information range set by the transmission information limiting unit 35.
[0075] When the information presence / absence determination unit 36 determines that the necessary information is not included in the detection information within the limited information range, the notification proposal unit 37 notifies the remote commander R. Alternatively, the notification proposal unit 37 proposes to the remote commander R to change the position and attitude of the autonomously driving vehicle 2 or to change the detection area of the external sensor 22. When making this notification and proposal, the notification proposal unit 37 transmits a notice to that effect to the remote instruction server 10, and can make the notification and proposal via the commander interface 3 of the remote instruction device 1.
[0076] Note that the change in the position and attitude of the autonomous vehicle 2 here refers to, for example, changing the position of the autonomous vehicle 2, or changing the position and orientation. Specifically, the change in the position and attitude of the autonomous vehicle 2 may refer to, for example, moving the autonomous vehicle 2 forward from the current location, or steering the autonomous vehicle 2 to the right or left while moving the autonomous vehicle 2 forward from the current location. As the position and attitude of the autonomous vehicle 2 changes, the detection area of the external sensor 22 changes.
[0077] Furthermore, changing the detection region of the external sensor 22 may include, for example, changing the mounting position and orientation of the sensor of the external sensor 22. Changing the mounting position and orientation of the sensor includes, for example, at least one of changing the mounting angle (sensor orientation) of the sensor of the external sensor 22 and rotating the sensor around the detection direction of the sensor as an axis. In other words, changing the mounting position and orientation includes changing at least one of the yaw angle, pitch angle, and roll angle of the sensor. Changing the mounting position and orientation may also include translating the position of the sensor (sliding in at least one of the horizontal and vertical directions).
[0078] Furthermore, changing the detection area of the external sensor 22 may mean, for example, changing a range to be extracted from the detection area of the external sensor 22. In other words, changing the detection area of the external sensor 22 may mean, for example, changing an interesting part to be extracted from the detection information.
[0079] The sensor angle of view change unit 38 changes the detection area of the external sensor 22. In this embodiment, when an instruction to change the detection area of the external sensor 22 is given from the remote commander R, the sensor angle of view change unit 38 changes the detection area of the external sensor 22 based on the instruction to change the detection area. Here, the sensor angle of view change unit 38 can fix the amount of change (amount of change in the detection area) per instruction to change the detection area. In this case, the remote commander R issues an instruction to change the detection area multiple times until the desired area is included in the detection area. This makes it possible to appropriately change the detection area of the external sensor 22 while preventing the detection area of the external sensor 22 from becoming an unintended state even if the instruction to change is delayed or interrupted.
[0080] When autonomously driving vehicle 2 is equipped with a remote instruction sensor for a remote commander, sensor angle of view change unit 38 may change the detection area of the remote instruction sensor.
[0081] When the remote instruction determination unit 34 determines that a remote instruction should be requested, the driving status information transmission unit 39 transmits driving status information of the autonomously driven vehicle 2 to the remote instruction server 10. The driving status information of the autonomously driven vehicle 2 includes information that enables the remote commander R to recognize the status of the autonomously driven vehicle 2. As described above, the range of the detection information of the external sensor 22 included in the driving status information is set to a limited information range by the transmission information limiting unit 35.
[0082] Specifically, the driving situation information of the autonomously driven vehicle 2 includes detection information from an on-board sensor of the autonomously driven vehicle 2 and / or information generated from the detection information from the on-board sensor (for example, an overhead image of the autonomously driven vehicle 2). The detection information from the on-board sensor includes detection information from the external sensor 22.
[0083] For example, the detection information of the external sensor 22 may include a camera image of the front of the autonomous vehicle 2 captured by a camera. The detection information of the external sensor 22 may include a camera image of the surroundings of the autonomous vehicle 2, including the sides and rear of the autonomous vehicle 2. The detection information of the external sensor 22 may also include object information detected by a radar sensor. The detection information of the external sensor 22 may include an identification result of the type of object. The driving situation information may include information on the external environment of the autonomous vehicle 2 recognized by the external environment recognition unit 32 based on the detection information of the external sensor 22.
[0084] Furthermore, the detection information of the on-board sensor may include detection information of the internal sensor 23. The detection information of the internal sensor 23 may include information on the vehicle speed of the autonomously driven vehicle 2 detected by a vehicle speed sensor. The detection information of the internal sensor 23 may include information on the yaw rate of the autonomously driven vehicle 2 detected by a yaw rate sensor. The detection information of the internal sensor 23 may include information on the steering angle of the autonomously driven vehicle 2. The driving situation information may include information on the driving state of the autonomously driven vehicle 2 recognized by the driving state recognition unit 33 based on the detection information of the internal sensor 23.
[0085] Furthermore, the driving status information of the autonomous vehicle 2 may include position information of the autonomous vehicle 2. The driving status information of the autonomous vehicle 2 may include information about occupants (presence or absence of occupants, number of occupants). The driving status information of the autonomous vehicle 2 may include information about a route according to a selectable remote instruction of the remote commander R. The route will be described later.
[0086] The trajectory generation unit 40 generates a trajectory to be used for autonomous driving of the autonomous vehicle 2. The trajectory generation unit 40 generates a trajectory for autonomous driving based on a preset target route, map information, position information of the autonomous vehicle 2, the external environment of the autonomous vehicle 2, and the driving state of the autonomous vehicle 2. The trajectory corresponds to a driving plan for autonomous driving.
[0087] The course includes a route [path] along which the vehicle travels in autonomous driving and a vehicle speed plan for autonomous driving. The route is a planned trajectory along which the vehicle is driven autonomously on a target route. The route can be, for example, data on steering angle changes (steering angle plan) of the autonomous vehicle 2 according to positions on the target route. The positions on the target route are, for example, set longitudinal positions set at predetermined intervals (for example, 1 m) in the travel direction of the target route. The steering angle profile is data in which a target steering angle is associated with each set longitudinal position.
[0088] The target route is set based on, for example, the destination, map information, and position information of the autonomous vehicle 2. The target route may be set further taking into consideration traffic information such as congestion. The target route may be set by a well-known navigation system. The destination may be set by a passenger of the autonomous vehicle 2, or may be automatically proposed by the autonomous driving ECU 20 or the navigation system.
[0089] The route generation unit 40 generates a route along which the autonomous vehicle 2 will travel, based on, for example, the target route, map information, the external environment of the autonomous vehicle 2, and the traveling state of the autonomous vehicle 2. The route generation unit 40 generates a route such that, for example, the autonomous vehicle 2 passes through the center (center in the lane width direction) of a lane included in the target route.
[0090] The vehicle speed plan is data in which a target vehicle speed is associated with each set longitudinal position, for example. The set longitudinal position may be set based on the driving time of the autonomous vehicle 2, rather than the distance. The set longitudinal position may be set as the position the vehicle will reach in 1 second, or the position the vehicle will reach in 2 seconds, for example. In this case, the vehicle speed plan can also be expressed as data according to the driving time.
[0091] The route generation unit 40 generates a vehicle speed plan based on traffic regulation information such as the legal maximum speed included in the route and map information. A speed preset for a position or section on the map may be used instead of the legal maximum speed. The route generation unit 40 generates a route for autonomous driving from the route and the vehicle speed plan. Note that the route generation method in the route generation unit 40 is not limited to the above-mentioned content, and any well-known method related to autonomous driving may be adopted. The same applies to the content of the route.
[0092] The route generation unit 40 generates a route in advance according to a remote instruction when a remote instruction is requested from the remote instruction server 10 by the remote instruction determination unit 34, or when the autonomous vehicle 2 approaches an intersection or the like that is the target of a remote instruction. The content of the remote instruction is determined in advance according to the situation of the autonomous vehicle 2. For example, the content of the remote instruction when turning right at an intersection includes a remote instruction to "proceed (start turning right)" and a remote instruction to "stop (pending decision)". The content of the remote instruction when turning right at an intersection may include a remote instruction to go straight without turning right (remote instruction to change route), or may include a remote instruction to make an emergency evacuation.
[0093] For example, in a situation where the autonomous vehicle 2 is to turn right at an intersection, the path generating unit 40 generates a path for the autonomous vehicle 2 to turn right at the intersection in response to a remote instruction to start turning right. The path generating unit 40 may update the path in response to changes in the external environment until the remote instruction is received. Furthermore, when there is a remote instruction to switch from turning right at the intersection to going straight at the intersection, the path generating unit 40 may generate in advance a path for going straight at the intersection.
[0094] When a remote instruction for emergency evacuation is present, the path generating unit 40 may generate a path for emergency evacuation in advance. The path for emergency evacuation is generated so as to stop the autonomous vehicle 2 in any of the evacuation spaces set in advance on the map. The path generating unit 40 recognizes the presence or absence of an obstacle in each evacuation space based on, for example, the external environment, and generates a path for emergency evacuation so as to stop the autonomous vehicle 2 in an empty evacuation space. Note that the path generating unit 40 does not necessarily need to generate a path in advance, and may generate a path corresponding to the remote instruction after receiving the remote instruction.
[0095] The autonomous driving control unit 41 executes autonomous driving of the autonomous vehicle 2. The autonomous driving control unit 41 executes autonomous driving of the autonomous vehicle 2, for example, based on the external environment of the autonomous vehicle 2, the driving state of the autonomous vehicle 2, and the route generated by the route generation unit 40. The autonomous driving control unit 41 executes autonomous driving of the autonomous vehicle 2 by sending a control signal to the actuator 26.
[0096] When the driving status information transmission unit 39 requests the remote instruction server 10 to send a remote instruction, the autonomous driving control unit 41 waits to receive the remote instruction from the remote instruction server 10. When the autonomous driving control unit 41 requests a remote instruction after the autonomous driving vehicle 2 has stopped, the autonomous driving control unit 41 maintains the stopped state until it receives the remote instruction.
[0097] When a licensed occupant is on board, the automatic driving control unit 41 may request the occupant to make a decision or to drive manually if a remote instruction is not received even after a preset waiting time has elapsed. When a remote instruction is not received even after the waiting time has elapsed and the occupant is unable to make a decision or to drive manually (such as when there is no occupant on board), the automatic driving control unit 41 may automatically perform an emergency evacuation.
[0098] In addition, when an instruction to change the position and attitude of the autonomous vehicle 2 is given from the remote commander R after the driving situation information transmission unit 39 transmits driving situation information to the remote instruction server 10 to request a remote instruction, the autonomous driving control unit 41 changes the position and attitude of the autonomous vehicle 2 based on the instruction to change the position and attitude. Here, the autonomous driving control unit 41 can fix the amount of change per instruction to change the position and attitude of the autonomous vehicle 2 (the amount of change in the attitude (direction, position) of the autonomous vehicle 2). In this case, the remote commander R issues an instruction to change the position and attitude of the autonomous vehicle 2 multiple times until the desired area is included in the detection area. This makes it possible to appropriately change the position and attitude of the autonomous vehicle 2 while preventing the position and attitude of the autonomous vehicle 2 from becoming an unintended state even if a delay or interruption occurs in the change instruction. Note that the driving situation information transmission unit 39 transmits the changed driving situation information to the remote instruction server 10 after an instruction to change the position and attitude of the autonomous vehicle 2 is issued.
[0099] (Configuration of remote control device) The configuration of the remote instruction device 1 according to this embodiment will be described below with reference to the drawings. As shown in FIG.
[0100] First, a hardware configuration of the remote instruction server 10 will be described. Fig. 8 is a block diagram showing an example of the hardware configuration of the remote instruction server 10. As shown in Fig. 8, the remote instruction server 10 is configured as a general computer including a processor 10a, a storage unit 10b, a communication unit 10c, and a user interface 10d. The user in this case means a user (administrator, etc.) of the remote instruction server 10.
[0101] The processor 10a operates various operating systems to control the remote instruction server 10. The processor 10a is a computing unit such as a CPU including a control device, an arithmetic unit, a register, etc. The processor 10a controls the storage unit 10b, the communication unit 10c, and the user interface 10d. The storage unit 10b is configured to include at least one of a memory and a storage. The memory is a recording medium such as a ROM or a RAM. The storage is a recording medium such as a HDD.
[0102] The communication unit 10c is a communication device for communicating via the network N. A network device, a network controller, a network card, etc. can be used for the communication unit 10c. The user interface 10d is an input / output unit of the remote instruction server 10 for a user such as an administrator. The user interface 10d includes an output device such as a display and a speaker, and an input device such as a touch panel. The remote instruction server 10 does not necessarily need to be provided in a facility, and may be mounted on a moving object such as a vehicle.
[0103] Fig. 9 is a block diagram showing an example of the configuration of the remote instruction device 1. As shown in Fig. 9, the commander interface 3 is an input / output section of the remote instruction device 1 for the remote commander R. The commander interface 3 has an output section 3a and an instruction input section 3b.
[0104] The output unit 3a is a device that outputs information used for remotely instructing the autonomously driven vehicle 2 to the remote commander R. The output unit 3a includes a display that outputs images and a speaker that outputs sounds.
[0105] As an example, the display displays an image of the view ahead of the autonomous vehicle 2 captured by a camera of the autonomous vehicle 2 (an image of the scenery ahead). The display may have multiple display screens, and may display images of the sides and / or rear of the autonomous vehicle 2. The display is not particularly limited as long as it has a configuration capable of providing visual information to the remote commander R. The display may be a wearable device that is worn over the eyes of the remote commander R.
[0106] The speaker is, for example, a headset speaker worn on the head of the remote commander R. The speaker, for example, conveys the situation of the autonomous vehicle 2 (for example, the situation such as turning right at an intersection) by voice to the remote commander R. The speaker does not necessarily have to be a headset, and may be a fixed type.
[0107] The output unit 3a may provide information to the remote commander R by vibration. The output unit 3a may have, for example, a vibration actuator provided on the seat of the remote commander R. The output unit 3a may alert the remote commander R to the approach of another vehicle to the autonomous vehicle 2 by vibration. The output unit 3a may have a vibration actuator on each of the left and right sides of the seat, and may vibrate the vibration actuator at a position according to the approaching direction of the other vehicle. The output unit 3a may have a wearable vibration actuator attached to the body of the remote commander R. The output unit 3a can provide information to the remote commander R by vibrating the vibration actuators attached to each position of the body according to the approaching direction of the other vehicle.
[0108] The instruction input unit 3b is a device to which a remote instruction is input by a remote commander R. Here, Fig. 10 is a diagram showing an example of the instruction input unit 3b. The instruction input unit 3b shown in Fig. 10 employs a gate-type lever structure. Fig. 10 shows a lever La, a monitoring start button Lb, and a cross groove Lc.
[0109] The lever La is a lever operated by the remote commander R. The lever La has, for example, a grip portion at the upper end and a shaft portion extending from the grip portion toward a cross groove (cross gate) Lc. A monitoring start button Lb is provided on the side of the grip portion of the lever La. The position of the monitoring start button Lb is not particularly limited, and it may be on the left side or right side of the grip portion. The monitoring start button Lb may be provided on the same surface as the cross groove Lc.
[0110] The monitoring start button Lb is a button that is pressed when the remote commander R starts monitoring the status of the autonomously driven vehicle 2. The remote instruction device 1 may recognize that the remote commander R has started monitoring when the monitoring start button Lb is pressed. The monitoring start button Lb also functions as an unlock button for the lever La. In other words, the lever La is unlocked and can be moved only while the monitoring start button Lb is pressed or for a certain period of time after the monitoring start button Lb is pressed. It should be noted that the monitoring start button Lb does not necessarily have to be provided.
[0111] The cross groove Lc is a groove into which the shaft of the lever La fits and through which the lever La moves in response to the operation of the remote commander R. In the instruction input unit 3b shown in Fig. 10, a remote instruction is input by switching the position of the lever La along the cross groove Lc. As shown in Fig. 10, for example, the upward direction of the cross groove Lc can be used to input the instruction "Go", the downward direction can be used to input the instruction "Stop", the right direction can be used to input the instruction "Change lane", and the left direction can be used to input the instruction "Cancel".
[0112] The remote commander R, for example, by pressing the monitoring start button Lb and moving the lever La upwards, inputs a remote command to "proceed" to the autonomously driven vehicle 2. The remote commander R, for example, by pressing the monitoring start button Lb and moving the lever La downwards, inputs a remote command to "stop" to the autonomously driven vehicle 2. If the remote commander R wishes to cancel the previous remote command, he or she inputs a command to "cancel" by pressing the monitoring start button Lb and moving the lever La leftwards.
[0113] Note that the indication of "proceed" or the like in the cross groove Lc may be a changeable digital indication. The indications of "proceed", "change lane", etc. may be changed depending on the remote instruction situation of the autonomously driven vehicle 2. For example, "change lane" may be displayed as "overtake" depending on the situation. In this case, by moving the lever La to the right, a remote instruction to overtake can be given to the autonomously driven vehicle 2.
[0114] The remote instruction to "stop" may be replaced with "pending judgment." In the case of "stop," the autonomous vehicle 2 stops regardless of its position, but in the case of "pending judgment," autonomous driving continues until a position where a remote instruction is required (for example, a stop line in front of a traffic light). The instruction input unit 3b may be capable of inputting "stop" and "pending judgment" separately. In the case where a monitoring start button Lb is present, the remote commander R may continue to press the monitoring start button Lb, which may be treated as an instruction to "pending judgment."
[0115] Also, instead of a cross groove, a straight groove that allows the selection of two remote command inputs, proceed and stop (or hold), may be used, or a stepped groove used in a vehicle shift lever may be used. In addition, a separate button for emergency evacuation may be provided. Emergency evacuation may be one of the remote commands that can be selected by operating the lever La.
[0116] In addition, various input methods can be adopted for the instruction input unit 3b. For the instruction input unit 3b, a button, a touch panel, or various switches such as a toggle switch or a rocker switch may be adopted. For the instruction input unit 3b, a keyboard or a voice input device may be adopted. For the instruction input unit 3b, a button cover may be attached to prevent erroneous operation. For the instruction input unit 3b, a button and a switch may be used in combination, a button and a handle may be used in combination, or a pedal and a lever may be used in combination, in order to prevent erroneous operation. For the instruction input unit 3b, remote instructions may be input by combining two or more of lever operation, button operation, touch panel operation, pedal operation, and voice input.
[0117] In addition, in the case of virtual buttons such as a touch panel, the display position of the button may not be fixed to prevent the remote commander R from reflexively operating the button without fully understanding the situation of the autonomously driven vehicle 2. The remote commander R may be notified of the contents of the remote instruction (proceed, stop, etc.) input by the remote commander R by voice and / or image display. The image display may be a text display, and the remote commander R may be notified by a change in color.
[0118] An Enable button may be used for the instruction input unit 3b. In this case, the Enable button is a button that does not transmit a signal unless the pressing amount is within a set range, and does not transmit a signal when the pressing amount is light or deep. A two-stage switch that changes the information transmitted when the button is pressed lightly and when the button is pressed all the way down may be used for the instruction input unit 3b. A dial (e.g., a rotary switch) that selects the remote instruction content by rotating a disk-shaped rotating disk may be used for the instruction input unit 3b. The dial has multiple notches, and each notch corresponds to a remote instruction content such as "proceed" or "stop."
[0119] Next, a description will be given of the functional configuration of the remote instruction server 10. As shown in Fig. 9, the remote instruction server 10 includes a remote instruction request receiving unit 11, an information providing unit 12, and a remote instruction transmitting unit 13.
[0120] The remote instruction request receiving unit 11 receives a request for a remote instruction when the autonomously driven vehicle 2 requests a remote instruction from the remote instruction server 10. In addition, the remote instruction request receiving unit 11 acquires driving status information of the autonomously driven vehicle 2 that has requested a remote instruction, by transmission from the autonomously driven vehicle 2. Note that the remote instruction request receiving unit 11 may also acquire driving status information of an autonomously driven vehicle 2 that has not requested a remote instruction.
[0121] The information providing unit 12 provides various types of information to the remote commander R. When the remote instruction request receiving unit 11 receives a remote instruction request, the information providing unit 12 requests the remote commander R in charge to input a remote instruction via the commander interface 3.
[0122] Further, the information providing unit 12 provides information on the autonomous vehicle 2 to the remote commander R based on the driving situation information of the autonomous vehicle 2 acquired by the remote instruction request receiving unit 11. The information providing unit 12 displays an image of the autonomous vehicle 2 in front of the autonomous vehicle 2 on the display of the output unit 3a of the commander interface 3, for example. The information providing unit 12 may display an image seen from the vicinity of the driver's seat of the autonomous vehicle 2 by viewpoint conversion. The information providing unit 12 may display images of the sides and rear of the autonomous vehicle 2. The information providing unit 12 may display a panoramic image obtained by synthesizing images of the surroundings of the autonomous vehicle 2, or may display an overhead image generated by image synthesis and viewpoint conversion so as to look down on the autonomous vehicle 2. The information providing unit 12 may highlight objects in the image (for example, displaying other vehicles, etc., surrounded by a frame). When a traffic light is included in the image, the information providing unit 12 may display the recognition result of the lighting state of the traffic light on the display.
[0123] The information providing unit 12 may display various information on the display, not limited to camera images captured by the camera of the autonomous vehicle 2. The information providing unit 12 may display the status of the autonomous vehicle 2 that requested the remote instruction (such as when turning right at an intersection, when offsetting to avoid an obstacle, etc.) using text, icons, or the like. The information providing unit 12 may display the types of remote instructions (proceed, wait, etc.) that the remote commander R can select on the display. The information providing unit 12 may display information on the path of the autonomous vehicle 2 in response to the remote instruction (such as the trajectory of the autonomous vehicle 2 in response to the remote instruction to proceed) on the display.
[0124] The information providing unit 12 may display information about an object detected by a radar sensor of the autonomously driven vehicle 2. The information about the object may be displayed as an icon in the overhead image. When the type of the object is identified, an icon according to the type of the object may be displayed. The information providing unit 12 may display map information about the surroundings of the autonomously driven vehicle 2, which is acquired based on the position information of the autonomously driven vehicle 2, on the display. The map information may be held by the remote instruction server 10, or may be acquired from another server or the like. The map information about the surroundings of the autonomously driven vehicle 2 may be acquired from the autonomously driven vehicle 2.
[0125] The information providing unit 12 may display on a display the road traffic information acquired based on the position information of the autonomously driven vehicle 2. The road traffic information includes at least one of congestion sections, construction section information, accident position information, etc. The road traffic information can be acquired from, for example, a traffic information center.
[0126] The information providing unit 12 may display information about the vehicle speed of the autonomously driven vehicle 2 on the display, and may display information about the steering angle of the autonomously driven vehicle 2 on the display. The information providing unit 12 may display information about the gradient of the road on which the autonomously driven vehicle 2 is located on the display. If the autonomously driven vehicle 2 has an interior camera, the information providing unit 12 may display an image of the interior of the autonomously driven vehicle 2 as necessary. The information providing unit 12 may display the passenger riding status and / or luggage loading status of the autonomously driven vehicle 2 on the display.
[0127] The information providing unit 12 provides sound information to the remote commander R through a speaker of the output unit 3a of the commander interface 3. The information providing unit 12 may output the situation of the autonomous vehicle 2 (when turning right at an intersection, when offsetting to avoid an obstacle, etc.) as sound from the speaker. The information providing unit 12 may output the approach of other vehicles around the autonomous vehicle 2 as sound or voice from the speaker. The information providing unit 12 may output the sound (noise) around the autonomous vehicle 2 as it is from the speaker. The information providing unit 12 may output the voice of an occupant in the vehicle cabin from the speaker as necessary. Note that providing information through a speaker is not essential.
[0128] Additionally, when the output unit 3a has a vibration actuator, the information providing unit 12 may provide information to the remote commander R by vibration. In this case, the information providing unit 12 can provide information (attention) to the remote commander R by vibrating a vibration actuator at a position corresponding to a direction in which attention should be paid, such as the approaching direction of another vehicle to the autonomously driven vehicle 2 or the direction in which a pedestrian is present.
[0129] As described above, the range of the detection information of the external sensor 22 included in the driving situation information is set to a limited information range based on the external situation. In this case, the information providing unit 12 provides the remote commander R with information on the autonomous vehicle 2 based on the driving situation information including the detection information set to the limited information range. In this way, even if the range of the detection information of the external sensor 22 is set to the limited information range, the detection information includes a detection area that includes an area that should be confirmed based on the external situation. Therefore, the remote commander R can make a decision on a remote instruction based on the driving situation information including the detection information set to the limited information range.
[0130] Furthermore, based on an instruction from the notification suggestion unit 37, the information providing unit 12 uses the output unit 3a of the commander interface 3 to notify the remote commander R that the necessary information is not included in the detection information within the limited information range. Alternatively, based on an instruction from the notification suggestion unit 37, the information providing unit 12 uses the output unit 3a of the commander interface 3 to present to the remote commander R the contents of a proposal to change the position and attitude of the autonomously driven vehicle 2 or to change the detection area of the external sensor 22.
[0131] When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction transmission unit 13 transmits the input remote instruction to the autonomously driven vehicle 2. When the information provision unit 12 transmits the remote instruction input by the remote commander R to the autonomously driven vehicle 2, it may continue to transmit information about the autonomously driven vehicle 2 to the remote commander R, or may switch to information about another autonomously driven vehicle 2 requesting a remote instruction.
[0132] Furthermore, when the remote commander R issues an instruction to change the position and attitude of the autonomous vehicle 2 or an instruction to change the detection area of the external sensor 22 to the output unit 3a of the commander interface 3, the remote instruction transmission unit 13 transmits the input change instruction to the autonomous vehicle 2. The remote commander R can issue an instruction to change the position and attitude of the autonomous vehicle 2, etc., based on a notification by the information providing unit 12 using the output unit 3a that the necessary information is not included in the detection information within the limited information range. Alternatively, the remote commander R can issue an instruction to change the position and attitude of the autonomous vehicle 2, etc., based on the proposed content of a change in the position and attitude of the autonomous vehicle 2, etc., by the information providing unit 12 using the output unit 3a. The remote commander R can also issue an instruction to change the position and attitude of the autonomous vehicle 2, etc., when it checks the information provided based on the driving situation information of the autonomous vehicle 2 and determines that the necessary information is not included.
[0133] (Processing flow when a remote instruction target situation occurs) Next, a flow of processing performed by the autonomous driving ECU 20 of the autonomous driving vehicle 2 when a remote instruction target situation occurs will be described with reference to the flowchart in Fig. 11. Note that the processing shown in Fig. 11 is started when the remote instruction determination unit 34 determines that a remote instruction target situation requiring a remote instruction has occurred.
[0134] 11, when the remote instruction determination unit 34 determines that the situation has become a remote instruction target situation, the autonomous driving ECU 20 executes an operation to be performed when the remote instruction target situation occurs (S101). In the present embodiment, as an example, when the remote instruction target situation occurs, the autonomous driving ECU 20 stops the autonomous vehicle 2 at a position according to the remote instruction target situation. For example, when the remote instruction target situation is a situation in which the autonomous vehicle enters an intersection, the autonomous driving ECU 20 may stop the autonomous vehicle 2 at the position of a stop line set up at the intersection, and when the remote instruction target situation is a situation in which a stopped vehicle ahead has been detected, the autonomous driving ECU 20 may stop the autonomous vehicle 2 at a position a predetermined distance before the stopped vehicle.
[0135] Next, the remote instruction determination unit 34 transmits a remote instruction request to the remote commander R (remote instruction server 10) (S102). The transmission information limiting unit 35 sets the range of information to be transmitted to the remote commander R, among the detection information detected by the external sensor 22, to a limited information range. Then, the driving status information transmitting unit 39 transmits driving status information of the autonomously driven vehicle 2, including the detection information whose information range has been set to the limited information range, to the remote instruction server 10 (S103).
[0136] When a remote instruction request is made from the autonomously driven vehicle 2, the remote commander R issues a remote instruction to the autonomously driven vehicle 2 based on the driving conditions of the autonomously driven vehicle 2 transmitted from the driving condition information transmission unit 39. When the autonomously driven control unit 41 receives a remote instruction from the remote commander R (S104), it executes an operation (e.g., start entering an intersection, start turning right, etc.) according to the remote instruction (S105).
[0137] Below, specific examples of each process performed by the autonomous vehicle 2 and the remote commander R when various remote instruction target situations occur will be described.
[0138] (Turn right at the intersection) As an example, a case will be described in which the autonomous vehicle 2 turns right at an intersection as shown in FIG. (Process A1) The remote instruction determination unit 34 recognizes that the autonomous vehicle 2 has reached an intersection where it needs to turn right (this is a situation that is subject to remote instruction) based on the position information of the autonomous vehicle 2, the map information in the map database 24, and the route generated by the route generation unit 40. (Process A2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stop position. The stop position here may be the position of a stop line painted on the road surface, or may be a predetermined position other than a stop line.
[0139] (Process A3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process A4) As described above, when turning right at an intersection, the remote commander R, for example, needs to confirm the absence of both a vehicle traveling straight in the oncoming lane and a pedestrian crossing the road to which the vehicle is to turn right when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects, among the sensors of the external sensor 22, a sensor whose detection area includes the front of the autonomous vehicle 2 and a sensor whose detection area includes the right front, and sets only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information and the course, to the limited information range.
[0140] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0141] Here, in step A4, the transmission information limiting unit 35 may further set the limited information range based on the recognition result of the external environment recognition unit 32. Specifically, for example, the limited information range may be set based on the presence or absence of an oncoming vehicle X waiting to turn right. Here, as described above, as an example, when an oncoming vehicle X waiting to turn right in front of the autonomous vehicle 2 is not recognized by the external environment recognition unit 32, the transmission information limiting unit 35 may not include detection information of a sensor having a detection area in the forward direction among the multiple sensors of the external sensor 22 in the limited information range, and may set only detection information of a sensor having a detection area in the right forward direction as detection information within the limited information range. Also, in the case shown in FIG. 3, when a pedestrian about to cross the road to be turned right is not recognized by the external environment recognition unit 32, the transmission information limiting unit 35 may not include detection information of a sensor having a detection area in the right forward direction as detection information within the limited information range.
[0142] (Process A5) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process A6) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts turning right.
[0143] (Turning right at an intersection: angle of view changes) As an example, a case will be described in which, as shown in FIG. 12(a), when the autonomous vehicle 2 turns right at an intersection, the position and attitude of the autonomous vehicle 2 or the detection area of the external sensor 22 is changed. (Process B1) The remote instruction determination unit 34 recognizes that the autonomous vehicle 2 has reached an intersection at which to turn right (this is a remote instruction target situation), in the same manner as in step A1 above. (Process B2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0144] (Process B3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process B4) The transmission information limiting unit 35 sets the range of information to be transmitted to the remote commander R to the limited information range, similar to A4 above. Then, the driving status information transmitting unit 39 transmits the driving status information of the autonomous vehicle 2, including the detection information whose range is set to the limited information range, to the remote instruction server 10. Note that here, the transmission information limiting unit 35 sets only the detection information of the sensor whose detection area includes the front of the autonomous vehicle 2 as the detection information within the limited information range. However, similar to step A4 above, the detection information of the sensor whose detection area includes the right front may also be included as the detection information within the limited information range so that pedestrians crossing the road at the destination of the right turn can be confirmed.
[0145] (Process B5) The remote commander R checks the detection information within the limited information range contained in the driving condition information received from the autonomous vehicle 2, and if desired information such as necessary information is not included, it gives the autonomous vehicle 2 at least one of an instruction to change the position and attitude of the autonomous vehicle 2 and an instruction to change the detection area of the external sensor 22.
[0146] In addition, when a notification is given via the commander interface 3 of the remote instruction device 1 that the necessary information is not included in the detection information within the limited information range, the remote commander R may give an instruction to change the position and attitude of the autonomous vehicle 2 based on this notification. In addition, when a proposal to change the position and attitude of the autonomous vehicle 2 or to change the detection area of the external sensor 22 is given via the commander interface 3 of the remote instruction device 1, the remote commander R may give an instruction to change the position and attitude of the autonomous vehicle 2 based on this proposal. In addition, when no notification or proposal is given, the remote commander R may give an instruction to change the position and attitude of the autonomous vehicle 2 based on the confirmation result of the driving situation information. The necessary information here may be, for example, an oncoming straight lane located behind the oncoming vehicle X waiting to turn right in front of the autonomous vehicle 2, or a crosswalk ahead of the right turn.
[0147] (Process B6) When the remote commander R issues an instruction to change the position and attitude of the autonomously driven vehicle 2, the autonomously driven control unit 41 changes the position and attitude of the autonomously driven vehicle 2 based on the position and attitude change instruction. Here, for example, as shown in Fig. 12(b), the remote commander R moves the autonomously driven vehicle 2 forward from its current position while steering it to the right and stopping it so that the oncoming straight lane (necessary information) located behind the oncoming vehicle X waiting to turn right in front of the autonomously driven vehicle 2 can be detected without being blocked by the oncoming vehicle X.
[0148] Furthermore, when an instruction to change the detection area of the external sensor 22 is given by the remote commander R, the sensor angle of view change unit 38 changes the detection area of the external sensor 22 based on the instruction to change the detection area. Here, for example, as shown in Fig. 12(c), the remote commander R translates the position of the sensor whose detection area is in front of the autonomous vehicle 2 to a position on the right side of the autonomous vehicle 2 so that the oncoming straight lane (necessary information) located behind the oncoming vehicle X waiting to turn right in front of the autonomous vehicle 2 can be detected as much as possible without being blocked by the oncoming vehicle X. This makes it easier for the sensor that detects in front of the autonomous vehicle 2 to detect the oncoming straight lane adjacent to the oncoming right turn lane in which the oncoming vehicle X is located.
[0149] (Process B7) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process B8) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts turning right.
[0150] (Turn left at the intersection) As an example, a case will be described in which an autonomous vehicle 2 turns left at an intersection as shown in FIG. (Process C1) The remote instruction determination unit 34 recognizes that the autonomous vehicle 2 has reached an intersection at which to turn left (this is a remote instruction target situation), in the same manner as in step A1 above. (Process C2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0151] (Process C3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process C4) When turning left at an intersection, for example, when issuing a remote instruction, the remote commander R needs to confirm the absence of both pedestrians crossing the road to which the autonomous vehicle 2 is to turn left and motorcycles or the like approaching from behind that may be hit when turning left. For this reason, for example, the transmission information limiting unit 35 selects, from among the sensors of the external sensor 22, a sensor whose detection area includes the left side of the autonomous vehicle 2 and a sensor whose detection area includes the rear of the autonomous vehicle 2, and sets only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information and the route, as the limited information range.
[0152] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0153] Here, in step C4, the transmission information limiting unit 35 may further set the limited information range based on the recognition result of the external environment recognition unit 32. Specifically, for example, the limited information range may be set based on the presence or absence of a following vehicle (another vehicle behind the autonomous vehicle 2) of the autonomous vehicle 2. Here, for example, if a following vehicle exists behind the autonomous vehicle 2, an area is blocked by the following vehicle when the rear of the autonomous vehicle 2 is detected by the external sensor 22. This reduces the recognition accuracy of motorcycles, bicycles, etc. passing by the left side of the autonomous vehicle 2 from behind. In such a case, it is preferable that the autonomous vehicle 2 travels based on a remote instruction from the remote commander R. However, if there is no following vehicle behind the autonomous vehicle 2, the autonomous vehicle 2 can automatically check the rear and make a left turn by autonomous driving. Therefore, as an example, when the external environment recognition unit 32 does not recognize a following vehicle behind the autonomous vehicle 2, the transmission information limiting unit 35 may not include detection information from the multiple sensors of the external sensor 22 that have a detection area behind the vehicle in the limited information range, and may set only detection information from the sensor that has a detection area to the left as detection information within the limited information range.
[0154] Similarly, in the case shown in Figure 13, if a pedestrian attempting to cross the road after turning left is not recognized by the external environment recognition unit 32, the transmission information limiting unit 35 does not need to include the detection information of the sensor whose detection area is to the left as detection information within the limited information range.
[0155] In the example shown in Figure 13, as an example, the transmission information limiting unit 35 sets the detection information to the left and rear as the detection information within the limited information range, but the detection information from a sensor whose detection area is in front of the autonomous vehicle 2 may also be included in the detection information within the limited information range.
[0156] (Step C5) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Step C6) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts turning left.
[0157] (Go straight at the intersection: There is a traffic light) As an example, a case will be described in which an autonomous vehicle 2 travels straight through an intersection where traffic lights are installed, as shown in FIG. (Process D1) The remote instruction determination unit 34 recognizes that the autonomous vehicle 2 has reached an intersection where the autonomous vehicle 2 can travel straight ahead (this is a remote instruction target situation), in the same manner as in step A1 above. (Process D2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0158] (Process D3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process D4) As described above, when the autonomous vehicle 2 travels straight through an intersection where a traffic light is installed, the remote commander R, for example, needs to check the light information of the traffic light when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor that includes the traffic light in its detection area from among the multiple sensors that the external sensor 22 has. Here, as described above, the transmission information limiting unit 35 selects a camera as the sensor. Then, the transmission information limiting unit 35 extracts a portion that includes the light projecting portion S of the traffic light from the captured image (detection information) of the selected camera, and sets the extracted extracted image A as detection information within the limited information range.
[0159] That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R, among the detection information detected by the external sensor 22, to a limited information range, based on the map information and the route. Then, the driving status information transmitting unit 39 transmits to the remote instruction server 10 the driving status information of the autonomously driving vehicle 2, including the detection information whose information range has been set to the limited information range.
[0160] Note that, instead of extracting only the portion including the light projector S from the captured image of the camera, the transmission information limiting unit 35 selects only the captured image (detection information) of a camera capable of distinguishing color from among cameras, radars, and lidars whose detection areas are in front of the autonomously driving vehicle 2. Then, the transmission information limiting unit 35 may set the captured image of the selected camera as detection information within the limited information range.
[0161] (Process D5) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process D6) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts moving straight ahead.
[0162] In addition, when going straight through an intersection where a traffic light is installed, a request for a remote instruction from the remote commander R may be made only in a predetermined case. For example, the request for a remote instruction may be made only in a case where it is difficult for the external environment recognition unit 32 to recognize the color of the traffic light due to backlighting, fog, or the like.
[0163] (Go straight at the intersection: no traffic lights) As an example, a case will be described in which autonomous vehicle 2 travels straight through a narrow alley intersection where no traffic lights are installed, as shown in FIG. 6(a). (Process E1) The remote instruction determination unit 34 recognizes that the autonomous vehicle 2 has reached an intersection where the autonomous vehicle 2 can travel straight ahead (this is a remote instruction target situation), in the same manner as in step A1 above. (Process E2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0164] (Process E3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process E4) As described above, when the autonomous vehicle 2 goes straight through an intersection of narrow alleys, the remote commander R, for example, needs to check the situation of the road R2 intersecting the road R1 on which the autonomous vehicle 2 is traveling (the situation on the left and right of the autonomous vehicle 2) when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor that detects the right side and a sensor that detects the left side of the autonomous vehicle 2 from among the multiple sensors included in the external sensor 22, and sets only the detection information detected by the selected sensors as detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information and the route, to the limited information range.
[0165] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0166] (Process E5) The remote commander R checks the detection information within the limited information range included in the driving situation information received from the autonomous vehicle 2, and if desired information such as necessary information is not included, it instructs the autonomous vehicle 2 to change the position and attitude of the autonomous vehicle 2. In the example shown in Fig. 6(a), the detection area of the sensor that detects the right and the detection area of the sensor that detects the left are blocked by fences W1 and W2, so the remote commander R instructs the autonomous vehicle 2 to change the position and attitude. However, the remote commander R may also instruct the autonomous vehicle 2 to change the detection area of the external sensor 22.
[0167] When a notification is given via the commander interface 3 of the remote instruction device 1 that the necessary information is not included in the detection information within the limited information range, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this notification. When a proposal to change the position and attitude of the autonomously driven vehicle 2 is given via the commander interface 3 of the remote instruction device 1, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this proposal. When no notification or proposal is given, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on the confirmation result of the driving situation information.
[0168] (Process E6) When the remote commander R issues an instruction to change the position and attitude of the autonomously driven vehicle 2, the autonomously driven control unit 41 changes the position and attitude of the autonomously driven vehicle 2 based on the instruction to change the position and attitude. Here, for example, as shown in Fig. 6(b), the remote commander R moves the autonomously driven vehicle 2 forward from its current location and stops it so that the detection areas of the sensors that detect the right and the left are not blocked by the fences W1 and W2 (so that the blocked areas are reduced).
[0169] (Process E7) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process E8) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts moving straight ahead.
[0170] (Entering a roundabout) As an example, a case will be described in which an autonomous vehicle 2 enters a roundabout as shown in Fig. 14. Note that in this case, the vehicle travels clockwise in the roundabout. (Process F1) The remote instruction determination unit 34 recognizes that the autonomously driven vehicle 2 has reached the roundabout (is in a remote instruction target situation), similarly to step A1 above. (Process F2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0171] (Process F3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process F4) When entering a roundabout, the remote commander R needs to check other vehicles circling the roundabout when issuing a remote instruction, for example. For this reason, for example, the transmission information limiting unit 35 selects a sensor whose detection area includes the right front of the autonomous vehicle 2 from among the sensors possessed by the external sensor 22, and sets only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information and the route, to the limited information range.
[0172] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0173] (Process F5) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process F6) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts entering the roundabout while turning left.
[0174] (Overtaking a stopped vehicle) As an example, a case will be described in which the autonomous vehicle 2 overtakes a stopped vehicle X1 (obstacle) that is stopped on the driving lane of the autonomous vehicle 2, as shown in FIG. 4(a). (Process G1) The external environment recognition unit 32 recognizes a stopped vehicle X1 on the driving lane of the autonomous vehicle 2. The remote instruction determination unit 34 recognizes, based on the stopped vehicle X1 recognized by the external environment recognition unit 32 and the path generated by the path generation unit 40, that the autonomous vehicle 2 is in a situation to overtake the stopped vehicle X1 (a situation subject to remote instruction). (Process G2) The autonomous driving ECU 20 stops the autonomous vehicle 2 at a predetermined stop position. Here, for example, the autonomous driving ECU 20 stops the autonomous vehicle 2 at a position a predetermined distance before the stopped vehicle X1.
[0175] (Process G3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process G4) As described above, when the remote instruction device 1 overtakes the stopped vehicle X1, the remote commander R, for example, needs to confirm the absence of both a vehicle traveling straight in the oncoming lane and a vehicle attempting to overtake the stopped vehicle from behind the autonomous vehicle 2 (in the case of a road with one lane in each direction) when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor that detects the front of the autonomous vehicle 2 and a sensor that detects the rear of the autonomous vehicle 2 from among the multiple sensors that the external sensor 22 has, and sets only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the external environment (the recognized stopped vehicle X1) and the course, to the limited information range.
[0176] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0177] Here, in step G4, the transmission information limiting unit 35 may further set the limited information range based on the recognition result of the external environment recognition unit 32. Specifically, for example, the limited information range may be set based on the presence or absence of a following vehicle (another vehicle behind the autonomous vehicle 2) of the autonomous vehicle 2. Here, for example, if a following vehicle exists behind the autonomous vehicle 2, an area blocked by the following vehicle occurs when the rear of the autonomous vehicle 2 is detected by the external sensor 22. This reduces the recognition accuracy of a vehicle moving straight ahead behind the autonomous vehicle 2 (a vehicle traveling in an adjacent lane to the right of the lane to which the autonomous vehicle 2 is traveling) that attempts to overtake the autonomous vehicle 2 from behind by passing to the right of the autonomous vehicle 2. In such a case, it is preferable that the autonomous vehicle 2 travels based on a remote instruction from the remote commander R. However, if there is no following vehicle behind the autonomous vehicle 2, the autonomous vehicle 2 can automatically check the vehicle moving straight ahead behind the autonomous vehicle 2 by autonomous driving and overtake the stopped vehicle X1. Therefore, as an example, when the external environment recognition unit 32 does not recognize a following vehicle behind the autonomous vehicle 2, the transmission information limiting unit 35 may not include sensor detection information from the multiple sensors possessed by the external sensor 22 that have a rear detection area in the limited information range, and may set only detection information from sensors that have a forward detection area as detection information within the limited information range.
[0178] (Process G5) The remote commander R checks the detection information within the limited information range contained in the driving condition information received from the autonomous vehicle 2, and if desired information such as necessary information is not included, it gives the autonomous vehicle 2 at least one of an instruction to change the position and attitude of the autonomous vehicle 2 and an instruction to change the detection area of the external sensor 22.
[0179] Here, as an example, when issuing a remote instruction, the remote commander R needs to check the situation of the oncoming lane ahead of the autonomous vehicle 2 (the adjacent lane to the right of the lane the autonomous vehicle 2 is traveling on). For this reason, the oncoming lane is set in advance as necessary information for such a situation. In the example shown in FIG. 4(a), the remote commander R may issue an instruction to change the position and attitude of the autonomous vehicle 2, for example, in a case where a large part of the detection area of a sensor that detects the front of the autonomous vehicle 2 is blocked by a stopped vehicle X1, making it difficult to check the oncoming lane.
[0180] When a notification is given via the commander interface 3 of the remote instruction device 1 that the necessary information is not included in the detection information within the limited information range, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this notification. When a proposal to change the position and attitude of the autonomously driven vehicle 2 or to change the detection area of the external sensor 22 is given via the commander interface 3 of the remote instruction device 1, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this proposal. When no notification or proposal is given, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on the confirmation result of the driving situation information.
[0181] (Process G6) When the remote commander R issues an instruction to change the position and attitude of the autonomously driven vehicle 2, the autonomously driven control unit 41 changes the position and attitude of the autonomously driven vehicle 2 based on the position and attitude change instruction. Here, for example, as shown in Fig. 4(b), the remote commander R causes the autonomously driven vehicle 2 to move forward from its current position and steer rightward to stop it so that a vehicle traveling straight in the oncoming lane can be detected without being blocked by the stopped vehicle X1.
[0182] Furthermore, when an instruction to change the detection area of the external sensor 22 is given by the remote commander R, the sensor angle of view change unit 38 changes the detection area of the external sensor 22 based on the instruction to change the detection area. Here, for example, as shown in Fig. 4(c), the remote commander R translates the position of the sensor whose detection area is in front of the autonomous vehicle 2 to a position on the right side of the autonomous vehicle 2 so that vehicles traveling straight in the oncoming lane can be detected as much as possible without being blocked by the stopped vehicle X1. This makes it easier for the sensor that detects in front of the autonomous vehicle 2 to detect vehicles traveling straight in the oncoming lane.
[0183] (Process G7) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process G8) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts overtaking the stopped vehicle X1.
[0184] (changing lanes to avoid construction zone) As an example, as shown in FIG. 7, a case will be described where a construction section K exists in the driving lane of the autonomous vehicle 2, and the autonomous vehicle 2 changes lanes to an oncoming lane to avoid the construction section K. (Process H1) The external environment recognition unit 32 recognizes that the autonomous vehicle 2 has reached the start point of the construction section K (that the situation is subject to remote instruction) based on the position information of the autonomous vehicle 2, the map information in the map database 24, and the route generated by the route generation unit 40. Here, as an example, it is assumed that the map information includes the position information of the construction section K. (Process H2) The autonomous driving ECU 20 stops the autonomous vehicle 2 at a predetermined stop position. Here, for example, the autonomous driving ECU 20 stops the autonomous vehicle 2 at a stop line set up just before the construction section K.
[0185] (Process H3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process H4) When changing lanes to the oncoming lane to avoid the construction section K, the remote commander R, for example, needs to confirm that there is no vehicle traveling straight in the oncoming lane when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor that detects the front of the autonomous vehicle 2 from among the multiple sensors that the external sensor 22 has, and sets only the detection information detected by the selected sensor as detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information (construction section K) and the route, to the limited information range.
[0186] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0187] Here, if the traffic director P is recognized by the external environment recognition unit 32, in step H4, the transmission information limiting unit 35 selects a sensor that includes the traffic director P in its detection area from among the multiple sensors that the external sensor 22 has. Here, the transmission information limiting unit 35 selects a camera as the sensor. Then, the transmission information limiting unit 35 may extract a portion that includes the traffic director P from the captured image (detection information) of the selected camera, and set the extracted extracted image A1 as detection information within the limited information range. In other words, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the external environment (recognized traffic director P) and the route, to the limited information range.
[0188] (Process H5) The remote commander R checks the detection information within the limited information range contained in the driving condition information received from the autonomous vehicle 2, and if desired information such as necessary information is not included, it gives the autonomous vehicle 2 at least one of an instruction to change the position and attitude of the autonomous vehicle 2 and an instruction to change the detection area of the external sensor 22.
[0189] Here, as an example, when issuing a remote instruction, the remote commander R needs to check the situation of the oncoming lane ahead of the autonomous vehicle 2 (the adjacent lane to the right of the lane in which the autonomous vehicle 2 is traveling). For this reason, the oncoming lane is set in advance as necessary information for such a situation. In the example shown in FIG. 7, the remote commander R may issue an instruction to change the position and attitude of the autonomous vehicle 2, for example, in cases where the oncoming lane cannot be detected by a sensor that detects the front of the autonomous vehicle 2 (there are many parts that cannot be detected). The remote commander R may issue an instruction to change the position and attitude of the autonomous vehicle 2, for example, in cases where the traffic guide P is not included or only partially included in the image captured by the camera.
[0190] When a notification is given via the commander interface 3 of the remote instruction device 1 that the necessary information is not included in the detection information within the limited information range, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this notification. When a proposal to change the position and attitude of the autonomously driven vehicle 2 or to change the detection area of the external sensor 22 is given via the commander interface 3 of the remote instruction device 1, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on this proposal. When no notification or proposal is given, the remote commander R may issue an instruction to change the position and attitude of the autonomously driven vehicle 2 based on the confirmation result of the driving situation information.
[0191] (Process H6) When the remote commander R issues an instruction to change the position and attitude of the autonomously driven vehicle 2, the autonomous driving control unit 41 changes the position and attitude of the autonomously driven vehicle 2 based on the instruction to change the position and attitude. In addition, when the remote commander R issues an instruction to change the detection area of the external sensor 22, the sensor angle of view change unit 38 changes the detection area of the external sensor 22 based on the instruction to change the detection area.
[0192] (Process H7) The remote commander R checks the driving situation information transmitted from the autonomous vehicle 2 and issues remote instructions. When a captured image of the traffic director P is transmitted from the autonomous vehicle 2, the remote commander R can check the guiding action of the traffic director P and issue remote instructions. (Process H8) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts changing lanes to the oncoming lane to avoid the construction section K.
[0193] (If there is falling debris) As an example, a case will be described in which a fallen object D (obstacle) is present on the driving lane ahead of the autonomously driven vehicle 2 as shown in FIG. (Process J1) The external environment recognition unit 32 recognizes a fallen object D on the driving lane of the autonomous vehicle 2. The remote instruction determination unit 34 recognizes, based on the fallen object D recognized by the external environment recognition unit 32 and the path generated by the path generation unit 40, that the autonomous vehicle 2 is in a situation where it needs to avoid the fallen object D by going over the fallen object D or by changing lanes (a situation subject to remote instruction). (Process J2) The autonomous driving ECU 20 stops the autonomous vehicle 2 at a predetermined stop position. Here, for example, the autonomous driving ECU 20 stops the autonomous vehicle 2 at a position a predetermined distance before the fallen object D.
[0194] (Process J3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process J4) When a falling object D is present, the remote commander R, for example, needs to confirm whether the autonomous vehicle 2 can overcome the falling object D when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor that detects the front of the autonomous vehicle 2 from among the multiple sensors included in the external sensor 22, and sets only the detection information detected by the selected sensor as detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the external environment (the recognized falling object D) and the course, to the limited information range.
[0195] In this case, since the falling object D is a stationary object, the transmission information limiting unit 35 may select a camera or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits to the remote instruction server 10 driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0196] The transmission information limiting unit 35 may extract a portion including a falling object D from the image captured by the selected camera, and set the extracted extracted image A2 as detection information within the limited information range.
[0197] (Process J5) The remote commander R checks the driving situation information transmitted from the autonomous vehicle 2 and issues a remote instruction. Here, for example, the remote commander R determines whether the autonomous vehicle 2 can overcome the fallen object D. Then, the remote commander R may issue, for example, a remote instruction to overcome the fallen object D, or a remote instruction to avoid the fallen object D by changing lanes. (Process J6) When the automatic driving control unit 41 receives a remote instruction from the remote commander R, it starts to step over the falling object D or to change lanes.
[0198] (When crossing a pedestrian crossing) As an example, a case will be described in which an autonomous vehicle 2 passes through a pedestrian crossing as shown in FIG. (Process K1) The remote instruction determination unit 34 recognizes that the autonomously driven vehicle 2 has reached a pedestrian crossing (a remote instruction target situation), in the same manner as in step A1 above. (Process K2) The autonomous driving ECU 20 stops the autonomous driving vehicle 2 at a predetermined stopping position, similar to step A2 described above.
[0199] (Process K3) The remote instruction determination unit 34 requests a remote instruction from the remote commander R. (Process K4) When passing through a crosswalk, the remote commander R, for example, needs to confirm the absence of pedestrians (pedestrians crossing the crosswalk) entering the crosswalk from both sides of the road when issuing a remote instruction. For this reason, for example, the transmission information limiting unit 35 selects a sensor whose detection area includes the right front of the autonomous vehicle 2 and a sensor whose detection area includes the left front of the autonomous vehicle 2 from among the sensors possessed by the external sensor 22, and sets only the detection information detected by the selected sensor as the detection information within the limited information range. In other words, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R from among the detection information detected by the external sensor 22, based on the map information and the course, to the limited information range.
[0200] Note that, here, the transmission information limiting unit 35 may select any type of sensor, namely, a camera, a radar, or a LIDAR, which are given as examples of sensors included in the external sensor 22. Then, the driving status information transmitting unit 39 transmits, to the remote instruction server 10, driving status information of the autonomously driving vehicle 2, including detection information whose information range is set to the limited information range.
[0201] (Process K5) The remote commander R checks the driving status information transmitted from the autonomous vehicle 2 and issues remote instructions. (Process K6) When the autonomous driving control unit 41 receives a remote instruction from the remote commander R, it starts the autonomous driving vehicle 2 and starts passing through the crosswalk.
[0202] Hereinafter, a specific example of setting the limited information range performed by the transmission information limiting unit 35 when a remote instruction target situation other than the above-mentioned remote instruction target situation occurs will be described.
[0203] For example, a case where the autonomous vehicle 2 changes lanes to the adjacent lane on the left side will be described. In this case, the transmission information limiting unit 35 can select, as an example, a sensor whose detection area includes the front of the autonomous vehicle 2 and a sensor whose detection area includes the left rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range. For example, a case where the autonomous vehicle 2 changes lanes to the adjacent lane on the right side will be described. In this case, the transmission information limiting unit 35 can select, as an example, a sensor whose detection area includes the front of the autonomous vehicle 2 and a sensor whose detection area includes the right rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range.
[0204] That is, when the autonomous vehicle 2 changes lanes to the left or right, the transmission information limiting unit 35 can set the range of information detected by the external sensor 22 to be transmitted to the remote commander R as a limited information range based on the map information and the route. Note that when the autonomous vehicle 2 changes lanes to the left or right, the transmission information limiting unit 35 may select any type of sensor, i.e., a camera, a radar, or a lidar, which are given as examples of sensors that the external sensor 22 has.
[0205] For example, a case where the autonomous vehicle 2 enters a railroad crossing will be described. In this case, the transmission information limiting unit 35 can select, as an example, a sensor that includes the front of the autonomous vehicle 2 in its detection area among the sensors possessed by the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22, based on the map information and the route, as the limited information range. Note that the transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, or a lidar, which are given as examples of sensors possessed by the external sensor 22. In this case, for example, even if it is difficult for the external environment recognition unit 32 to recognize whether or not there is a space for the autonomous vehicle 2 to enter beyond the railroad crossing, the autonomous vehicle 2 can travel based on remote control by the remote commander R.
[0206] For example, a case where the vehicle starts after stopping at a pedestrian crossing where no traffic lights are installed will be described. In this case, the transmission information limiting unit 35 can select, as an example, a sensor that includes the front of the autonomous vehicle 2 in its detection area among the sensors possessed by the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22, based on the map information and the route, as the limited information range. Note that the transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, or a lidar, which are given as examples of sensors possessed by the external sensor 22. In this case, the remote commander R can confirm the absence of pedestrians on the pedestrian crossing based on the driving situation information transmitted from the autonomous vehicle 2, and issue a remote instruction (an instruction to start).
[0207] For example, a case will be described in which the autonomous vehicle 2 stops once in front of a lane perpendicular to the driving lane and then restarts. In this case, the transmission information limiting unit 35 can select, as an example, a sensor that includes the front side of the autonomous vehicle 2 in its detection area among the sensors possessed by the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22 based on the map information and the course, as the limited information range. Note that the transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, or a lidar, which are given as examples of sensors possessed by the external sensor 22. In this case, the remote commander R can confirm a moving obstacle in a lane perpendicular to the driving lane of the autonomous vehicle 2 and issue a remote instruction (an instruction to start).
[0208] For example, a case where the vehicle passes through a toll gate on a highway will be described. In this case, the transmission information limiting unit 35 can select, as an example, a sensor that includes the front of the autonomous vehicle 2 in its detection area among the sensors of the external sensor 22, and set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22 based on the map information and the route, as the limited information range. Note that the transmission information limiting unit 35 can select the camera of the external sensor 22 so that the remote commander R can check the display of the ETC gate.
[0209] For example, a case where lanes are split on a highway will be described. When the autonomous vehicle 2 splits to the left, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the left front of the autonomous vehicle 2 and a sensor whose detection area includes the rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. When the autonomous vehicle 2 splits to the right, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the right front of the autonomous vehicle 2 and a sensor whose detection area includes the rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. The transmission information limiting unit 35 can set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22, based on the map information and the course, to the limited information range. The transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, or a lidar, which are given as examples of sensors possessed by the external sensor 22.
[0210] For example, a case where the autonomous vehicle 2 merges into an adjacent lane on a highway will be described. When the autonomous vehicle 2 merges into the adjacent lane on the left side, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the left front of the autonomous vehicle 2 and a sensor whose detection area includes the left rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. When the autonomous vehicle 2 merges into the adjacent lane on the right side, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the right front of the autonomous vehicle 2 and a sensor whose detection area includes the right rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. The transmission information limiting unit 35 can set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R, among the detection information detected by the external sensor 22, to the limited information range, based on the map information and the route. The transmission information limiting unit 35 may select any type of sensor from among the cameras, radars, and lidars given as examples of sensors included in the external sensor 22.
[0211] For example, a case will be described in which there is a lane merging into the driving lane of the autonomous vehicle 2 on a highway. When there is a lane merging from the left side into the driving lane of the autonomous vehicle 2, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the left front of the autonomous vehicle 2 and a sensor whose detection area includes the left rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. When there is a lane merging from the right side into the driving lane of the autonomous vehicle 2, the transmission information limiting unit 35 selects, as an example, a sensor whose detection area includes the right front of the autonomous vehicle 2 and a sensor whose detection area includes the right rear of the autonomous vehicle 2, among the sensors possessed by the external sensor 22. The transmission information limiting unit 35 can set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22 based on the map information and the route, as the limited information range. The transmission information limiting unit 35 may select any type of sensor from among the cameras, radars, and lidars given as examples of sensors included in the external sensor 22.
[0212] For example, a case will be described where the vehicle arrives at the destination and stops at a parking space provided on the road surface. Here, as shown in FIG. 17, a parked vehicle X1 (obstacle) exists on the parking space Z, and the autonomous vehicle 2 cannot automatically stop in the parking space Z (remote instruction target situation), so a remote instruction is requested. The autonomous vehicle 2 stops in front of the parking space Z and requests a remote instruction. In this case, as an example, the transmission information limiting unit 35 can select a sensor that includes the front of the autonomous vehicle 2 in its detection area among the sensors possessed by the external sensor 22 so that the parking space Z is included in the detection area, and set only the detection information detected by the selected sensor as the detection information within the limited information range. That is, the transmission information limiting unit 35 can set the range of information to be transmitted to the remote commander R among the detection information detected by the external sensor 22, based on the map information (the position of the parking space Z (destination)) and the course, to the limited information range. The transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, or a lidar, which are given as examples of sensors possessed by the external sensor 22. The transmission information limiting unit 35 may select a camera among the sensors possessed by the external sensor 22 so as to be able to confirm a parking space Z marked on the road surface. Here, the remote commander R checks the driving situation information transmitted from the autonomous vehicle 2, and can issue a remote instruction to, for example, stop the autonomous vehicle 2 in the parking space Z, or a remote instruction to, for example, drop off a passenger on the spot (deemed to have arrived), etc.
[0213] For example, a case will be described in which the vehicle arrives near a destination building and enters the building from a road. In this case, the transmission information limiting unit 35 can adopt a method for setting the limited information range similar to the method for turning left or right at an intersection, depending on the direction of entering the building from the road.
[0214] For example, a case will be described in which the autonomous vehicle 2 starts after arriving at a destination and passengers have finished getting on and off. In addition, here, as shown in FIG. 18, it is assumed that the autonomous vehicle 2 starts from a state where it is stopped in a parking space Z (a remote instruction target situation). For example, when the autonomous vehicle 2 starts, it is necessary to recognize the surroundings of the autonomous vehicle 2, but the accuracy of the surrounding recognition of the autonomous vehicle 2 may be low, and it may be difficult for the autonomous vehicle 2 to start automatically. For this reason, the autonomous vehicle 2 may request a remote instruction when starting. In this case, as an example, the transmission information limiting unit 35 may select a sensor from among the sensors possessed by the external sensor 22 so that detection information of the entire periphery of the autonomous vehicle 2 can be obtained, and the detection information detected by the selected sensor may be set as detection information within the limited information range. Here, the transmission information limiting unit 35 may extract only detection information within a range close to the autonomous vehicle 2 (within a predetermined range), and set the extracted detection information as detection information within the limited information range. The transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, a lidar, or a sonar sensor, which are given as examples of sensors included in the external sensor 22. The transmission information limiting unit 35 may also select a camera system that captures images of the surroundings of the autonomously driven vehicle 2 with multiple cameras and displays the entire circumference of the autonomously driven vehicle 2 in a single image by stitching together the captured images.
[0215] For example, a case where the vehicle arrives at a destination and passengers are allowed to board and disembark will be described. For example, when passengers get on and off the autonomous vehicle 2, it is necessary to recognize the situation in front of the door of the autonomous vehicle 2. However, the accuracy of the recognition in front of the door may be low, and it may be difficult for the autonomous vehicle 2 to automatically open and close the door to allow passengers to board and disembark. For this reason, the autonomous vehicle 2 may request a remote instruction in a situation where a passenger is allowed to board and disembark (a situation subject to a remote instruction). In this case, the transmission information limiting unit 35 may, for example, select a sensor from among the sensors possessed by the external sensor 22 so as to obtain detection information in front of the door of the autonomous vehicle 2, and set the detection information detected by the selected sensor as detection information within the limited information range. Here, the transmission information limiting unit 35 may extract only detection information within a range (within a predetermined range) close to the autonomous vehicle 2, and set the extracted detection information as detection information within the limited information range. Note that the transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, a lidar, or a sonar sensor, which are given as examples of sensors possessed by the external sensor 22.
[0216] For example, a case where the autonomous vehicle 2 is reversed from a stopped state will be described. For example, when the autonomous vehicle 2 is reversed, it is necessary to recognize the situation behind the autonomous vehicle 2, but the accuracy of recognition of the close area behind the autonomous vehicle 2 may be low, and it may be difficult for the autonomous vehicle 2 to reverse automatically. For this reason, the autonomous vehicle 2 may request a remote instruction in a situation where it is reversed (a situation subject to a remote instruction). In this case, as an example, the transmission information limiting unit 35 may select a sensor from among the sensors possessed by the external sensor 22 so that detection information behind the autonomous vehicle 2 can be obtained, and set the detection information detected by the selected sensor as detection information within the limited information range. Here, the transmission information limiting unit 35 may extract only detection information within a range close to the autonomous vehicle 2 (within a predetermined range), and set the extracted detection information as detection information within the limited information range. Note that the transmission information limiting unit 35 may select any type of sensor, such as a camera, a radar, a lidar, or a sonar sensor, which are given as examples of sensors possessed by the external sensor 22.
[0217] As described above, in the vehicle remote instruction system 100, the transmission information limiting unit 35 sets the range of information to be transmitted to the remote commander R to a limited information range, so that only detection information (necessary detection information) corresponding to the external situation and route of the autonomously driven vehicle 2, among the detection information detected by the external sensor 22, can be transmitted to the remote commander R. This enables the vehicle remote instruction system 100 to appropriately execute remote instructions by the remote commander R, while reducing the data volume of the detection information transmitted from the autonomously driven vehicle 2 to the remote commander R (remote instruction server 10).
[0218] The transmission information limiting unit 35 can determine which of the multiple sensors included in the external sensor 22 is to transmit detection information to the remote commander R based on the external situation and the course, and set the detection information detected by the determined sensor as detection information within the limited information range. In this case, even if the vehicle remote instruction system 100 has multiple sensors as the external sensor 22, it can reduce the data volume of the detection information transmitted to the remote commander R by determining which sensor is to transmit detection information in accordance with the external situation and setting the detection information detected by the determined sensor as detection information within the limited information range.
[0219] The transmission information limiting unit 35 can extract a portion to be transmitted to the remote commander R from the detection information detected by the external sensor 22 based on the external situation and the course, and set the detection information of the extracted portion as detection information within the limited information range. In this case, the vehicle remote instruction system 100 can reduce the data volume of the detection information to be transmitted to the remote commander R by extracting a portion to be transmitted from the detection information of the external sensor 22 and setting the extracted portion as detection information within the limited information range.
[0220] In the vehicle remote instruction system 100, when the necessary information is not included in the detection information within the limited information range, the notification suggestion unit 37 can notify the remote commander R or suggest a change in the position and attitude of the autonomously driven vehicle 2 or a change in the detection area of the external sensor 22. In this case, the vehicle remote instruction system 100 can notify the remote commander R that the necessary information is not included in the limited information range, or suggest to the remote commander R that the necessary information is not included in the limited information range and it is necessary to change the position and attitude of the autonomously driven vehicle 2 or change the detection area of the external sensor 22. This allows the remote commander R to take action based on the notification or suggestion from the vehicle remote instruction system 100.
[0221] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment. For example, it is not essential that the information presence / absence determination unit 36 determines the presence or absence of necessary information within the limited information range and the notification suggestion unit 37 performs notification, etc.
[0222] The remote instruction device 1 may be mounted on a vehicle. In this case, a remote commander R also rides in the vehicle. The remote instruction server 10 may be a cloud server configured from ECUs of multiple vehicles. [Explanation of symbols]
[0223] 2...Autonomous driving vehicle, 22...External sensor, 35...Transmission information limiting unit, 36...Information presence / absence determining unit, 37...Notification suggesting unit, 100...Vehicle remote control system, R...Remote commander.
Claims
1. A vehicle control device which recognizes an external environment based on images captured by cameras including a front camera which captures an image in front of a vehicle, a right camera whose imaging area is to the right of the imaging area of the front camera, a left camera whose imaging area is to the left of the imaging area of the front camera, and a rear camera which captures an image behind the vehicle, and which controls the driving of the vehicle according to remote instructions regarding the driving of the vehicle provided from a remote commander, a transmission unit that transmits images captured by the right camera and the left camera to a remote instruction device operated by the remote commander outside the vehicle when the vehicle travels through an intersection where no traffic lights are installed, When the situation of an intersecting road intersecting the road on which the vehicle is traveling cannot be detected from the captured images of the right camera and the left camera, the remote instruction device transmits notification information to the remote instruction device, A vehicle control device that, after transmitting the notification information, causes the vehicle to move forward a predetermined distance and then stops the vehicle, or changes the mounting position and attitude of the camera.
2. The vehicle control device according to claim 1 , wherein the transmission unit does not transmit images captured by at least one of the front camera and the rear camera to the remote instruction device when the vehicle is traveling through an intersection where no traffic lights are installed.
3. 3. The vehicle control device according to claim 1, further comprising: a remote control unit that starts the vehicle when an instruction to change the position and attitude of the vehicle is received from the remote commander; or when the remote instruction based on a confirmation result of the images captured by the right camera and the left camera and transmitted by the transmission unit is received from the remote instruction device.
4. The vehicle control device according to claim 1 , wherein the situation of the intersecting road cannot be detected when an area blocked by a fence is included in the imaging ranges of the right camera and the left camera.
5. A vehicle control method carried out by a vehicle control device which recognizes an external environment based on images captured by cameras including a front camera which captures an image in front of a vehicle, a right camera whose imaging area is to the right of the imaging area of the front camera, a left camera whose imaging area is to the left of the imaging area of the front camera, and a rear camera which captures an image behind the vehicle, and controls the driving of the vehicle according to remote instructions regarding the driving of the vehicle provided from a remote commander, a first transmission step of transmitting images captured by the right camera and the left camera to a remote instruction device operated by the remote commander outside the vehicle when the vehicle is traveling through an intersection where no traffic lights are installed; a second transmission step of transmitting notification information to the remote instruction device when a situation of an intersecting road intersecting the road on which the vehicle is traveling cannot be detected from the images captured by the right camera and the left camera; a stopping step of causing the vehicle to move forward a predetermined distance and then stop the vehicle after transmitting the notification information, or a changing step of changing the mounting position and attitude of the camera.
6. 6. The vehicle control method according to claim 5, wherein in the first transmission step, the image captured by at least one of the front camera and the rear camera is not transmitted to the remote instruction device.
7. 7. The vehicle control method according to claim 5 or 6, further comprising a starting step of starting the vehicle when an instruction to change the position and attitude of the vehicle is received from the remote commander, or when the remote instruction based on the confirmation results of the images captured by the right camera and the left camera transmitted in the first transmission step is received from the remote instruction device.
8. The vehicle control method according to claim 5 , wherein the situation of the intersecting road cannot be detected when an area blocked by a fence is included in the imaging ranges of the right camera and the left camera.
9. A vehicle control program that recognizes the external environment based on images captured by cameras including a front camera that captures an image in front of a vehicle, a right camera whose imaging area is to the right of the imaging area of the front camera, a left camera whose imaging area is to the left of the imaging area of the front camera, and a rear camera that captures an image behind the vehicle, and operates an ECU of a vehicle control device that controls the driving of the vehicle based on remote instructions regarding the driving of the vehicle provided from a remote commander, When the vehicle is traveling through an intersection where no traffic lights are installed, the ECU is operated as a transmission unit that transmits images captured by the right camera and the left camera to a remote instruction device operated by the remote commander outside the vehicle, When the situation of an intersecting road intersecting the road on which the vehicle is traveling cannot be detected from the images captured by the right camera and the left camera, the ECU is operated to transmit notification information to the remote instruction device; a vehicle control program that operates the ECU to, after transmitting the notification information, move the vehicle forward a predetermined distance and then stop the vehicle, or to change the mounting position and attitude of the camera.
10. 10. The vehicle control program according to claim 9, wherein the transmission unit operates the ECU so as not to transmit images captured by at least one of the front camera and the rear camera to the remote control device when the vehicle is traveling through an intersection where no traffic lights are installed.
11. 11. The vehicle control program according to claim 9 or 10, wherein the ECU is operated to start the vehicle when an instruction to change the position and attitude of the vehicle is received from the remote commander, or when the remote instruction based on a confirmation result of the images captured by the right camera and the left camera and transmitted by the transmission unit is received from the remote instruction device.
12. 10. The vehicle control program according to claim 9, wherein the situation of the intersecting road cannot be detected when an area blocked by a fence is included in the imaging ranges of the right camera and the left camera.
Citation Information
Patent Citations
Guiding image display device for vehicle
JP1999003500A
Automatic driving support system, automatic driving support method and computer program
JP2016060336A
Moving body, moving body control system, moving body control method
JP2018160134A
Remote control device
JP2018180771A
