Vehicle Remote Indicator System

Abstract

The present invention provides a vehicle remote instruction system (100), which sends a remote instruction request from an autonomous vehicle (2) to a remote instruction device (1), and controls the driving of the autonomous vehicle (2) based on the remote instruction sent from the remote instruction device (1) in response to the remote instruction request. The vehicle remote instruction system (100) comprises: a delay judgment unit (39) which judges whether a communication delay occurs between the remote instruction device (1) and the autonomous vehicle (2); and a rejection unit (40) which rejects the remote instruction sent in response to the remote instruction request when the delay judgment unit (39) judges that a communication delay occurs.

Description

Technical Field

[0001] The present disclosure relates to a vehicle remote instruction system for controlling the travel of an autonomous driving vehicle based on remote instructions sent from a remote instruction device. Background Art

[0002] For example, Japanese Patent Application Laid-Open No. 2016-71585 describes a vehicle remote instruction system that controls the driving of an autonomous driving vehicle by remotely instructing the autonomous driving vehicle from a remote instruction device.

[0003] In the vehicle remote instruction system, when a communication delay occurs between the remote instruction device and the autonomous vehicle, the driving control of the autonomous vehicle executed by receiving the remote instruction is switched to safety control. Under the safety control, the vehicle behavior for implementing the received remote instruction is different from the vehicle behavior during normal control. Summary of the invention

[0004] In the vehicle remote instruction system described in the above document, when a communication delay occurs, the driving control is switched to the safety control state, and the driving of the autonomous vehicle is controlled based on the remote instruction. However, when a communication delay occurs, it is possible that the content of the remote instruction from the remote instruction device itself is not appropriate. Therefore, in the technical field, a solution is sought that can appropriately control the driving of the autonomous vehicle even when a communication delay occurs between the remote instruction device and the autonomous vehicle.

[0005] One embodiment of the present invention involves a vehicle remote instruction system, which sends a remote instruction request from an autonomous driving vehicle to a remote instruction device, and controls the driving of the autonomous driving vehicle based on the remote instruction sent from the remote instruction device in response to the remote instruction request. The vehicle remote instruction system has: a delay judgment unit, which judges whether a communication delay occurs between the remote instruction device and the autonomous driving vehicle; and a rejection unit, which rejects the remote instruction sent in response to the remote instruction request when the delay judgment unit determines that a communication delay occurs.

[0006] For example, in the case of a communication delay, the surrounding conditions may have changed when the autonomous vehicle receives the remote instruction, so that the received remote instruction is inappropriate. Therefore, in the vehicle remote instruction system, when it is determined that a communication delay has occurred, the remote instruction sent in response to the remote instruction request is rejected. In this way, the control of the driving of the autonomous vehicle based on inappropriate remote instructions can be suppressed. Therefore, even if a communication delay occurs between the remote instruction device and the autonomous vehicle, the vehicle remote instruction system can appropriately control the driving of the autonomous vehicle by rejecting the remote instruction.

[0007] Alternatively, the vehicle remote instruction system may further include: a sending time acquisition unit, which acquires the time when the remote instruction device sends the remote instruction, i.e., the instruction sending time; and a receiving time acquisition unit, which acquires the time when the autonomous driving vehicle receives the remote instruction, i.e., the instruction receiving time. When the difference between the instruction sending time acquired by the sending time acquisition unit and the instruction receiving time acquired by the receiving time acquisition unit is greater than a first threshold, the delay judgment unit judges that a communication delay has occurred. In this case, the vehicle remote instruction system can use the instruction sending time and the instruction receiving time to judge whether a communication delay has occurred.

[0008] Alternatively, the vehicle remote instruction system may further include: a request time acquisition unit that acquires the time when the autonomous vehicle sends a remote instruction request to the remote instruction device, i.e., the request sending time; and a reception time acquisition unit that acquires the time when the autonomous vehicle receives the remote instruction, i.e., the instruction receiving time. When the difference between the request sending time acquired by the request time acquisition unit and the instruction receiving time acquired by the reception time acquisition unit is greater than a second threshold, the delay judgment unit judges that a communication delay has occurred. In this case, the vehicle remote instruction system can use the request sending time and the instruction receiving time to more appropriately judge whether a communication delay has occurred.

[0009] Alternatively, the vehicle remote instruction system may further include: a sensor information acquisition unit for transmitting, which acquires sensor information for transmitting instruction; and a sensor information acquisition unit for receiving, which acquires sensor information for receiving instruction, wherein the autonomous driving vehicle has a sensor information transmission unit for transmitting sensor information of an onboard sensor for detecting an external environment to the remote instruction device, and the remote instruction device has a sensor information receiving unit for receiving sensor information transmitted from the sensor information transmission unit, wherein the sensor information acquisition unit for transmitting acquires sensor information received by the sensor information receiving unit at the time when the remote instruction device transmits the remote instruction as sensor information for transmitting instruction, and the sensor information acquisition unit for receiving acquires sensor information detected by the onboard sensor at the time when the autonomous driving vehicle receives the remote instruction as sensor information for receiving instruction, and the delay determination unit determines whether a communication delay has occurred based on the sensor information for transmitting instruction acquired by the sensor information acquisition unit for transmitting and the sensor information for receiving instruction acquired by the sensor information acquisition unit for receiving. In this case, the vehicle remote instruction system can more appropriately determine whether a communication delay has occurred based on the sensor information for transmitting instruction and the sensor information for receiving instruction.

[0010] Alternatively, the vehicle remote instruction system may further include: a sensor information acquisition unit at a time of request, which acquires sensor information detected by an on-board sensor that detects the external environment of the autonomous driving vehicle at the time when the autonomous driving vehicle sends a remote instruction request as sensor information at a time of instruction request; a sensor information acquisition unit at a time of reception, which acquires sensor information detected by an on-board sensor at the time when the autonomous driving vehicle receives the remote instruction as sensor information at a time of instruction reception, and a delay judgment unit judges whether a communication delay has occurred based on the sensor information at the time of instruction request acquired by the sensor information acquisition unit at a time of request and the sensor information at the time of instruction reception acquired by the sensor information acquisition unit at a time of reception. In this case, the vehicle remote instruction system can more appropriately judge whether a communication delay has occurred based on the sensor information at the time of instruction request and the sensor information at the time of instruction reception.

[0011] According to one embodiment of the present disclosure, even when a communication delay occurs between a remote instruction device and the autonomous driving vehicle, the driving of the autonomous driving vehicle can be appropriately controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a diagram showing an example of the overall overview of the vehicle remote instruction system according to the first embodiment.

[0013] Figure 2 This is a block diagram showing an example of the configuration of an autonomous driving vehicle of the vehicle remote instruction system according to the first embodiment.

[0014] Figure 3 This is a block diagram showing an example of the hardware configuration of a remote instruction server of the vehicle remote instruction system according to the first embodiment.

[0015] Figure 4 This is a block diagram showing an example of the configuration of the remote pointing device according to the first embodiment.

[0016] Figure 5 This is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device.

[0017] Figure 6 This is a flowchart showing an example of a rejection determination process of a remote instruction performed by an autonomous driving vehicle.

[0018] Figure 7 This is a block diagram showing an example of the configuration of an autonomous driving vehicle of the vehicle remote instruction system according to the second embodiment.

[0019] Figure 8 This is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device.

[0020] Fig. 9 This is a flowchart showing an example of a rejection determination process of a remote instruction performed by an autonomous driving vehicle.

[0021] Fig.10 This is a block diagram showing an example of the configuration of an autonomous driving vehicle of the vehicle remote instruction system according to the third embodiment.

[0022] Fig.11 (a) is a diagram showing sensor information at the time of instruction transmission received by the remote instruction request receiving unit at the time when the remote instruction device transmits the remote instruction.

[0023] Fig.11 (b) is a diagram showing sensor information at the time of instruction reception detected by an external sensor at the time when the autonomous driving vehicle receives the remote instruction.

[0024] Fig.12 This is a block diagram showing an example of the configuration of a remote pointing device according to the third embodiment.

[0025] Fig.13 This is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device.

[0026] Fig.14 This is a flowchart showing an example of a rejection determination process of a remote instruction executed by an autonomous driving vehicle.

[0027] Fig.15 This is a block diagram showing an example of the configuration of an autonomous driving vehicle of the vehicle remote instruction system according to the fourth embodiment.

[0028] Fig.16 This is a flowchart showing an example of a rejection determination process of a remote instruction executed by an autonomous driving vehicle. DETAILED DESCRIPTION

[0029] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and repeated descriptions are omitted.

[0030] (First Embodiment)

[0031] A first embodiment of the vehicle remote instruction system will be described. Figure 1 This is a diagram showing an example of the overall overview of the vehicle remote instruction system according to the first embodiment. Figure 1 The vehicle remote instruction system 100 shown is a system in which a remote commander R remotely instructs an autonomous vehicle 2 . The remote instruction refers to an instruction from the remote commander R regarding the determination of the driving of the autonomous vehicle 2 .

[0032] The remote instruction may include, for example, at least one of an instruction to start a right turn at an intersection, an instruction to start entering an intersection with a traffic light, an instruction to start entering an intersection with poor visibility, and an instruction to start a lane change. In countries or regions where vehicles drive on the right, the instruction to start a right turn at an intersection may be replaced with an instruction to start a left turn at an intersection. An intersection with poor visibility may also be an intersection where it is difficult to visually identify the intersecting road conditions due to walls or the like when entering the intersection. Intersections with poor visibility are, for example, pre-registered in map information. Whether or not an intersection with poor visibility is an intersection may also be determined based on the direction of entry of the autonomous driving vehicle 2 relative to the intersection.

[0033] (Composition of vehicle remote indication system)

[0034] like Figure 1 As shown, the vehicle remote instruction system 100 includes a remote instruction device 1 to which a remote commander R inputs remote instructions. The remote instruction device 1 is communicably connected to a plurality of autonomous driving vehicles 2 via a network N. The network N is a wireless communication network. Various information is transmitted from the autonomous driving vehicle 2 to the remote instruction device 1.

[0035] In the vehicle remote instruction system 100, for example, in response to a remote instruction request from the autonomous vehicle 2, a remote commander R is requested to input a remote instruction. The remote commander R inputs the remote instruction to the commander interface 3 of the remote instruction device 1. The remote instruction device 1 sends the remote instruction to the autonomous vehicle 2 through the network N. As described above, the remote instruction device 1 sends the remote instruction in response to the remote instruction request from the autonomous vehicle 2. The autonomous vehicle 2 automatically drives according to the remote instruction.

[0036] In addition, in the vehicle remote instruction system 100, the number of remote commanders R is not limited, and can be one person or more than two people. There is no particular limit to the number of autonomous driving vehicles 2 that can communicate with the vehicle remote instruction system 100. It can be a method in which multiple remote commanders R alternately give remote instructions to one autonomous driving vehicle 2, or it can be a method in which one remote commander R gives remote instructions to more than two autonomous driving vehicles 2.

[0037] (Composition of autonomous vehicles)

[0038] First, an example of the configuration of the automatic driving vehicle 2 will be described. Figure 2 2 is a block diagram showing an example of the configuration of the autonomous driving vehicle 2. Figure 2As shown, the autonomous driving vehicle 2 has an autonomous driving ECU 20 as an example. The autonomous driving ECU 20 is an electronic control unit having a CPU, a ROM, a RAM, etc. In the autonomous driving ECU 20, various functions are realized by, for example, loading a program recorded in the ROM into the RAM and executing the program loaded into the RAM by the CPU. The autonomous driving ECU 20 may also be composed of a plurality of electronic units.

[0039] The automatic driving ECU 20 is connected to a GPS (Global Positioning System) receiving unit 21 , an external sensor 22 (on-vehicle sensor), an internal sensor (sensor information transmitting unit) 23 , a map database 24 , a communication unit 25 , and an actuator 26 .

[0040] The GPS receiving unit 21 receives signals from three or more GPS satellites to measure the position of the autonomous driving vehicle 2 (for example, the latitude and longitude of the autonomous driving vehicle 2 ). The GPS receiving unit 21 transmits the measured position information of the autonomous driving vehicle 2 to the autonomous driving ECU 20 .

[0041] The external sensor 22 is a vehicle-mounted sensor that detects the external environment of the autonomous driving vehicle 2. The external sensor 22 sends the detected detection information (sensor information) to the autonomous driving ECU 20. The external sensor 22 includes at least a camera. The camera is a camera device that captures the external environment of the autonomous driving vehicle 2. The camera is, for example, set on the inner side of the front windshield of the autonomous driving vehicle 2 to capture the front of the vehicle. The camera sends the detection information (capture information) related to the external environment of the autonomous driving vehicle 2 to the autonomous driving ECU 20. The camera can be a single-lens camera or a stereo camera. Multiple cameras can be set up, and in addition to the front of the autonomous driving vehicle 2, the left and right sides and the rear can also be captured. The autonomous driving vehicle 2 can also have an external camera facing the remote commander. The external camera facing the remote commander captures at least the front of the autonomous driving vehicle 2. The external camera facing the remote commander can be composed of multiple cameras that capture the surroundings including the sides and rear of the autonomous driving vehicle 2.

[0042] The external sensor 22 may include a radar sensor. A radar sensor is a detection device that uses radio waves (such as millimeter waves) or light to detect objects around the autonomous driving vehicle 2. Radar sensors include, for example, millimeter wave radars or optical radars (LIDAR: Light Detection and Ranging). The radar sensor detects objects by automatically sending radio waves or light to the periphery of the autonomous driving vehicle 2 and receiving radio waves or light reflected by the object. The radar sensor sends the detected detection information (object information) to the autonomous driving ECU 20. In addition to fixed objects such as guardrails and buildings, objects also include moving objects such as pedestrians, bicycles, and other vehicles. In addition, the external sensor 22 may also include a sound detection sensor that detects external sounds of the autonomous driving vehicle 2.

[0043] The internal sensor 23 is a vehicle-mounted sensor that detects the driving state of the autonomous driving vehicle 2. The internal sensor 23 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the autonomous driving vehicle 2. As the vehicle speed sensor, a wheel speed sensor that is set relative to the wheels of the autonomous driving vehicle 2 or a drive shaft that rotates integrally with the wheels and detects the rotation speed of each wheel can be used. The vehicle speed sensor sends the detected vehicle speed information (wheel speed information) to the autonomous driving ECU 20.

[0044] The acceleration sensor is a detector that detects the acceleration of the autonomous driving vehicle 2. The acceleration sensor includes, for example, a front and rear acceleration sensor that detects the acceleration in the front and rear directions of the autonomous driving vehicle 2. The acceleration sensor may also include a lateral acceleration sensor that detects the lateral acceleration of the autonomous driving vehicle 2. The acceleration sensor, for example, sends the acceleration information of the autonomous driving vehicle 2 to the autonomous driving ECU 20. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) of the center of gravity of the autonomous driving vehicle 2 around the vertical axis. As the yaw rate sensor, for example, a gyro sensor can be used. The yaw rate sensor sends the detected yaw rate information of the autonomous driving vehicle 2 to the autonomous driving ECU 20.

[0045] The map database 24 is a database that records map information. The map database 24 is formed in a recording device such as an HDD carried by the autonomous driving vehicle 2. Map information includes road location information, road shape information (such as curvature information), intersection and fork location information, etc. Map information may also include traffic control information such as statutory speed associated with the location information. Map information may also include target information for obtaining the location information of the autonomous driving vehicle 2. As targets, road signs, road surface markings, traffic lights, utility poles, etc. can be used. The map database 24 can also be constructed in a server that can communicate with the autonomous driving vehicle 2. The server is not limited to the remote indication server 10.

[0046] The communication unit 25 is a communication device that controls wireless communication with the outside of the automatic driving vehicle 2. The communication unit 25 transmits and receives various information with the remote instruction server 10 via the network N.

[0047] The actuator 26 is a device for controlling the autonomous driving vehicle 2. The actuator 26 includes at least a driving actuator, a braking actuator, and a steering actuator. The driving actuator controls the amount of air supplied to the engine (throttle opening) according to a control signal from the autonomous driving ECU 20, thereby controlling the driving force of the autonomous driving vehicle 2. In the case where the autonomous driving vehicle 2 is a hybrid vehicle, in addition to the amount of air supplied to the engine, a control signal from the autonomous driving ECU 20 is input to the electric motor as a power source to control the driving force. In the case where the autonomous driving vehicle 2 is an electric vehicle, a control signal from the autonomous driving ECU 20 is input to the electric motor as a power source to control the driving force. In these cases, the electric motor as a power source constitutes the actuator 26.

[0048] The brake actuator controls the brake system according to the control signal from the automatic driving ECU 20, and controls the braking force applied to the wheels of the automatic driving vehicle 2. As the brake system, for example, a hydraulic brake system can be used. The steering actuator controls the driving of the auxiliary motor that controls the steering torque in the electric power steering system according to the control signal from the automatic driving ECU 20. Thus, the steering actuator controls the steering torque of the automatic driving vehicle 2.

[0049] Next, an example of the functional configuration of the automatic driving ECU 20 is described. The automatic driving ECU 20 includes a vehicle position acquisition unit 31, an external environment recognition unit 32, a driving state recognition unit 33, a remote instruction request unit (sensor information transmission unit) 34, a route generation unit 35, a remote instruction reception unit 36, a transmission time acquisition unit 37, a reception time acquisition unit 38, a delay determination unit 39, a rejection unit 40, and an automatic driving control unit 41.

[0050] The vehicle position acquisition unit 31 acquires the position information of the autonomous driving vehicle 2 based on the position information of the GPS receiving unit 21 and the map information of the map database 24. In addition, the vehicle position acquisition unit 31 may also acquire the position information of the autonomous driving vehicle 2 by using the target information of the map information included in the map database 24 and the detection result of the external sensor 22 through the SLAM (Simultaneous Localization and Mapping) technology. The vehicle position acquisition unit 31 may also identify the lateral position of the autonomous driving vehicle 2 relative to the lane (the position of the autonomous driving vehicle 2 in the lane width direction) based on the positional relationship between the lane dividing line and the autonomous driving vehicle 2, and include it in the position information. The vehicle position acquisition unit 31 may also acquire the position information of the autonomous driving vehicle 2 by other known methods.

[0051] The external environment recognition unit 32 recognizes the external environment of the autonomous driving vehicle 2 based on the detection results of the external sensor 22. The external environment includes the relative positions of the surrounding objects relative to the autonomous driving vehicle 2. The external environment may also include the relative speed and movement direction of the surrounding objects relative to the autonomous driving vehicle 2. The external environment may also include the types of objects such as other vehicles, pedestrians, bicycles, etc. The types of objects can be identified by well-known methods such as pattern matching. The external environment may also include the results of lane line recognition (white line recognition) around the autonomous driving vehicle 2. The external environment may also include the recognition results of the lighting state of the traffic light. The external environment recognition unit 32 can, for example, recognize the lighting state of the traffic light in front of the autonomous driving vehicle 2 (whether it is a lighting state that allows passage or a lighting state that prohibits passage, etc.) based on the image of the camera of the external sensor 22.

[0052] The driving state recognition unit 33 recognizes the driving state of the autonomous driving vehicle 2 based on the detection result of the internal sensor 23. The driving state includes the speed of the autonomous driving vehicle 2, the acceleration of the autonomous driving vehicle 2, and the yaw rate of the autonomous driving vehicle 2. Specifically, the driving state recognition unit 33 recognizes the speed of the autonomous driving vehicle 2 based on the speed information of the speed sensor. The driving state recognition unit 33 recognizes the acceleration of the autonomous driving vehicle 2 based on the speed information of the acceleration sensor. The driving state recognition unit 33 recognizes the direction of the autonomous driving vehicle 2 based on the yaw rate information of the yaw rate sensor.

[0053] The remote instruction request unit 34 determines whether to request a remote instruction from the remote commander R. The remote instruction request unit 34 determines whether to request a remote instruction based on at least one of the position information of the autonomous driving vehicle 2 acquired by the vehicle position acquisition unit 31 and the map information of the map database 24, the external environment recognized by the external environment recognition unit 32, and the route generated by the route generation unit 35 described later.

[0054] The remote instruction request unit 34 can, for example, determine that a remote instruction should be requested when the autonomous driving vehicle 2 reaches a preset remote instruction request location. The preset remote instruction request location can be included in the map information stored in the map database 24. The remote instruction request unit 34 can determine whether the remote instruction request location has been reached based on the position information of the autonomous driving vehicle 2 and the map information (information of the remote instruction request location).

[0055] When the remote instruction request unit 34 determines that the remote instruction should be requested, it requests the remote commander R to perform the remote instruction to the remote instruction device 1. The request for remote instruction includes, for example, identification information of the autonomous driving vehicle 2. In addition, the remote instruction request unit 34 may request the remote instruction with a margin in advance. The remote instruction request unit 34 may also determine that the remote instruction should be requested when the distance between the remote instruction request point (intersection, etc.) that is the object of the remote instruction and the autonomous driving vehicle 2 is less than a certain distance. The remote instruction request unit 34 may also use the remaining time to arrive instead of the distance.

[0056] When it is determined that a remote instruction should be requested, the remote instruction request unit 34 sends the detection information (sensor information) of the external sensor 22 that detects the external environment to the remote instruction device 1. The detection information of the vehicle-mounted sensor includes an image of the camera of the external sensor 22. The detection information of the vehicle-mounted sensor may also include, for example, an image in front of the autonomous driving vehicle 2 captured by the camera of the autonomous driving vehicle 2. The detection information of the vehicle-mounted sensor may also include images of the side and rear of the autonomous driving vehicle 2 and the surroundings of the autonomous driving vehicle 2. In addition, the detection information of the vehicle-mounted sensor may also include object information detected by the radar sensor of the external sensor 22. The detection information of the vehicle-mounted sensor may also include the recognition result of the type of the object.

[0057] The route generation unit 35 generates an autonomous driving route (trajectory) for the autonomous driving vehicle 2. The route generation unit 35 generates the autonomous driving route based on a preset driving path, map information, position information of the autonomous driving vehicle 2, the external environment of the autonomous driving vehicle 2, and the driving state of the autonomous driving vehicle 2.

[0058] The driving route refers to the route that the autonomous driving vehicle 2 travels during autonomous driving. The route generation unit 35 obtains the driving route for autonomous driving based on the destination, map information, and position information of the autonomous driving vehicle 2, for example. The driving route can also be set by a known navigation system. The destination can be set by the occupant of the autonomous driving vehicle 2, or can be automatically recommended by the autonomous driving ECU 20 or the navigation system.

[0059] The route includes the path along which the vehicle travels in an autonomous driving manner and the vehicle speed curve in the autonomous driving. The path is a prescribed trajectory along which the vehicle travels in the autonomous driving on the driving path. The path can be set, for example, as data (steering angle curve) of the steering angle change of the autonomous driving vehicle 2 corresponding to the position on the driving path. The position on the driving path refers to, for example, a set longitudinal position set at a prescribed interval (e.g., 1 m) in the direction of travel of the driving path. The steering angle curve refers to data associated with a target steering angle for each set longitudinal position.

[0060] The route generation unit 35 generates a route for the autonomous driving vehicle 2 to travel, for example, based on the travel route, map information, the external environment of the autonomous driving vehicle 2, and the travel state of the autonomous driving vehicle 2. The route generation unit 35 generates the route in such a way that the autonomous driving vehicle 2 passes through the center of the lane included in the travel route (the center in the lane width direction).

[0061] The vehicle speed curve is data that associates a target vehicle speed with each set longitudinal position, for example. In addition, the set longitudinal position may be set based on the travel time of the autonomous driving vehicle 2 instead of the distance. The set longitudinal position may also be set as the vehicle's arrival position after 1 second or the vehicle's arrival position after 2 seconds.

[0062] The route generation unit 35 generates a speed curve based on, for example, traffic control information such as the legal speed contained in the route and map information. Instead of the legal speed, a set speed pre-set for a location or road section on the map may be used. The route generation unit 35 generates an autonomous driving route based on the route and the speed curve. In addition, the route generation method in the route generation unit 35 is not limited to the above content, and a known method related to autonomous driving may be used. The same applies to the content of the route.

[0063] In the present embodiment, when a remote instruction request is sent from the remote instruction request unit 34 to the remote instruction device 1, as an example, the route generation unit 35 generates a first route and a second route. For example, the first route is a route that places more emphasis on safety than the second route. In the present embodiment, as an example, the first route is set as a route that stops the autonomous driving vehicle 2 when the autonomous driving vehicle 2 arrives at the remote instruction request location. For example, the second route is a route that places more emphasis on the efficiency of the driving of the autonomous driving vehicle 2 than the first route. The second route is a route corresponding to the remote instruction at the remote instruction request location. In the present embodiment, as an example, the second route is set as a route that causes the autonomous driving vehicle 2 to travel at the remote instruction request location based on the remote instruction from the remote instruction device 1.

[0064] Specifically, for example, when the autonomous driving vehicle 2 is turning right at the intersection, the route generation unit 35 generates a route for the autonomous driving vehicle 2 to turn right at the intersection as the second route in a manner corresponding to the remote instruction to start the right turn. The route generation unit 35 may also update the second route according to changes in the external environment until the remote instruction is received. In addition, the route generation unit 35 may also generate a route for going straight at the intersection in advance when there is a remote instruction to switch from turning right at the intersection to going straight at the intersection.

[0065] The remote instruction receiving unit 36 ​​receives the remote instruction transmitted from the remote instruction device 1 in response to the remote instruction request in the remote instruction requesting unit 34 .

[0066] The transmission time acquisition unit 37 acquires the instruction transmission time, which is the time when the remote instruction device 1 transmits the remote instruction. Specifically, the transmission time acquisition unit 37 acquires the instruction transmission time that the remote instruction device 1 transmits together with the remote instruction. The configuration of the remote instruction device 1 transmitting the instruction transmission time together with the remote instruction will be described in detail later.

[0067] The reception time acquisition unit 38 acquires the instruction reception time, which is the time when the automatic driving vehicle 2 receives the remote instruction from the remote instruction device 1. Specifically, the reception time acquisition unit 38 acquires the time when the remote instruction receiving unit 36 ​​receives the remote instruction from the remote instruction device 1 as the instruction reception time.

[0068] Here, in this embodiment, "time" is used to determine whether there is a communication delay between the remote instruction device 1 and the autonomous driving vehicle 2. Therefore, it is necessary to synchronize the time used by the remote instruction device 1 and the autonomous driving vehicle 2. For example, the remote instruction device 1 and the autonomous driving vehicle 2 may also use the GPS time used by GPS to synchronize the time. However, the remote instruction device 1 and the autonomous driving vehicle 2 are not limited to using the GPS time to synchronize the time. The remote instruction device 1 and the autonomous driving vehicle 2 can synchronize the time by various well-known methods. For example, the time can also be synchronized by using an NTP (Network Time Protocol) server for time calibration that is set in the remote instruction device 1, the autonomous driving vehicle 2 or other locations. In other embodiments described below, when the communication delay is determined by using the time, the time can be synchronized between the remote instruction device and the autonomous driving vehicle by the same method as described above.

[0069] The delay determination unit 39 determines whether a communication delay occurs between the remote instruction device 1 and the autonomous vehicle 2. Here, as a cause of the communication delay, for example, when data is sent and received between the autonomous vehicle 2 and the remote instruction device 1, the data passes through a plurality of access points, servers, or base stations. In addition, the data sent from the autonomous vehicle 2 to the remote instruction device 1 includes detection information of the external sensor 22, and as a cause of the communication delay, for example, the data volume of the detection information is large.

[0070] Specifically, the delay determination unit 39 determines whether there is a communication delay based on the difference between the instruction transmission time acquired by the transmission time acquisition unit 37 and the instruction reception time acquired by the reception time acquisition unit 38. Here, when the difference between the instruction transmission time when the remote instruction is transmitted from the remote instruction device 1 and the instruction reception time when the remote instruction is received by the remote instruction receiving unit 36 ​​is greater than a predetermined first threshold, the delay determination unit 39 determines that a communication delay has occurred. When the difference between the instruction transmission time and the instruction reception time is less than the first threshold, the delay determination unit 39 determines that no communication delay has occurred.

[0071] The first threshold value may be determined, for example, based on whether control of the driving of the automatic driving vehicle 2 is permitted according to the remote instruction, even if the remote instruction is earlier in time due to a communication delay.

[0072] When the delay determination unit 39 determines that a communication delay has occurred, the rejection unit 40 rejects the remote instruction sent from the remote instruction device 1 in response to the remote instruction request. Here, "rejection" means that the autonomous driving vehicle 2 does not follow the remote instruction sent in response to the remote instruction request.

[0073] When the rejection unit 40 rejects the remote instruction, the rejection unit 40 notifies the remote instruction device 1 of the rejection through the communication unit 25. The rejection notification received by the remote instruction device 1 can be used for various processes such as statistical processing related to communication delay.

[0074] The automatic driving control unit 41 performs automatic driving of the automatic driving vehicle 2. The automatic driving control unit 41 performs automatic driving of the automatic driving vehicle 2 based on, for example, the external environment of the automatic driving vehicle 2, the driving state of the automatic driving vehicle 2, and the route generated by the route generation unit 35. The automatic driving control unit 41 performs automatic driving of the automatic driving vehicle 2 by sending a control signal to the actuator 26.

[0075] When the autonomous driving vehicle 2 reaches the remote instruction request location, the autonomous driving control unit 41 stops the autonomous driving vehicle 2 based on the first route. That is, when the remote instruction request unit 34 sends a remote instruction request to the remote instruction device 1, the autonomous driving control unit 41 maintains the parking state of the autonomous driving vehicle 2 until the remote instruction from the remote instruction device 1 is received.

[0076] When receiving the remote instruction, the automatic driving control unit 41 switches the route used for driving control from the first route to the second route, and starts automatic driving of the automatic driving vehicle 2 based on the second route. That is, the automatic driving vehicle 2 starts to turn right from the parked state.

[0077] However, when the rejection unit 40 rejects the remote instruction when receiving the remote instruction from the remote instruction device 1, the automatic driving control unit 41 does not switch the route used for driving control from the first route to the second route corresponding to the remote instruction, but maintains the first route. That is, the automatic driving control unit 41 does not perform the control of driving according to the remote instruction. In this case, the automatic driving control unit 41 may also request the occupants of the automatic driving vehicle 2 to make a judgment on the driving or switch to manual driving. In addition, the automatic driving control unit 41 may also automatically perform emergency evacuation when neither the judgment of the occupants nor manual driving is possible. When the automatic driving control unit 41 does not perform the control of driving based on the remote instruction (switching of the route), various controls other than the control based on the remote instruction may be adopted as the control of the driving of the automatic driving vehicle 2.

[0078] When a remote instruction cannot be received even after a preset time, the automatic driving control unit 41 may also request the occupant of the automatic driving vehicle 2 to make a decision on driving or switch to manual driving. In addition, the automatic driving control unit 41 may also automatically perform emergency evacuation when neither the occupant's decision nor manual driving is possible.

[0079] (Composition of remote indicating device)

[0080] Next, the structure of the remote indicating device 1 according to the present embodiment will be described with reference to the accompanying drawings. Figure 1 As shown, the remote instruction device 1 includes a remote instruction server 10 and an instructor interface 3 .

[0081] First, the hardware configuration of the remote instruction server 10 will be described. Figure 3 1 is a block diagram showing an example of the hardware configuration of the remote instruction server 10. Figure 3 As shown, the remote instruction server 10 is configured as a general-purpose computer including a processor 10a, a storage unit 10b, a communication unit 10c, and a user interface 10d.

[0082] The processor 10a controls the remote instruction server 10 by operating various operating systems. The processor 10a is a computing unit such as a CPU [Central Processing Unit] including a control device, a computing device, a register, etc. The processor 10a centrally controls the storage unit 10b, the communication unit 10c, and the user interface 10d. The storage unit 10b is configured to include at least one of a memory and a storage system. The memory is a recording medium such as a ROM [Read Only Memory] and a RAM [Random Access Memory]. The storage system is a recording medium such as a HDD [Hard Disk Drive].

[0083] The communication unit 10c is a communication device for communicating via the network N. The communication unit 10c can use a network device, a network controller, a network card, etc. The user interface 10d is an input and output unit of the remote instruction server 10 for users such as administrators. The user interface 10d includes output devices such as a display and a speaker, and input devices such as a touch panel. In addition, the remote instruction server 10 does not necessarily have to be installed in a facility, and can also be installed in a mobile body such as a vehicle.

[0084] Figure 4 1 is a block diagram showing an example of the configuration of the remote instruction device 1. Figure 4 As shown, the commander interface 3 is an input / output unit of the remote instruction device 1 for the remote commander R. The commander interface 3 includes an output unit 3a and an instruction input unit 3b.

[0085] The output unit 3a is a device that outputs information for remote instruction of the automatic driving vehicle 2 to the remote commander R. The output unit 3a includes a display that outputs image information and a speaker that outputs sound information.

[0086] The display, for example, displays an image in front of the autonomous driving vehicle 2 (image of the scenery in front) captured by the camera of the autonomous driving vehicle 2. The display may have a plurality of display screens, and may display images to the side and / or rear of the autonomous driving vehicle 2. The display is not particularly limited as long as it has a configuration capable of providing visual information to the remote commander R. The display may also be a wearable device that is mounted to cover the eyes of the remote commander R.

[0087] The speaker is, for example, a head-mounted speaker mounted on the head of the remote commander R. The speaker transmits the status of the automated driving vehicle 2 (for example, when turning right at an intersection) to the remote commander R through sound. The speaker does not necessarily have to be head-mounted, but may be fixed.

[0088] The instruction input unit 3b is a device for inputting remote instructions from the remote commander R. The instruction input unit 3b includes, for example, an operating lever. In the instruction input unit 3b, for example, a remote instruction to drive the autonomous driving vehicle 2 is input by pushing the operating lever to the far side in the front-rear direction of the remote commander R, and a remote instruction to decelerate or stop the autonomous driving vehicle 2 is input by pushing the operating lever to the front side in the front-rear direction of the remote commander R.

[0089] The instruction input unit 3b may also include a button, and the remote commander R may input a remote instruction by pressing the button and simultaneously turning down the operating lever. The instruction input unit 3b may also include a touch panel. The touch panel may also be shared with the display of the output unit 3a. The instruction input unit 3b may also include an operating pedal.

[0090] The instruction input unit 3b may also have a voice recognition function and a gesture recognition function. The gesture of the remote commander R may be recognized by a camera and / or a radar sensor mounted on the commander interface 3. In the instruction input unit 3b, the remote instruction may be input by a combination of two or more of the operation of an operating lever, the operation of a button, the operation of a touch panel, the operation of an operating pedal, voice input, and gestures.

[0091] Next, the functional structure of the remote instruction server 10 is described. Figure 4 As shown, the remote instruction server 10 includes a remote instruction request receiving unit (sensor information receiving unit) 11 , an information providing unit 12 , and a remote instruction sending unit 13 .

[0092] The remote instruction request receiving unit 11 receives a remote instruction request sent from the autonomous driving vehicle 2 to the remote instruction device 1. In addition, the remote instruction request receiving unit 11 receives, together with the remote instruction request, detection information (sensor information) of the external sensor 22, which is sent from the remote instruction request unit 34 of the autonomous driving vehicle 2 that sent the remote instruction request.

[0093] When the remote instruction request receiving unit 11 receives the remote instruction request, the information providing unit 12 requests the responsible remote commander R to input the remote instruction through the commander interface 3 .

[0094] In addition, the information providing unit 12 provides information about the autonomous vehicle 2 to the remote commander R based on the detection information of the external sensor 22 in the autonomous vehicle 2 acquired by the remote instruction request receiving unit 11. The information providing unit 12 displays the image in front of the autonomous vehicle 2 on the display of the output unit 3a of the commander interface 3, for example. The information providing unit 12 may also display an image seen from the vicinity of the driver's seat of the autonomous vehicle 2 by converting the perspective. The information providing unit 12 may also display images of the side and rear of the autonomous vehicle 2. The information providing unit 12 may also display a panoramic image synthesized by images taken around the autonomous vehicle 2, or may display a bird's-eye view image generated by looking down on the autonomous vehicle 2 by image synthesis and perspective conversion. The information providing unit 12 may also perform highlighting of objects in the image (for example, displaying other vehicles surrounded by a frame). The information providing unit 12 may also display the recognition result of the lighting state of the signal light on the display when the signal light is included in the image.

[0095] The information providing unit 12 is not limited to the images captured by the camera of the autonomous driving vehicle 2, and various information may be displayed on the display. The information providing unit 12 may also display the status of the autonomous driving vehicle 2 for which the remote instruction is requested (such as the status of turning right at an intersection) using text or icons. The information providing unit 12 may also display the types of remote instructions that the remote commander R can select on the display. The information providing unit 12 may also display information related to the route of the autonomous driving vehicle 2 corresponding to the remote instruction (such as the trajectory of the autonomous driving vehicle 2 traveling in accordance with the traveling remote instruction) on the display.

[0096] The information providing unit 12 may also display information about objects detected by the radar sensor of the autonomous driving vehicle 2. The information about the object may be displayed as an icon in the overhead image. When the type of the object is identified, an icon corresponding to the type of the object may also be displayed. The information providing unit 12 may also display map information about the surroundings of the autonomous driving vehicle 2 obtained based on the position information of the autonomous driving vehicle 2 on the display. The map information may be located in the remote indication server 10, or may be obtained from other servers, etc. The map information about the surroundings of the autonomous driving vehicle 2 may also be obtained from the autonomous driving vehicle 2.

[0097] The information providing unit 12 may display the speed information of the autonomous driving vehicle 2 on the display, and may also display the information of the steering angle of the autonomous driving vehicle 2 on the display. The information providing unit 12 may also display the information of the slope of the road on which the autonomous driving vehicle 2 is located on the display. When the autonomous driving vehicle 2 has an in-cabin camera, the information providing unit 12 may display the image of the cabin of the autonomous driving vehicle 2 as needed.

[0098] The information providing unit 12 may provide voice information to the remote commander R through the speaker of the output unit 3a of the commander interface 3. In this case, the information providing unit 12 may output, for example, other vehicles approaching around the autonomous driving vehicle 2 as voice or sound from the speaker. In addition, the information provision through the speaker is not essential.

[0099] When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction sending unit 13 sends the input remote instruction to the autonomous driving vehicle 2. In addition, the remote instruction sending unit 13 sends the time when the remote instruction is sent (the time when the remote instruction is sent), that is, the instruction sending time, to the autonomous driving vehicle 2 together with the remote instruction.

[0100] When a remote instruction input by a remote commander R is sent to the autonomous driving vehicle 2, the information providing unit 12 may continue to transmit the information of the autonomous driving vehicle 2 to the remote commander R, or may switch to the information of another autonomous driving vehicle 2 requesting the remote instruction.

[0101] (Processing flow of vehicle remote indication system)

[0102] The flow of a process in which the remote instruction device 1 transmits a remote instruction to the automatically driven vehicle 2 based on a remote instruction request will be described. Figure 5 1 is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device 1. Figure 5 The processing shown is started when the remote instruction request receiving unit 11 receives a remote instruction request.

[0103] like Figure 5 As shown, if the remote instruction request receiving unit 11 receives a remote instruction request from the autonomous driving vehicle 2, the information providing unit 12 requests the remote commander R to input a remote instruction (S101). When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction sending unit 13 sends the input remote instruction to the autonomous driving vehicle 2 (S102). In addition, when sending a remote instruction, the remote instruction sending unit 13 sends the instruction sending time together with the remote instruction to the autonomous driving vehicle 2 (S103).

[0104] Next, the flow of control performed according to the received remote instruction or the process of rejecting the remote instruction after a remote instruction request is made in the automatic driving vehicle 2 will be described. Figure 6 1 is a flowchart showing an example of a rejection judgment process of a remote instruction executed by the automatic driving vehicle 2. Figure 6 The illustrated process starts when the autonomous driving vehicle 2 arrives at the remote instruction request point and determines that the remote instruction request unit 34 should request a remote instruction.

[0105] like Figure 6 As shown, when it is determined that a remote instruction should be requested, the remote instruction request unit 34 sends a remote instruction request to the remote instruction device 1 (S201). The remote instruction receiving unit 36 ​​receives the remote instruction sent from the remote instruction device 1 (S202). The sending time acquisition unit 37 acquires the instruction sending time sent by the remote instruction device 1 together with the remote instruction (S203). The receiving time acquisition unit 38 acquires the instruction receiving time when the remote instruction receiving unit 36 ​​receives the remote instruction (S204).

[0106] The delay determination unit 39 determines whether a communication delay has occurred based on whether the difference between the instruction transmission time and the instruction reception time is greater than or equal to a first threshold value (S205). When the difference between the instruction transmission time and the instruction reception time is not greater than or equal to the first threshold value (S205: No), the automatic driving control unit 41 switches the route used for driving control from the first route to the second route, and starts automatic driving of the automatic driving vehicle 2 based on the second route (S206).

[0107] When the difference between the instruction transmission time and the instruction reception time is greater than the first threshold value (S205: Yes), the rejection unit 40 rejects the remote instruction sent from the remote instruction device 1 in response to the remote instruction request (S207). As a result, the automatic driving control unit 41 does not switch the route used for driving control from the first route to the second route, but maintains the first route. In addition, when the rejection unit 40 rejects the remote instruction, the content of the rejection of the remote instruction is notified to the remote instruction device 1 via the communication unit 25 (S208).

[0108] As described above, when a communication delay occurs, the autonomous driving vehicle 2 rejects the remote instruction sent from the remote instruction device 1 in response to the remote instruction request. Here, for example, when a communication delay occurs between the remote instruction device 1 and the autonomous driving vehicle 2, the surrounding conditions may have changed when the autonomous driving vehicle 2 receives the remote instruction, and the received remote instruction may be inappropriate. Therefore, in the vehicle remote instruction system 100, when it is determined that a communication delay has occurred, the remote instruction sent in response to the remote instruction request is rejected. Thereby, it is possible to suppress the situation where the driving of the autonomous driving vehicle 2 is controlled based on inappropriate remote instructions. Therefore, even if a communication delay occurs between the remote instruction device 1 and the autonomous driving vehicle 2, the vehicle remote instruction system 100 can appropriately control the driving of the autonomous driving vehicle 2 by rejecting the remote instruction.

[0109] When the difference between the instruction sending time of sending the remote instruction and the instruction receiving time of receiving the remote instruction is greater than the first threshold, the delay determination unit 39 determines that a communication delay has occurred. In this case, the vehicle remote instruction system 100 can use the instruction sending time and the instruction receiving time to more appropriately determine whether a communication delay has occurred.

[0110] (Second Embodiment)

[0111] A second embodiment of the vehicle remote indication system will be described. Hereinafter, the differences from the vehicle remote indication system 100 involved in the first embodiment will be mainly described, and the same or similar components will be marked with the same reference numerals and detailed description will be omitted. Figure 7 As shown, the vehicle remote instruction system 100A according to the present embodiment is a system for performing remote instructions from a remote instruction device 1 to an autonomous driving vehicle 2A based on an operation of a remote commander R.

[0112] (Composition of autonomous vehicles)

[0113] As an example, the autonomous driving vehicle 2A has an autonomous driving ECU 20A. The autonomous driving ECU 20A functionally has a request timing acquisition unit 42 and a delay determination unit 39A, instead of the transmission timing acquisition unit 37 and the delay determination unit 39 of the autonomous driving ECU 20 of the autonomous driving vehicle 2 involved in the first embodiment. The following description will focus on the request timing acquisition unit 42 and the delay determination unit 39A.

[0114] The request time acquisition unit 42 acquires the request transmission time when the autonomous driving vehicle 2 transmits the remote instruction request to the remote instruction device 1. Here, the request time acquisition unit 42 acquires and stores the time when the remote instruction request unit 34 transmits the remote instruction request as the request transmission time.

[0115] The delay determination unit 39A determines whether a communication delay has occurred between the remote instruction device 1 and the automatic driving vehicle 2A. Specifically, the delay determination unit 39A determines whether a communication delay has occurred based on the difference between the request transmission time acquired by the request time acquisition unit 42 and the instruction reception time acquired by the reception time acquisition unit 38. Here, when the difference between the request transmission time when the remote instruction request unit 34 transmits the remote instruction request and the instruction reception time when the remote instruction receiving unit 36 ​​receives the remote instruction is greater than the second predetermined threshold, the delay determination unit 39A determines that a communication delay has occurred. When the difference between the request transmission time and the instruction reception time is less than the second threshold, the delay determination unit 39A determines that no communication delay has occurred.

[0116] The second threshold value may be determined, for example, based on whether control of the driving of the automatic driving vehicle 2 is permitted according to the remote instruction, even if the remote instruction is earlier in time due to a communication delay.

[0117] In addition, in the determination of communication delay in this embodiment, unlike the first embodiment, the time when the remote instruction is sent, that is, the instruction sending time, is not used. Therefore, the remote instruction sending unit 13 of the remote instruction server 10 does not need to send the instruction sending time together with the remote instruction to the automatic driving vehicle 2.

[0118] (Processing flow of vehicle remote indication system)

[0119] The flow of a process in which the remote instruction device 1 transmits a remote instruction to the automatically driven vehicle 2 based on a remote instruction request will be described. Figure 8 1 is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device 1. Figure 8 The processing shown is started when the remote instruction request receiving unit 11 receives a remote instruction request.

[0120] like Figure 8 As shown, if the remote instruction request receiving unit 11 receives a remote instruction request from the autonomous driving vehicle 2A, the information providing unit 12 requests the remote commander R to input a remote instruction (S301). When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction sending unit 13 sends the input remote instruction to the autonomous driving vehicle 2A (S302).

[0121] Next, the flow of performing control according to the received remote instruction or rejecting the remote instruction after a remote instruction request is made in the automatic driving vehicle 2A is described. Fig. 9 This is a flowchart showing an example of a rejection determination process of a remote instruction executed by the automatic driving vehicle 2A. Fig. 9 The processing shown is started when the autonomous driving vehicle 2A arrives at the remote instruction request point and determines that the remote instruction request unit 34 should request a remote instruction.

[0122] like Fig. 9 As shown, when it is determined that a remote instruction should be requested, the remote instruction request unit 34 sends a remote instruction request to the remote instruction device 1 (S401). The request time acquisition unit 42 acquires the time when the remote instruction request unit 34 sends the remote instruction request, that is, the request sending time (S402). The remote instruction receiving unit 36 ​​receives the remote instruction sent from the remote instruction device 1 (S403). The reception time acquisition unit 38 acquires the instruction reception time when the remote instruction receiving unit 36 ​​receives the remote instruction (S404).

[0123] The delay determination unit 39A determines whether there is a communication delay based on whether the difference between the request transmission time and the instruction reception time is greater than or equal to the second threshold value (S405). The processing of S406 to S408 is similar to that of the first embodiment. Figure 6 The processing of S206 to S208 described above is the same, and the detailed description is omitted.

[0124] As described above, when a communication delay occurs, the autonomous driving vehicle 2A rejects the remote instruction sent from the remote instruction device 1 in response to the remote instruction request. When the difference between the request sending time when the remote instruction request is sent and the instruction receiving time when the remote instruction is received is greater than the second threshold, the delay determination unit 39A determines that a communication delay has occurred. In this case, the vehicle remote instruction system 100A can use the request sending time and the instruction receiving time to more appropriately determine whether a communication delay has occurred.

[0125] (Third Embodiment)

[0126] A third embodiment of the vehicle remote instruction system is described. The vehicle remote instruction system of this embodiment is different from the vehicle remote instruction systems 100 and 100A involved in the first and second embodiments, and uses detection information of an external sensor instead of time to determine whether a communication delay has occurred. In the following, the description will focus on the differences from the vehicle remote instruction system 100 involved in the first embodiment, and the same or similar components are marked with the same reference numerals, and detailed descriptions are omitted.

[0127] like Fig.10 As shown, the vehicle remote instruction system 100B according to the present embodiment is a system in which a remote instruction device 1B performs remote instruction on an autonomous driving vehicle 2B based on an operation of a remote commander R.

[0128] (Composition of autonomous vehicles)

[0129] As an example, the autonomous driving vehicle 2B has an autonomous driving ECU 20B. The autonomous driving ECU 20B functionally has a transmission time sensor information acquisition unit 43, a reception time sensor information acquisition unit 44, and a delay judgment unit 39B, which replace the transmission time acquisition unit 37, the reception time acquisition unit 38, and the delay judgment unit 39 of the autonomous driving ECU 20 of the autonomous driving vehicle 2 involved in the first embodiment. The following description focuses on the transmission time sensor information acquisition unit 43, the reception time sensor information acquisition unit 44, and the delay judgment unit 39B.

[0130] The sending time sensor information acquisition unit 43 acquires the detection information received by the remote instruction request receiving unit 11 of the remote instruction device 1B at the time when the remote instruction device 1B sends the remote instruction as the instruction sending time sensor information. Specifically, the sending time sensor information acquisition unit 43 acquires the instruction sending time sensor information sent by the remote instruction device 1B together with the remote instruction. The configuration of the instruction sending time sensor information sent by the remote instruction device 1B will be described in detail later.

[0131] The reception-time sensor information acquisition unit 44 acquires the detection information detected by the external sensor 22 at the time when the autonomous driving vehicle 2B receives the remote instruction as the instruction reception-time sensor information. Specifically, the reception-time sensor information acquisition unit 44 acquires the detection information detected by the external sensor 22 at the time when the remote instruction receiving unit 36 ​​receives the remote instruction from the remote instruction device 1B as the instruction reception-time sensor information.

[0132] The delay determination unit 39B determines whether a communication delay has occurred based on the instruction transmission sensor information acquired by the transmission sensor information acquisition unit 43 and the instruction reception sensor information acquired by the reception sensor information acquisition unit 44. In addition, the instruction transmission sensor information and the instruction reception sensor information used for determination include, for example, a camera image, an optical radar point group, a target detected by a radar sensor, and the like.

[0133] Here, the delay determination unit 39B compares the sensor information when the instruction is sent and the sensor information when the instruction is received, and determines whether a communication delay has occurred based on the deviation between the two. For example, when the deviation between the sensor information when the instruction is sent and the sensor information when the instruction is received is greater than a predetermined first deviation threshold, the delay determination unit 39B can determine that a communication delay has occurred. For example, when the deviation between the sensor information when the instruction is sent and the sensor information when the instruction is received is less than a predetermined first deviation threshold, the delay determination unit 39B can determine that no communication delay has occurred.

[0134] The first deviation threshold can be determined, for example, as follows, based on whether the driving of the autonomous driving vehicle 2B is allowed to be controlled according to the received remote instruction, even if a deviation occurs between the external environment of the autonomous driving vehicle 2B used when the remote instruction is input and the external environment when the remote instruction is received.

[0135] Fig.11 (a) and Fig.11 (b) is a diagram showing the detection information of the external sensor 22 when the autonomous driving vehicle 2 is about to turn right (here, a camera image in front of the autonomous driving vehicle 2B). Fig.11(a) is a diagram showing detection information (instruction transmission time sensor information) received by the remote instruction request receiving unit 11 of the remote instruction device 1B at the time when the remote instruction device 1B transmits the remote instruction. Fig.11 (b) is a diagram showing detection information (instruction reception time sensor information) detected by the external sensor 22 at the time when the autonomous driving vehicle 2B receives the remote instruction. Fig.11 (a) and Fig.11 The vehicle V in (b) is an oncoming vehicle traveling in front of the autonomous driving vehicle 2B. If the communication delay between the remote instruction device 1B and the autonomous driving vehicle 2B increases, the difference in the detection information between the two also increases. Fig.11 (a) and Fig.11 In the example shown in (b), if the communication delay increases, the deviation of the position of the vehicle V also increases.

[0136] An example of a method of comparing the sensor information at the time of instruction transmission and the sensor information at the time of instruction reception performed by the delay determination unit 39B will be described below.

[0137] (Comparison of sensor low level)

[0138] The delay determination unit 39B can directly compare the detection values ​​(sensor values) of the sensor information when the instruction is sent and the sensor information when the instruction is received. For example, when the sensor information when the instruction is sent and the sensor information when the instruction is received are camera images, the delay determination unit 39 uses a method such as POC (Phase Only Correlation: pure phase correlation) to match Fig.11 (a) and Fig.11 (b) The delay determination unit 39B can calculate the correlation between the camera images by using the correlation value obtained when performing matching. The delay determination unit 39B can use the correlation between the camera images as the deviation between the sensor information when indicating transmission and the sensor information when indicating reception. Similarly, when the sensor information when indicating transmission and the sensor information when indicating reception are optical radar point groups, the delay determination unit 39B can also directly compare the detection values ​​(sensor values) with each other.

[0139] (Comparison of recognition results)

[0140] The delay judgment unit 39B can respectively identify the external environment of the autonomous driving vehicle 2B when the remote instruction is sent and the external environment of the autonomous driving vehicle 2B when the remote instruction is received based on the sensor information when the instruction is sent and the sensor information when the instruction is received, and judge the deviation based on the recognition result of the external environment. In this case, for example, the delay judgment unit 39B can also extract only the specified moving obstacles when recognizing the respective external environments when the remote instruction is sent and when the remote instruction is received, and compare the position information of the extracted moving obstacles. For example, the extracted moving obstacles can be all dynamic obstacles related to the route of the autonomous driving vehicle 2B.

[0141] (Composition of remote indicating device)

[0142] like Fig.12 As shown, the remote instruction device 1B includes a remote instruction server 10B. The remote instruction server 10B functionally includes a remote instruction request receiving unit 11, an information providing unit 12, and a remote instruction transmitting unit 13B.

[0143] When the remote commander R inputs a remote instruction to the instruction input unit 3B of the commander interface 3, the remote instruction sending unit 13B sends the remote instruction to the autonomous driving vehicle 2B. In addition, the remote instruction sending unit 13B sends the detection information (detection information being received) received by the remote instruction request receiving unit 11 at the time when the remote instruction sending unit 13B sends the remote instruction as the instruction sending time sensor information to the autonomous driving vehicle 2B together with the remote instruction.

[0144] (Processing flow of vehicle remote indication system)

[0145] The following describes a process flow of the remote instruction device 1B transmitting a remote instruction to the automatically driven vehicle 2B based on a remote instruction request. Fig.13 1B is a flowchart showing an example of a remote instruction transmission process executed by the remote instruction device 1B. Fig.13 The processing shown is started when the remote instruction request receiving unit 11 receives a remote instruction request.

[0146] like Fig.13 As shown, if the remote instruction request receiving unit 11 receives a remote instruction request from the autonomous driving vehicle 2B, the information providing unit 12 requests the remote commander R to input a remote instruction (S501). When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction sending unit 13B sends the input remote instruction to the autonomous driving vehicle 2B (S502). In addition, when sending a remote instruction, the remote instruction sending unit 13B sends the sensor information when the instruction is sent together with the remote instruction to the autonomous driving vehicle 2B (S503).

[0147] Next, the flow of performing control according to the received remote instruction or rejecting the remote instruction after a remote instruction request is made in the automatic driving vehicle 2B is described. Fig.14 This is a flowchart showing an example of a rejection determination process of a remote instruction executed by the automatic driving vehicle 2B. Fig.14 The processing shown is started when the autonomous driving vehicle 2B arrives at the remote instruction request point and determines that the remote instruction request unit 34 should request a remote instruction.

[0148] like Fig.14 As shown in FIG. 6 , when it is determined that a remote instruction should be requested, the remote instruction request unit 34 sends a remote instruction request to the remote instruction device 1B (S601). The remote instruction receiving unit 36 ​​receives the remote instruction sent from the remote instruction device 1B (S602). The sending sensor information acquisition unit 43 acquires the instruction sending sensor information sent by the remote instruction device 1B together with the remote instruction (S603). The receiving sensor information acquisition unit 44 acquires the detection information detected by the external sensor 22 at the time when the remote instruction receiving unit 36 ​​receives the remote instruction from the remote instruction device 1B as the instruction receiving sensor information (S604).

[0149] The delay determination unit 39B determines whether there is a communication delay based on whether the difference between the sensor information at the time of instruction transmission and the sensor information at the time of instruction reception is greater than the first difference threshold (S605). The processing of S606 to S608 is similar to that of the first embodiment. Figure 6 The processing of S206 to S208 described above is the same, and the detailed description is omitted.

[0150] As described above, when a communication delay occurs, the automatic driving vehicle 2B rejects the remote instruction sent from the remote instruction device 1B in response to the remote instruction request. The delay determination unit 39B determines whether a communication delay occurs based on the instruction sending sensor information acquired by the sending sensor information acquisition unit 43 and the instruction receiving sensor information acquired by the receiving sensor information acquisition unit 44. In this case, the vehicle remote instruction system 100B can more appropriately determine whether a communication delay occurs based on the instruction sending sensor information and the instruction receiving sensor information.

[0151] (Fourth Embodiment)

[0152] A fourth embodiment of the vehicle remote instruction system is described. The vehicle remote instruction system of this embodiment, like the vehicle remote instruction system 100B involved in the third embodiment, uses detection information of an external sensor to determine whether a communication delay has occurred. In the following, the description will focus on the differences from the vehicle remote instruction system 100B involved in the third embodiment, and the same or similar components are marked with the same reference numerals, and detailed descriptions are omitted.

[0153] like Fig.15 As shown, the vehicle remote instruction system 100C according to the present embodiment is a system that performs remote instruction from a remote instruction device 1B to an autonomous driving vehicle 2C based on an operation by a remote commander R.

[0154] (Composition of autonomous vehicles)

[0155] As an example, the autonomous driving vehicle 2C has an autonomous driving ECU 20C. The autonomous driving ECU 20C functionally has a sensor information acquisition unit 45 and a delay judgment unit 39C on request, instead of the sensor information acquisition unit 43 and the delay judgment unit 39B on transmission of the autonomous driving ECU 20B of the autonomous driving vehicle 2B involved in the third embodiment. The following description will focus on the sensor information acquisition unit 45 and the delay judgment unit 39C on request.

[0156] The request-time sensor information acquisition unit 45 acquires the detection information detected by the external sensor 22 at the time when the automatic driving vehicle 2B sends the remote instruction request as the instruction request-time sensor information. Here, the request-time sensor information acquisition unit 45 acquires and stores the detection information detected by the external sensor 22 when the remote instruction request unit 34 sends the remote instruction request as the instruction request-time sensor information.

[0157] The delay determination unit 39C determines whether a communication delay has occurred based on the instruction request time sensor information acquired by the request time sensor information acquisition unit 45 and the instruction reception time sensor information acquired by the reception time sensor information acquisition unit 44. In addition, the instruction request time sensor information and the instruction reception time sensor information used for determination include, for example, a camera image, an optical radar point group, a target detected by a radar sensor, and the like.

[0158] Here, the delay determination unit 39C compares the sensor information when the instruction is requested and the sensor information when the instruction is received, and determines whether a communication delay has occurred based on the deviation between the two. For example, when the deviation between the sensor information when the instruction is requested and the sensor information when the instruction is received is greater than a predetermined second deviation threshold, the delay determination unit 39C can determine that a communication delay has occurred. For example, when the deviation between the sensor information when the instruction is requested and the sensor information when the instruction is received is less than a predetermined second deviation threshold, the delay determination unit 39C can determine that no communication delay has occurred.

[0159] The second deviation threshold can be determined, for example, based on whether the driving of the autonomous driving vehicle 2C is allowed to be controlled according to the received remote instruction, even if the external conditions of the autonomous driving vehicle 2C change during the period from when the remote instruction is requested to when the remote instruction is received.

[0160] As a comparison method used by the delay determination unit 39C to compare the instruction request sensor information and the instruction reception sensor information, a comparison method similar to that of the delay determination unit 39B according to the third embodiment may be used, for example.

[0161] In addition, in the determination of the communication delay in the present embodiment, unlike the third embodiment, the instruction transmission time sensor information received by the remote instruction request receiving unit 11 of the remote instruction device 1B at the time when the remote instruction device 1B transmits the remote instruction is not used. Therefore, the remote instruction transmitting unit 13B of the remote instruction server 10B does not need to transmit the instruction transmission time sensor information together with the remote instruction to the automatic driving vehicle 2C.

[0162] (Processing flow of vehicle remote indication system)

[0163] In the remote instruction device 1B, the process and use of sending the remote instruction to the automatic driving vehicle 2C based on the remote instruction request Figure 8 The flow of the remote instruction transmission process executed by the remote instruction device 1B according to the second embodiment described above is the same. Therefore, a detailed description of the flow of the remote instruction transmission process executed by the remote instruction device 1B will be omitted.

[0164] Next, the process of performing control according to the received remote instruction or rejecting the remote instruction after a remote instruction request is made in the automatic driving vehicle 2C is described. Fig.16 1 is a flowchart showing an example of a rejection judgment process of a remote instruction executed by the autonomous driving vehicle 2C. Fig.16 The processing shown is started when the autonomous driving vehicle 2C arrives at the remote instruction request point and determines that the remote instruction request unit 34 should request a remote instruction.

[0165] like Fig.16 As shown, when it is determined that a remote instruction should be requested, the remote instruction request unit 34 sends a remote instruction request to the remote instruction device 1B (S701). The request-time sensor information acquisition unit 45 acquires the detection information of the external sensor 22 at the time when the remote instruction receiving unit 36 ​​sends the remote instruction request as the instruction request-time sensor information (S702). The remote instruction receiving unit 36 ​​receives the remote instruction sent from the remote instruction device 1B (S703). The reception-time sensor information acquisition unit 44 acquires the detection information detected by the external sensor 22 at the time when the remote instruction receiving unit 36 ​​receives the remote instruction from the remote instruction device 1B as the instruction reception-time sensor information (S704).

[0166] The delay determination unit 39C determines whether there is a communication delay based on whether the difference between the sensor information at the time of the instruction request and the sensor information at the time of the instruction reception is greater than or equal to the second difference threshold (S705). The processing of S706 to S708 is similar to that of the first embodiment. Figure 6 The processing of S206 to S208 described above is the same, and the detailed description is omitted.

[0167] As described above, when a communication delay occurs, the autonomous driving vehicle 2C rejects the remote instruction sent from the remote instruction device 1B in response to the remote instruction request. The delay determination unit 39C determines whether a communication delay occurs based on the instruction request sensor information acquired by the request sensor information acquisition unit 45 and the instruction reception sensor information acquired by the reception sensor information acquisition unit 44. In this case, the vehicle remote instruction system 100C can more appropriately determine whether a communication delay occurs based on the instruction request sensor information and the instruction reception sensor information.

[0168] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments. The present disclosure can be implemented in various forms after various changes and improvements are made based on the above embodiments based on the knowledge of those skilled in the art.

[0169] The judgment processing in each delay judgment unit in the first embodiment to the fourth embodiment can be appropriately combined. For example, the judgment processing in the delay judgment unit 39 in the first embodiment and the judgment processing in the delay judgment unit 39B in the third embodiment can be combined. In this case, if it is determined that a delay has occurred in these judgment processes, the rejection unit can reject the remote instruction. For example, the judgment processing in the delay judgment unit 39A in the second embodiment and the judgment processing in the delay judgment unit 39C in the fourth embodiment can be combined. In this case, if it is determined that a delay has occurred in these judgment processes, the rejection unit can reject the remote instruction.

[0170] For example, the vehicle remote instruction system 100 involved in the first embodiment may not include the autonomous driving vehicle 2 in the components. In this case, the vehicle remote instruction system 100 is equivalent to the remote instruction device 1. Similarly, for other embodiments, the vehicle remote instruction system may not include the autonomous driving vehicle in the components.

[0171] In the vehicle remote instruction system 100 involved in the first embodiment, it is also possible that the remote instruction server 10 rather than the autonomous driving vehicle 2 determines whether the remote instruction is required. In this case, the remote instruction server 10 determines whether the autonomous driving vehicle 2 is in a state where a remote instruction should be requested based on the detection information of the external sensor 22 obtained from the autonomous driving vehicle 2. When it is determined that the autonomous driving vehicle 2 is in a state where a remote instruction should be requested, the remote instruction server 10 can send a standby instruction to the autonomous driving vehicle 2 and request the remote commander R to input a remote instruction. Similarly, for the vehicle remote instruction system involved in other embodiments, the remote instruction server can determine whether a remote instruction is required.

[0172] In addition, in the vehicle remote instruction system 100 involved in the first embodiment, the remote instruction device 1 can be mounted on the vehicle. In this case, the remote commander R is also in the vehicle. The remote instruction server 10 can also be a cloud server composed of ECUs of multiple vehicles. Similarly, for other embodiments, the remote instruction device can be mounted on the vehicle.

Claims

1. A vehicle remote instruction system, wherein a remote instruction request is sent from an autonomous driving vehicle to a remote instruction device, and the driving of the autonomous driving vehicle is controlled based on the remote instruction sent from the remote instruction device in response to the remote instruction request, The vehicle remote indication system is characterized by having: a delay determination unit mounted on the autonomous driving vehicle and configured to determine whether a predetermined communication delay that makes it impossible to control the driving of the autonomous driving vehicle according to the remote instruction occurs between the remote instruction device and the autonomous driving vehicle; a rejecting unit for rejecting the remote instruction sent in response to the remote instruction request when the delay determining unit determines that the communication delay has occurred; a transmission-time sensor information acquisition unit that acquires sensor information when the instruction is transmitted; and a receiving sensor information acquisition unit, which acquires the sensor information when the instruction is received, The autonomous driving vehicle includes a sensor information transmitting unit that transmits sensor information of a vehicle-mounted sensor that detects an external environment to the remote instruction device. The remote instruction device includes a sensor information receiving unit that receives the sensor information transmitted from the sensor information transmitting unit. The transmission-time sensor information acquisition unit acquires the sensor information received by the sensor information receiving unit at the time when the remote instruction device sends the remote instruction as the instruction transmission-time sensor information, The reception-time sensor information acquisition unit acquires the sensor information detected by the vehicle-mounted sensor at the time when the autonomous driving vehicle receives the remote instruction as the instruction-received sensor information, The delay determination unit determines whether the communication delay has occurred based on the instruction transmission sensor information acquired by the transmission sensor information acquisition unit and the instruction reception sensor information acquired by the reception sensor information acquisition unit.

2. A vehicle remote instruction system, wherein a remote instruction request is sent from an autonomous vehicle to a remote instruction device, and the driving of the autonomous vehicle is controlled based on the remote instruction sent from the remote instruction device in response to the remote instruction request, The vehicle remote indication system is characterized by having: a delay determination unit mounted on the autonomous driving vehicle and configured to determine whether a predetermined communication delay that makes it impossible to control the driving of the autonomous driving vehicle according to the remote instruction occurs between the remote instruction device and the autonomous driving vehicle; a rejecting unit for rejecting the remote instruction sent in response to the remote instruction request when the delay determining unit determines that the communication delay has occurred; a sensor information acquisition unit at a time of request, which acquires sensor information detected by a vehicle-mounted sensor that detects an external environment of the autonomous driving vehicle at a time when the autonomous driving vehicle sends the remote instruction request as sensor information at a time of instruction request; and a reception-time sensor information acquisition unit, which acquires the sensor information detected by the vehicle-mounted sensor at the time when the autonomous driving vehicle receives the remote instruction as the instruction-received sensor information, The delay determination unit determines whether the communication delay has occurred based on the instruction request sensor information acquired by the request sensor information acquisition unit and the instruction reception sensor information acquired by the reception sensor information acquisition unit.

3. The vehicle remote indication system according to claim 1 or 2, It further comprises: a sending time acquisition unit which acquires the time when the remote instruction device sends the remote instruction, that is, the instruction sending time; and a receiving time acquisition unit, which acquires the time when the autonomous driving vehicle receives the remote instruction, that is, the instruction reception time, The delay determination unit determines that the communication delay has occurred when the difference between the instructed transmission time acquired by the transmission time acquisition unit and the instructed reception time acquired by the reception time acquisition unit is equal to or larger than a first threshold.

4. The vehicle remote indication system according to claim 1 or 2, The system further comprises: a request time acquisition unit, which acquires the time when the automatic driving vehicle sends the remote instruction request to the remote instruction device, that is, the request sending time; and a receiving time acquisition unit, which acquires the time when the autonomous driving vehicle receives the remote instruction, that is, the instruction reception time, The delay determination unit determines that the communication delay has occurred when the difference between the request transmission time acquired by the request time acquisition unit and the instruction reception time acquired by the reception time acquisition unit is equal to or larger than a second threshold.

Images