Vehicle remote control system

Abstract

To appropriately provide, to a remote commander, information on a route of a remote autonomous driving vehicle to allow the remote commander to determine a remote instruction when autonomous driving is changed over into remote driving.SOLUTION: A vehicle remote instruction system comprises: a remote autonomous driving vehicle capable of performing autonomous driving and remote driving to travel based on a remote instruction from a remote commander; and a remote instruction device including an output section for outputting a front image of the remote autonomous driving vehicle, so as to allow the remote commander to perform a remote instruction related to the remote driving with the use of the front image. The remote instruction device includes: a superimposition image generation section for generating a superimposition image being the front image on which a route recognition auxiliary image for enabling the remote commander to recognize the route of the remote autonomous driving vehicle is superimposed based on a peripheral state of the remote autonomous driving vehicle when the autonomous driving is changed over into the remote driving; and an information provision section for allowing the output section to display the superimposition image so as to provide information to the remote commander.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a vehicle remote instruction system that drives a remotely operated automated vehicle based on remote instructions from a remote commander. [Background technology]

[0002] For example, Patent Document 1 describes a mobile body remote control system that, when remotely controlling a mobile body capable of autonomous driving, draws an autonomous driving trajectory superimposed on a remote control screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-016188 Summary of the Invention [Problem to be solved by the invention]

[0004] A remotely driven autonomous vehicle can perform both autonomous driving and remote driving based on remote instructions from a remote commander. When no remote instructions are requested from the remotely driven autonomous vehicle, the remote commander is typically not aware of information such as the route of the remotely driven autonomous vehicle that is required to make decisions on the remote instructions. Therefore, when switching from autonomous driving to remote driving, depending on the surrounding conditions of the remotely driven autonomous vehicle, for example, road signs may not be visible, which can make it difficult for the remote commander to immediately grasp the route of the remotely driven autonomous vehicle. Therefore, in this technical field, when switching from autonomous driving to remote driving, it is required to appropriately provide the remote commander with information on the route of the remotely driven autonomous vehicle that is required for the remote commander to make decisions on the remote instructions. [Means for solving the problem]

[0005] One aspect of the present invention is a vehicle remote instruction system comprising: a remotely driven autonomous vehicle capable of performing autonomous driving and remote driving based on remote instructions from a remote commander; and a remote instruction device having an output unit that outputs a forward image of the remotely driven autonomous vehicle, with the remote commander using the forward image to give remote instructions regarding the remote driving. The remote instruction device has a superimposed image generation unit that generates a superimposed image, which is a forward image on which a route recognition auxiliary image is superimposed to allow the remote commander to recognize the route of the remotely driven autonomous vehicle, based on the surrounding conditions of the remotely driven vehicle when switching from autonomous driving to remote driving, and an information provision unit that displays the superimposed image on the output unit to provide information to the remote commander.

[0006] According to one aspect of the present invention, the vehicle remote instruction system includes a superimposed image generation unit that generates a superimposed image based on the surrounding conditions of the remotely-driven autonomous vehicle when switching from autonomous driving to remote driving. The superimposed image is a forward image on which a route recognition auxiliary image is superimposed to enable the remote commander to recognize the route of the remotely-driven autonomous vehicle. The information provision unit displays the superimposed image on an output unit to provide information to the remote commander. This makes it easier for the remote commander to recognize the route of the remotely-driven autonomous vehicle according to the surrounding conditions of the remotely-driven vehicle by referring to the route recognition auxiliary image superimposed on the forward image when switching from autonomous driving to remote driving. Therefore, the vehicle remote instruction system makes it possible to appropriately provide the remote commander with information about the route of the remotely-driven autonomous vehicle that allows the remote commander to make remote instruction decisions when switching from autonomous driving to remote driving.

[0007] In one embodiment, the superimposed image generation unit determines whether or not to superimpose the route recognition auxiliary image on the forward image based on the surrounding conditions of the remotely driven autonomous vehicle when switching from autonomous driving to remote driving, and if the superimposed image generation unit determines that superimposition is not necessary, the information provision unit may provide information to the remote commander by displaying on the output unit a forward image without the route recognition auxiliary image superimposed. In this case, for example, interference between the route recognition auxiliary image and traffic information objects in the forward image is suppressed, thereby suppressing the route recognition auxiliary image from interfering with the remote commander's recognition of traffic information objects.

[0008] In one embodiment, the remote instruction device may have a route acquisition unit that acquires display information, including information on the position and destination of the remotely driven autonomous vehicle, used to display the route recognition assistance image in response to transmission from the remotely driven autonomous vehicle, and acquires the route of the remotely driven autonomous vehicle based on the acquired display information. In this case, the route recognition assistance image can be generated without using the route generated by the remotely driven autonomous vehicle, thereby suppressing an increase in communication burden when transmitting route information generated by the remotely driven autonomous vehicle would result in a communication burden. [Effects of the Invention]

[0009] According to the present invention, when switching from automatic driving to remote driving, it is possible to appropriately provide the remote commander with information about the route of the remotely operated automatic vehicle so that the remote commander can make decisions about remote instructions. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of an overall image of a vehicle remote instruction system according to an embodiment; [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of a remotely controlled autonomous driving vehicle. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a remote instruction server. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a remote instruction device. [Figure 5]10 is an example of a superimposed image illustrating a route recognition assist image. [Figure 6] 10 is an example of a superimposed image illustrating a route recognition assist image. [Figure 7] 10 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposed display. [Figure 8] 10 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposed display. [Figure 9] 10 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposed display. [Figure 10] 10 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposed display. [Figure 11] 10 is an example of a superimposed image illustrating a route recognition auxiliary image in which superimposed display is essential. [Figure 12] 10 is an example of a superimposed image illustrating a route recognition auxiliary image in which superimposed display is essential. [Figure 13] 10 is a flowchart illustrating an example of processing by an autonomous driving ECU. [Figure 14] 10 is a flowchart illustrating an example of processing by the remote instruction device. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, identical or corresponding elements will be designated by the same reference numerals, and redundant description will be omitted.

[0012] Fig. 1 is a diagram showing an example of an overall view of a vehicle remote instruction system according to an embodiment. The vehicle remote instruction system 100 shown in Fig. 1 is a system in which a remote commander R issues remote instructions regarding the driving of a remotely driven autonomous vehicle 2 based on detection information from an external sensor 22 that detects the external environment of the remotely driven autonomous vehicle 2. The remote instructions are instructions from the remote commander R regarding the driving of the remotely driven autonomous vehicle 2.

[0013] The remote instructions include instructions to proceed for the remotely driven autonomous vehicle 2 and instructions to stop for the remotely driven autonomous vehicle 2. The remote instructions may also include instructions to change lanes for the remotely driven autonomous vehicle 2. The remote instructions may also include instructions to offset and avoid obstacles ahead, instructions to overtake a leading vehicle, instructions to make an emergency evacuation, etc.

[0014] [Configuration of vehicle remote control system] As shown in FIG. 1, the vehicle remote instruction system 100 includes a remote instruction device 1 to which a remote commander R inputs remote instructions. The remote instruction device 1 is communicably connected to a plurality of remotely operated automatically driven vehicles 2 via a network N. The network N is a wireless communication network. Various types of information are sent to the remote instruction device 1 from the remotely operated automatically driven vehicles 2.

[0015] In the vehicle remote instruction system 100, for example, in response to a remote instruction request from the remotely driven automatically vehicle 2, the remote commander R is requested to input a remote instruction. The remote commander R inputs the remote instruction to the commander interface 3 of the remote instruction device 1 while referring to a captured image of the area ahead of the remotely driven automatically vehicle 2 (using the forward image). The remote instruction device 1 transmits the remote instruction to the remotely driven automatically vehicle 2 via the network N. The remotely driven automatically vehicle 2 drives automatically in accordance with the remote instruction.

[0016] The number of remote commanders R in vehicle remote instruction system 100 may be one or more. There is also no particular limit to the number of remotely driven vehicles 2 that can communicate with vehicle remote instruction system 100. A configuration in which multiple remote commanders R take turns giving remote instructions to one remotely driven vehicle 2, or a configuration in which one remote commander R gives remote instructions to two or more remotely driven vehicles 2, may also be used.

[0017] [Configuration of autonomous driving vehicle] First, an example of the configuration of the remotely automatically driven vehicle 2 will be described. FIG. 2 is a block diagram showing an example of the configuration of the remotely automatically driven vehicle 2. As shown in FIG. 2, the remotely automatically driven vehicle 2 has, as an example, an automatically driven ECU 20. The automatically driven ECU 20 is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The automatically driven ECU 20 realizes various functions, for example, by loading a program recorded in the ROM into the RAM and executing the program loaded into the RAM by the CPU. The automatically driven ECU 20 may be composed of multiple electronic units.

[0018] The autonomous driving ECU 20 is connected to a GPS (Global Positioning System) receiving unit 21, an external sensor 22, an internal sensor 23, a map database 24, a communication unit 25, and an actuator 26.

[0019] The GPS receiver 21 receives signals from three or more GPS satellites to measure the position of the remotely automatically driven vehicle 2 (for example, the latitude and longitude of the remotely automatically driven vehicle 2). The GPS receiver 21 transmits the measured position information of the remotely automatically driven vehicle 2 to the automatic driving ECU 20.

[0020] The external sensor 22 is an in-vehicle sensor that detects the external environment around the remotely automatically driven vehicle 2. The external sensor 22 transmits the detected information (sensor information) to the automatic driving ECU 20.

[0021] Specifically, the external sensor 22 includes at least a camera. The camera is an imaging device that captures images of the external environment of the remotely controlled autonomously driven vehicle 2. The camera is provided, for example, behind the windshield of the remotely controlled autonomously driven vehicle 2, and captures images of the area in front of the vehicle. The camera transmits image information related to the external environment of the remotely controlled autonomously driven vehicle 2 to the autonomous driving ECU 20. The camera may be a monocular camera or a stereo camera. The camera may also be a camera that uses visible light or an infrared camera.

[0022] The external sensor 22 may include a radar sensor. A radar sensor is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the remotely controlled autonomous vehicle 2. Examples of radar sensors include radar (millimeter wave radar) or LiDAR (Light Detection and Ranging). A radar sensor detects objects by transmitting radio waves or light to the area around the remotely controlled autonomous vehicle 2 and receiving the radio waves or light reflected by the objects. The radar sensor transmits information about the detected objects to the autonomous driving ECU 20. Objects include fixed objects such as guardrails and buildings, as well as moving objects such as pedestrians, bicycles, and other vehicles.

[0023] The internal sensor 23 is an on-board sensor that detects the driving state of the remotely automatically driven vehicle 2. The internal sensor 23 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the remotely automatically driven vehicle 2. As the vehicle speed sensor, a wheel speed sensor that is provided on the wheels of the remotely automatically driven vehicle 2 or on a drive shaft that rotates integrally with the wheels and detects the rotational speed of each wheel can be used. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the automatic driving ECU 20.

[0024] The acceleration sensor is a detector that detects the acceleration of the remotely automatically driven vehicle 2. The acceleration sensor includes, for example, a longitudinal acceleration sensor that detects the acceleration in the longitudinal direction of the remotely automatically driven vehicle 2. The acceleration sensor may also include a lateral acceleration sensor that detects the lateral acceleration of the remotely automatically driven vehicle 2. The acceleration sensor transmits, for example, acceleration information of the remotely automatically driven vehicle 2 to the automatic driving ECU 20. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the remotely automatically driven vehicle 2. A gyro sensor, for example, can be used as the yaw rate sensor. The yaw rate sensor transmits the detected yaw rate information of the remotely automatically driven vehicle 2 to the automatic driving ECU 20.

[0025] The map database 24 is a database that records map information. The map database 24 is formed, for example, in a storage device such as a hard disk drive (HDD) mounted on the remotely controlled autonomous vehicle 2. The map information includes road position information, road shape information (e.g., curvature information), and position information of intersections and branching points. The map information may also include traffic regulation information such as legal maximum speed limits associated with the position information. The map information may also include landmark information used to acquire the position information of the remotely controlled autonomous vehicle 2. Examples of landmarks that may be used include road signs, road markings, traffic lights, utility poles, etc. The map database 24 may be configured in a server that can communicate with the remotely controlled autonomous vehicle 2.

[0026] The communication unit 25 is a communication device that controls wireless communication with the outside of the remotely controlled autonomously driven vehicle 2. The communication unit 25 transmits and receives various information to and from the remote instruction device 1 (remote instruction server 10) via the network N.

[0027] The actuators 26 are devices used to control the remotely automatically driven vehicle 2. The actuators 26 include at least a drive actuator, a brake actuator, and a steering actuator. The drive actuator controls, for example, the amount of air supplied to the engine (throttle opening) in response to a control signal from the automatic driving ECU 20, thereby controlling the driving force of the remotely automatically driven vehicle 2. If the remotely automatically driven vehicle 2 is a hybrid vehicle, in addition to the amount of air supplied to the engine, a control signal from the automatic driving ECU 20 is input to a motor serving as a power source to control the driving force. If the remotely automatically driven vehicle 2 is an electric vehicle, a control signal from the automatic driving ECU 20 is input to a motor serving as a power source to control the driving force. In these cases, the motor serving as a power source constitutes the actuator 26.

[0028] The brake actuator controls the brake system in response to a control signal from the autonomous driving ECU 20, and controls the braking force applied to the wheels of the remotely autonomously driven vehicle 2. A hydraulic brake system, for example, can be used as the brake system. The steering actuator controls the drive of an assist motor, which controls the steering torque of the electric power steering system, in response to a control signal from the autonomous driving ECU 20. In this way, the steering actuator controls the steering torque of the remotely autonomously driven vehicle 2.

[0029] Next, we will explain an example of the functional configuration of the autonomous driving ECU 20. The autonomous driving ECU 20 has a vehicle position acquisition unit 31, an external environment recognition unit 32, a driving state recognition unit 33, a remote instruction determination unit 34, a display information acquisition unit 35, a display information transmission unit 36, a route generation unit 37, and an autonomous driving control unit 38.

[0030] The vehicle position acquisition unit 31 acquires position information (position on a map) of the remotely driven autonomous vehicle 2 based on the position information from the GPS receiver 21 and the map information from the map database 24. The vehicle position acquisition unit 31 may acquire the position information of the remotely driven autonomous vehicle 2 using SLAM (Simultaneous Localization and Mapping) technology, using target information included in the map information from the map database 24 and the detection results of the external sensor 22. The vehicle position acquisition unit 31 may recognize the lateral position of the remotely driven autonomous vehicle 2 relative to the lane (the position of the remotely driven autonomous vehicle 2 in the lane width direction) from the positional relationship between the lane markings and the remotely driven vehicle 2, and include this in the position information. The vehicle position acquisition unit 31 may also acquire the position information of the remotely driven autonomous vehicle 2 using other well-known methods.

[0031] The external environment recognition unit 32 recognizes the external environment of the remotely driven autonomous vehicle 2 based on the detection results of the external sensor 22. The external environment includes the relative positions of surrounding objects with respect to the remotely driven autonomous vehicle 2. The external environment may also include the relative speed and movement direction of surrounding objects with respect to the remotely driven autonomous vehicle 2. The external environment may also include types of objects, such as other vehicles, pedestrians, and bicycles. The types of objects can be identified using well-known techniques such as pattern matching. The external environment may also include the results of lane marking recognition (white line recognition) around the remotely driven autonomous vehicle 2. The external environment may also include the recognition results of the lighting status of traffic lights. For example, the external environment recognition unit 32 can recognize the lighting status of traffic lights ahead of the remotely driven autonomous vehicle 2 (whether the lighting is on to allow passage or to prohibit passage, etc.) based on images from the camera of the external sensor 22.

[0032] The driving state recognition unit 33 recognizes the driving state of the remotely automatically driven vehicle 2 based on the detection results of the internal sensor 23. The driving state includes the speed, acceleration, and yaw rate of the remotely automatically driven vehicle 2 of the remotely automatically driven vehicle 2. Specifically, the driving state recognition unit 33 recognizes the speed of the remotely automatically driven vehicle 2 based on vehicle speed information from a vehicle speed sensor. The driving state recognition unit 33 recognizes the acceleration of the remotely automatically driven vehicle 2 based on vehicle speed information from an acceleration sensor. The driving state recognition unit 33 recognizes the orientation of the remotely automatically driven vehicle 2 based on yaw rate information from a yaw rate sensor.

[0033] The remote instruction determination unit 34 determines whether the remotely controlled autonomous vehicle 2 should request a remote instruction from the remote commander R (remote instruction device 1). The remote instruction determination unit 34 determines whether a remote instruction should be requested based on at least one of the position information of the remotely controlled autonomous vehicle 2 acquired by the vehicle position acquisition unit 31 and the map information in the map database 24, the external environment recognized by the external environment recognition unit 32, and the route generated by the route generation unit 37, which will be described later.

[0034] The remote instruction determination unit 34 determines that a remote instruction should be requested, for example, when the remotely controlled autonomous vehicle 2 is in a remote instruction target situation. The remote instruction target situation is a situation that is preset as a situation in which the autonomous vehicle should request a remote instruction from the remote instruction device 1.

[0035] The remote instruction target situation may include, for example, at least one of the following situations: a situation in which the remotely controlled autonomous vehicle 2 turns right or left at an intersection, a situation in which the remotely controlled autonomous vehicle 2 enters an intersection with or without traffic lights, a situation in which the remotely controlled autonomous vehicle 2 approaches a merging point, a situation in which the remotely controlled autonomous vehicle 2 enters a roundabout, a situation in which the remotely controlled autonomous vehicle 2 passes through a crosswalk, a situation in which there is a stopped vehicle or obstacle ahead, a situation in which the remotely controlled autonomous vehicle 2 changes lanes to avoid a construction zone, a situation in which a decision to offset and avoid an obstacle ahead is required, a situation in which a stopped autonomous vehicle 2 starts moving, and a situation in which the remotely controlled autonomous vehicle 2 stops at a boarding point or a destination. Note that if the vehicle is in a country or region where traffic drives on the right side of the road, the situation in which the remotely controlled autonomous vehicle 2 turns right at an intersection can be replaced with a situation in which the remotely controlled autonomous vehicle 2 turns left at an intersection.

[0036] The remote instruction determination unit 34 determines that a remote instruction should be requested, for example, when the remotely controlled autonomous vehicle 2 is about to enter an intersection or turn right at an intersection. The remote instruction determination unit 34 may also determine that a remote instruction should be requested when there is an obstacle ahead of the remotely controlled autonomous vehicle 2 that must be avoided by offsetting.

[0037] The remote instruction determination unit 34 can recognize, for example, from the position information, map information, and target route of the remotely self-driving vehicle 2, that the remotely self-driving vehicle 2 is in a situation where it is about to turn right at an intersection, that the remotely self-driving vehicle 2 is about to enter an intersection with traffic lights, or that the remotely self-driving vehicle 2 is about to start changing lanes.

[0038] If the remote instruction determination unit 34 determines that a remote instruction should be requested, it requests a remote instruction from the remote commander R from the remote instruction server 10. The request for a remote instruction includes, for example, identification information for the remotely driven autonomous vehicle 2. The remote instruction determination unit 34 may request a remote instruction well in advance. The remote instruction determination unit 34 may also determine that a remote instruction should be requested when the distance between the remotely driven vehicle 2 and an intersection or the like that is the target of the remote instruction becomes equal to or less than a certain distance. The remaining time until arrival may be used instead of the distance.

[0039] The display information acquisition unit 35 acquires display information based on the calculation results of the autonomous driving ECU 20. The display information is information used by the remote instruction device 1 to display an image (route recognition auxiliary image) that allows the remote commander R to recognize the route of the remotely controlled autonomously driven vehicle 2.

[0040] The display information includes information on the position, destination, and driving status (e.g., vehicle speed) of the remotely autonomously driven vehicle 2 when switching from autonomous driving to remote driving. The position, destination, and driving status of the remotely autonomously driven vehicle 2 as display information may be information outside the operation design domain (ODD) of the autonomous driving system. The ODD is the range within which the autonomous driving system operates as designed.

[0041] The display information includes the surrounding conditions of the remotely automatically driven vehicle 2 when switching from automatic driving to remote driving. The surrounding conditions of the remotely automatically driven vehicle 2 include the external environment of the remotely automatically driven vehicle 2 detected by the external sensor 22 of the remotely automatically driven vehicle 2. The surrounding conditions of the remotely automatically driven vehicle 2 include an image of the front of the remotely automatically driven vehicle 2 captured by the external sensor 22 (camera) of the remotely automatically driven vehicle 2. The front image is an image of the front of the remotely automatically driven vehicle 2 captured by the camera of the remotely automatically driven vehicle 2. The front image is, for example, an image of the scenery ahead, an overhead image of the remotely automatically driven vehicle 2, etc.

[0042] The display information may include information on a vehicle-directed route. The vehicle-directed route is a route generated by the route generation unit 37 (described later) and is a route when switching from autonomous driving to remote driving.

[0043] The display information acquisition unit 35 may acquire information on the load on communication between the remotely controlled autonomous vehicle 2 and the remote instruction device 1 as display information using a known method, for example, based on the data capacity of the display information.

[0044] The display information acquisition unit 35 may acquire, for example, the reliability of the autonomous driving information of the remotely driven autonomous vehicle 2 as the display information. The reliability of the autonomous driving information refers to the reliability of the information transmitted from the remotely driven autonomous vehicle 2 that the remote instruction device 1 uses to provide the remote commander R with the route of the remotely driven autonomous vehicle 2. Examples of situations in which the reliability of the autonomous driving information decreases include a situation in which the route generated by the path generation unit 37 is restricted (e.g., a restriction on the length of the route) by other vehicles present around the remotely driven autonomous vehicle 2, a situation in which the traffic environment around the remotely driven vehicle 2 is congested and the route of the remotely driven vehicle 2 is changed in a short period of time, a situation in which the road environment around the remotely driven vehicle 2 is complex and the accuracy of route generation decreases, a situation in which the position information (e.g., localization) of the vehicle position acquisition unit 31 becomes unstable, a situation in which some of the external sensors 22 fail, and a situation in which the remotely driven autonomous vehicle 2 is traveling within the operating range of the autonomous driving system.

[0045] The display information transmission unit 36 ​​transmits the display information acquired by the display information acquisition unit 35 to the remote instruction device 1. For example, when the remote instruction determination unit 34 determines that a remote instruction should be requested, the display information transmission unit 36 ​​transmits the position, destination, driving state (vehicle speed), and surrounding conditions (external environment, forward image) of the remotely controlled autonomously driven vehicle 2 to the remote instruction device 1 as display information. The display information transmission unit 36 ​​may also transmit information on the communication load and the reliability of the autonomous driving information to the remote instruction device 1 as display information.

[0046] The display information transmitter 36 may transmit the vehicle-side route information to the remote instruction device 1 on the condition that the load on communication between the remotely controlled autonomous vehicle 2 and the remote instruction device 1 is less than a predetermined load, based on the data volume of the vehicle-side route information generated by the path generator 37, etc. In this case, by omitting the transmission of the vehicle-side route information when the communication load is greater than the predetermined load, an increase in the communication load can be suppressed, and a decrease in the responsiveness of the remote instruction device 1 in presenting information to the remote commander R can be suppressed.

[0047] The trajectory generation unit 37 generates a trajectory to be used for autonomous driving of the remotely autonomously driven vehicle 2. The trajectory generation unit 37 generates a trajectory for autonomous driving based on a preset target route, map information, position information of the remotely autonomously driven vehicle 2, the external environment of the remotely autonomously driven vehicle 2, and the driving status of the remotely autonomously driven vehicle 2. The trajectory corresponds to a driving plan for autonomous driving.

[0048] The course includes the path along which the vehicle will travel under autonomous driving and the vehicle speed plan for autonomous driving. The route is the planned trajectory along which the vehicle will travel under autonomous driving on the target route. The route can be, for example, data on the steering angle change (steering angle plan) of the remotely controlled autonomous vehicle 2 according to the position on the target route. The position on the target route is, for example, a set longitudinal position set at predetermined intervals (for example, 1 m) in the direction of travel of the target route. The steering angle profile is data in which a target steering angle is associated with each set longitudinal position.

[0049] The target route is set based on, for example, the destination, map information, and position information of the remotely driven autonomous vehicle 2. The target route may also be set taking into consideration traffic information such as congestion. The target route may be set by a well-known navigation system. The destination may be set by a passenger of the remotely driven autonomous vehicle 2, or may be automatically suggested by the autonomous driving ECU 20 or the navigation system.

[0050] The path generation unit 37 generates a path along which the remotely controlled autonomous vehicle 2 will travel, based on, for example, the target route, map information, the external environment of the remotely controlled autonomous vehicle 2, and the driving state of the remotely controlled autonomous vehicle 2. For example, the path generation unit 37 generates a path so that the remotely controlled autonomous vehicle 2 will pass through the center of the lane (center in the lane width direction) included in the target route.

[0051] The vehicle speed plan is data in which a target vehicle speed is associated with each set longitudinal position, for example. The set longitudinal position may be set based on the travel time of the remotely controlled autonomous vehicle 2 rather than the distance. The set longitudinal position may be set as the position the vehicle will reach in one second or two seconds, for example. In this case, the vehicle speed plan can also be expressed as data according to the travel time.

[0052] The route generation unit 37 generates a vehicle speed plan based on traffic regulation information such as the legal maximum speed included in the route and map information. A speed preset for a position or section on the map may be used instead of the legal maximum speed. The route generation unit 37 generates a route for autonomous driving from the route and the vehicle speed plan. Note that the route generation method in the route generation unit 37 is not limited to the above-described method, and any well-known method related to autonomous driving may be adopted. The same applies to the content of the route.

[0053] The automatic driving control unit 38 performs automatic driving of the remotely automatically driven vehicle 2. The automatic driving control unit 38 performs automatic driving of the remotely automatically driven vehicle 2 based on, for example, the external environment of the remotely automatically driven vehicle 2, the driving state of the remotely automatically driven vehicle 2, and the route generated by the route generation unit 37. The automatic driving control unit 38 performs automatic driving of the remotely automatically driven vehicle 2 by sending control signals to the actuator 26.

[0054] When the remote instruction determination unit 34 requests a remote instruction from the remote instruction server 10, the autonomous driving control unit 38 waits to receive the remote instruction from the remote instruction server 10. When the autonomous driving control unit 38 requests a remote instruction after the remotely driven autonomous vehicle 2 has stopped, the autonomous driving control unit 38 maintains the stopped state until it receives the remote instruction.

[0055] [Configuration of remote control device] The following describes the configuration of the remote instruction device 1 with reference to the drawings. As shown in FIG.

[0056] First, the hardware configuration of the remote instruction server 10 will be described. Fig. 3 is a block diagram showing an example of the hardware configuration of the remote instruction server 10. As shown in Fig. 3, the remote instruction server 10 is configured as a general computer including a processor 10a, a storage unit 10b, a communication unit 10c, and a user interface 10d. In this case, the user refers to a user (administrator, etc.) of the remote instruction server 10.

[0057] The processor 10a runs various operating systems to control the remote instruction server 10. The processor 10a is a computing unit such as a CPU including a control device, an arithmetic unit, a register, etc. The processor 10a controls the storage unit 10b, the communication unit 10c, and the user interface 10d. The storage unit 10b is configured to include at least one of a memory and a storage. The memory is a recording medium such as a ROM or a RAM. The storage is a recording medium such as a HDD.

[0058] The communication unit 10c is a communication device for communicating via the network N. A network device, a network controller, a network card, etc. can be used as the communication unit 10c. The user interface 10d is an input / output unit of the remote instruction server 10 for users such as administrators. The user interface 10d includes output devices such as a display and a speaker, and an input device such as a touch panel. The remote instruction server 10 does not necessarily have to be installed in a facility, but may be mounted on a mobile object such as a vehicle.

[0059] Fig. 4 is a block diagram showing an example of the configuration of the remote instruction device 1. As shown in Fig. 4, the commander interface 3 is an input / output unit of the remote instruction device 1 for the remote commander R. The commander interface 3 has an output unit 3a and an instruction input unit 3b.

[0060] The output unit 3a is a device that outputs information used to remotely instruct the remotely controlled autonomously driven vehicle 2 to the remote commander R. The output unit 3a includes a display that outputs images and a speaker that outputs sounds.

[0061] As an example, the display displays an image of the front of the remotely driven autonomous vehicle 2 (an image of the scenery ahead) captured by a camera on the remotely driven autonomous vehicle 2. As will be described later, the display can also display a superimposed image in which a route recognition assistance image is superimposed on an image of the front of the remotely driven autonomous vehicle 2 captured by a camera on the remotely driven vehicle 2. The display may have multiple display screens, and may display images of the sides and / or rear of the remotely driven autonomous vehicle 2. The display is not particularly limited as long as it has a configuration that can provide visual information to the remote commander R. The display may also be a wearable device that is worn over the eyes of the remote commander R.

[0062] The speaker is, for example, a headset speaker worn on the head of the remote commander R. The speaker, for example, communicates the situation of the remotely controlled autonomously driven vehicle 2 (such as the situation of turning right at an intersection) by voice to the remote commander R. The speaker does not necessarily have to be a headset, but may be a stationary speaker.

[0063] The output unit 3a may provide information to the remote commander R by vibration. The output unit 3a may have, for example, a vibration actuator provided on the seat of the remote commander R. The output unit 3a may alert the remote commander R to the approach of another vehicle to the remotely driven autonomous vehicle 2 by vibration. The output unit 3a may have vibration actuators on each of the left and right sides of the seat, and may vibrate the vibration actuators at positions depending on the approaching direction of the other vehicle. The output unit 3a may also have a wearable vibration actuator attached to the body of the remote commander R. The output unit 3a can provide information to the remote commander R by vibrating the vibration actuators attached to various positions on the body depending on the approaching direction of the other vehicle.

[0064] The instruction input unit 3b is a device to which remote instructions are input by the remote commander R. The instruction input unit 3b has, for example, an operation lever. In the instruction input unit 3b, for example, a remote instruction to move the remotely operated autonomous vehicle 2 forward is input by tilting the operation lever toward the back in the fore-and-aft direction of the remote commander R, and a remote instruction to slow down or stop the remotely operated autonomous vehicle 2 is input by tilting the operation lever toward the front-and-aft direction of the remote commander R.

[0065] The instruction input unit 3b may have a button, or the remote commander R may input a remote instruction by pushing a button and tilting an operation lever. The instruction input unit 3b may have a touch panel. The touch panel may be common to the display of the output unit 3a. The instruction input unit 3b may have an operation pedal.

[0066] The instruction input unit 3b may have a voice recognition function or a gesture recognition function. Gestures of the remote commander R can be recognized by a camera and / or a radar sensor mounted on the commander interface 3. The instruction input unit 3b may be configured to allow remote instructions to be input by combining two or more of operation of a control lever, operation of a button, operation of a touch panel, operation of a control pedal, voice input, and gestures.

[0067] Next, we will explain the functional configuration of the remote instruction server 10. As shown in Fig. 4, the remote instruction server 10 has a remote instruction request receiving unit 11, a route acquisition unit 12, a superimposed image generating unit 13, an information providing unit 14, and a remote instruction transmitting unit 15.

[0068] When the remotely controlled automatically driven vehicle 2 requests the remote instruction server 10 for a remote instruction, the remote instruction request receiving unit 11 receives the request for the remote instruction.

[0069] In response to a transmission from the remotely controlled autonomous vehicle 2, the route acquisition unit 12 acquires display information for the remotely controlled autonomous vehicle 2 that requested remote instructions. The route acquisition unit 12 acquires the route of the remotely controlled autonomous vehicle 2 that requested remote instructions based on the acquired display information. For example, the route acquisition unit 12 generates a route for the remotely controlled autonomous vehicle 2 from the position of the remotely controlled autonomous vehicle 2 that requested remote instructions to the destination based on the display information for the remotely controlled autonomous vehicle 2 received by the remote instruction request receiving unit 11. Generating a route by the route acquisition unit 12 in this manner can reduce communication traffic compared to when the remote instruction request receiving unit 11 receives a route generated by the route generation unit 37. Furthermore, because it is possible to generate a route in a manner different from the route generated by the route generation unit 37, a greater degree of freedom can be expected for the content of the superimposed image described below.

[0070] The route acquisition unit 12 may compare the route generated by the route acquisition unit 12 with the route generated by the route generation unit 37 of the remotely driven autonomous vehicle 2 that requested the remote instruction. After the route acquisition unit 12 generates a route, if there is no change in the route of the remotely driven autonomous vehicle 2 based on a change in the position or destination of the remotely driven autonomous vehicle 2 that requested the remote instruction, the route acquisition unit 12 may omit re-generation of the route (re-acquisition of the route) by the route acquisition unit 12.

[0071] Alternatively, for example, when the load on communication between the remotely-controlled autonomous vehicle 2 and the remote instruction device 1 is less than a predetermined load and information on the vehicle-side route is transmitted by the display information transmission unit 36, instead of generating a route in the route acquisition unit 12 based on the position and destination of the remotely-controlled autonomous vehicle 2, the route acquisition unit 12 may use (repurpose) the vehicle-side route to obtain a superimposed route from the vehicle-side route that corresponds to the route of the remotely-controlled autonomous vehicle 2 when switching from autonomous driving to remote driving.

[0072] The superimposed image generation unit 13 generates a superimposed image based on the surrounding conditions of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving. The superimposed image is an image that allows the remote commander R to recognize the route of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving. The superimposed image may include a route recognition auxiliary image superimposed on the forward image. The route recognition auxiliary image is an image used to supplement the forward image in order to allow the remote commander R to recognize the route of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving. The superimposed image generation unit 13 generates the route recognition auxiliary image and the superimposed image using, for example, the route acquired by the route acquisition unit 12.

[0073] The superimposed image generation unit 13 generates an image representing the superimposed route as a route recognition auxiliary image based on a superimposed route that corresponds to the route of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving, for example, from among the routes acquired by the route acquisition unit 12. The superimposed route is the route of the remotely driven autonomous vehicle 2 to be superimposed on the forward image when switching from autonomous driving to remote driving.

[0074] Next, specific examples of superimposed images will be described. FIG. 5 is an example of a superimposed image illustrating a route recognition auxiliary image. FIG. 5(a) shows an example of a route recognition auxiliary image showing a superimposed route when the remotely controlled autonomous vehicle 2 changes lanes from the lane in which it is traveling to the lane adjacent to the right. FIG. 5(b) shows an example of a route recognition auxiliary image showing a superimposed route when the remotely controlled autonomous vehicle 2 continues along the lane without changing lanes. FIG. 5(c) shows another example of a route recognition auxiliary image showing a superimposed route when the remotely controlled autonomous vehicle 2 changes lanes from the lane in which it is traveling to the lane adjacent to the right. FIG. 5(d) shows another example of a route recognition auxiliary image showing a superimposed route when the remotely controlled autonomous vehicle 2 continues along the lane without changing lanes.

[0075] As shown in Figures 5(a) and 5(b), the route recognition auxiliary image showing the superimposed route may be an image of lines 50, 51 with arrows superimposed directly on the lanes in the forward image 40 and extending along the route of the remotely driven autonomous vehicle 2. As shown in Figures 5(c) and 5(d), the route recognition auxiliary image showing the superimposed route may be an image of shaded regions 52, 53 superimposed directly on the lanes in the forward image 40 and extending along the route of the remotely driven autonomous vehicle 2.

[0076] The superimposed image generation unit 13 may change the type of route recognition auxiliary image based on the surrounding conditions of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving, if other vehicles around the remotely driven autonomous vehicle 2 make it difficult to see markings on the road surface in the forward image.

[0077] 6A and 6B are examples of superimposed images illustrating route recognition auxiliary images. FIG. 6A shows a situation in which other vehicles 200 around the remotely driven autonomous vehicle 2 make it difficult to see markings on the road surface in the forward image 41. In the example of FIG. 6A, the superimposed image generation unit 13 generates a superimposed image in which a route recognition auxiliary image, which displays a mark 54a emphasizing the direction of travel specified for the lane in which the remotely driven autonomous vehicle 2 is traveling, is superimposed on the upper left part of the forward image 41, along with a schematic diagram 54 showing multiple entrance lanes to the intersection where the remotely driven autonomous vehicle 2 is entering. In this case, the emphasized mark 54a corresponds to the superimposed route.

[0078] If there are multiple possible routes that the remotely driven autonomous vehicle 2 can take based on the surrounding conditions of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving, the superimposed image generation unit 13 may superimpose a route recognition auxiliary image showing the planned route that the remotely driven autonomous vehicle 2 will take onto the forward image.

[0079] Fig. 6(b) shows a situation in which there are multiple routes R1 to R3 that the remotely controlled autonomous vehicle 2 can take. In the example of Fig. 6(b), the superimposed image generation unit 13 generates a superimposed image in which a dashed arrow 55, which is a route recognition auxiliary image indicating the superimposed route, is superimposed directly on the lane in the forward image 42. The dashed arrow 55 indicates the superimposed route corresponding to route R3, which includes a lane change from the lane in which the remotely controlled autonomous vehicle 2 is traveling to the lane adjacent to the left, and a left turn from the lane adjacent to the left.

[0080] The superimposed image generation unit 13 may determine whether or not to superimpose a route recognition auxiliary image on the forward image based on the surrounding conditions of the remotely controlled autonomously driven vehicle 2 when switching from autonomous driving to remote driving. Superimposed display means superimposing a certain type of route recognition auxiliary image on the forward image.

[0081] For example, assuming that a certain type of route recognition auxiliary image is superimposed and displayed on the forward image when switching from autonomous driving to remote driving, the superimposed image generation unit 13 may determine that superimposed display is necessary if it is determined that the route recognition auxiliary image does not interfere with a traffic information object in the forward image. A traffic information object is an object in the forward image that the remote commander R needs to recognize as traffic information when having the remote commander R recognize the route of the remotely controlled autonomously driven vehicle 2. Examples of traffic information objects include traffic lights, crosswalks, road markings, road signs, and railroad crossings (gates). Note that, assuming the above assumption, if it is determined that the route recognition auxiliary image interferes with a traffic information object in the forward image, the superimposed image generation unit 13 may determine that superimposed display is unnecessary because it would hinder the remote commander R from recognizing the traffic information object.

[0082] Fig. 7 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposed display. Fig. 7 shows a scene in which a traffic light SG and a crosswalk CR (traffic information object) are present ahead of the remotely driven autonomous vehicle 2, and the remotely driven autonomous vehicle 2 is attempting to proceed through the crosswalk CR. A thick arrow 56 is shown as a route recognition auxiliary image indicating the superimposed route. The thick arrow 56 is one of several types of route recognition auxiliary images that indicate that the remotely driven autonomous vehicle 2 will proceed in the same lane.

[0083] FIG. 7 corresponds to a superimposed image when it is assumed that the superimposed image generation unit 13 has superimposed and displayed the thick arrow 56 on the forward image 43 when switching from autonomous driving to remote driving. Under this assumption, the superimposed image generation unit 13 determines that the thick arrow 56 interferes with the traffic light SG and the crosswalk CR in the forward image 43 in FIG. 7 based on the surrounding conditions of the remotely controlled autonomously driven vehicle 2 that have been coordinate-transformed onto the forward image 43. Therefore, the superimposed image generation unit 13 determines that the superimposed display of the thick arrow 56 in FIG. 7 is unnecessary. As a result, for example, the display of the output unit 3a displays the forward image 43 from FIG. 7 without the thick arrow 56, thereby preventing the thick arrow 56 from interfering with the remote commander R's recognition of the traffic light SG and the crosswalk CR.

[0084] Figure 8 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposition. In Figure 8, the remotely driven autonomous vehicle 2 is traveling on a straight road with one lane in each direction, and a two-dot chain arrow 57 is shown as a route recognition auxiliary image indicating the superimposed route. The two-dot chain arrow 57 is one of several types of route recognition auxiliary images that indicate that the remotely driven autonomous vehicle 2 will continue in the same lane without changing lanes.

[0085] In Figure 8, the remotely driven autonomous vehicle 2 is simply traveling on a straight road with one lane in each direction, so the route of the remotely driven autonomous vehicle 2 is self-evident and the route recognition assistance image can be omitted. Therefore, based on the position information and map information of the remotely driven autonomous vehicle 2, the superimposed image generation unit 13 determines that it is not necessary to superimpose the two-dot chain arrow 57 in Figure 8. As a result, although the two-dot chain arrow 57 is shown in Figure 8, the display of the output unit 3a displays the forward view image 44 from Figure 8 without the two-dot chain arrow 57.

[0086] 9 is a hypothetical example of a superimposed image illustrating a route recognition auxiliary image that does not require superimposition. In Fig. 9, the remotely driven autonomous vehicle 2 is traveling on a straight road with one lane in each direction, with a merging lane merging from the left ahead of the remotely driven autonomous vehicle 2, and a two-dot chain arrow 58 is shown as a route recognition auxiliary image of the same type as in Fig. 8.

[0087] In Figure 9, as in Figure 8, the remotely driven autonomous vehicle 2 is traveling on a straight road with one lane in each direction, and since the merging lane on the left is generally one-way, the route of the remotely driven autonomous vehicle 2 is self-evident and the route recognition assistance image can be omitted. Therefore, based on the position information and map information of the remotely driven autonomous vehicle 2, the superimposed image generation unit 13 determines that the superimposed display of the two-dot chain arrow 58 in Figure 9 is unnecessary. As a result, although the two-dot chain arrow 58 is shown in Figure 9, the display of the output unit 3a displays a forward image 45 from Figure 9 without the two-dot chain arrow 58.

[0088] In addition to the straight roads shown in Figures 8 and 9, if one of two lanes is under construction and cannot be used, and the remotely driven autonomous vehicle 2 drives on the lane that is not under construction, the route of the remotely driven autonomous vehicle 2 can be said to be obvious. Conversely, if there are multiple possible routes for the remotely driven autonomous vehicle 2, such as intersections and forks, the route of the remotely driven autonomous vehicle 2 can be said to be non-obvious.

[0089] Figure 10 is a hypothetical example of a superimposed image illustrating a route recognition assistance image that does not require superimposed display. In Figure 10, a remotely driven autonomous vehicle 2 is traveling on a two-lane road, and ahead of it is an intersection where the left-turn lane ends and only the straight-ahead lane continues. Even though the remotely driven autonomous vehicle 2 is traveling in the straight-ahead lane, a two-dot chain arrow 59 extends onto the left-turn lane on the forward image 46 as a route recognition assistance image of the same type as in Figure 8.

[0090] In FIG. 10 , the dash-dot-dotted arrow 59 should actually extend along the straight lane. However, for example, due to instability in the localization of the remotely controlled autonomous vehicle 2, the accuracy of the position information of the remotely controlled autonomous vehicle 2, which is the origin of the dash-dot-dotted arrow 59, may be reduced. In such cases, displaying the dash-dot-dotted arrow 59 may actually hinder the remote commander R from recognizing the route of the remotely controlled autonomous vehicle 2. Therefore, one solution is to omit the route recognition assistance image. Therefore, the superimposed image generation unit 13 determines that the superimposed display of the dash-dot-dotted arrow 59 in FIG. 10 is unnecessary based on the display information (the reliability of the autonomous driving information of the remotely controlled autonomous vehicle 2). As a result, although the dash-dot-dotted arrow 59 is shown on the forward image 46 in FIG. 10 , the display of the output unit 3 a may display the forward image 46 from FIG. 10 without the dash-dot-dotted arrow 59.

[0091] Incidentally, it is also possible to use a route recognition auxiliary image showing the superimposed route as an image that simply shows the route as information, rather than visually conveying information using an image that requires a starting point, such as an arrow. In this case, instead of omitting the route recognition auxiliary image as shown in Fig. 10, the superimposed image generating unit 13 may superimpose different types of route recognition auxiliary images on the forward image 46 and display them on the display of the output unit 3a.

[0092] For example, FIG. 11 is an example of a superimposed image illustrating a route recognition auxiliary image that requires superimposition. In FIG. 11, in the situation of FIG. 10, an image 60 with the character string "Go straight" surrounded by a frame is superimposed on top of the forward image 46 as a route recognition auxiliary image of a different type from that of FIG. 10. In this case, it can be said that the remote commander R can correctly recognize the route of the remotely driven autonomous vehicle 2. Therefore, the superimposed image generation unit 13 may determine, based on the display information (the reliability of the autonomous driving information of the remotely driven autonomous vehicle 2), that a type of route recognition auxiliary image that does not require a starting point, such as image 60 with the character string "Go straight" surrounded by a frame, is required to be superimposed. As a result, although a two-dot chain arrow 59 is shown in FIG. 11, on the display of the output unit 3a, the two-dot chain arrow 59 is omitted from FIG. 11, and image 60 with the character string "Go straight" surrounded by a frame is superimposed on top of the forward image 46.

[0093] 12 is an example of a superimposed image illustrating a route recognition auxiliary image that requires superimposition. In FIG. 12, in the situations of FIGS. 10 and 11, a large arrow image 61 indicating "go straight" is superimposed on top of the forward image 46 as a different type of route recognition auxiliary image from that in FIG. 11. Because the large arrow image 61 simply indicates the direction of the route, i.e., "go straight," it does not require a starting point and is superimposed on top of the forward image 46 regardless of the position information of the remotely controlled autonomous vehicle 2. In this case, it can be said that the remote commander R can correctly recognize the route of the remotely controlled autonomous vehicle 2. Therefore, the superimposed image generation unit 13 may determine, based on the display information (the reliability of the autonomous driving information of the remotely controlled autonomous vehicle 2), that a type of route recognition auxiliary image that does not require a starting point, such as the large arrow image 61, is required to be superimposed. As a result, although a two-dot dashed arrow 59 is shown in Figure 12, on the display of the output unit 3a, the two-dot dashed arrow 59 is omitted from Figure 11, and an image 61 of a large arrow is superimposed on top of the forward image 46.

[0094] The information providing unit 14 provides various types of information to the remote commander R. When the remote instruction request receiving unit 11 receives a remote instruction request, the information providing unit 14 requests the remote commander R in charge to input the remote instruction via the commander interface 3.

[0095] Furthermore, the information providing unit 14 provides information about the remotely controlled autonomous vehicle 2 to the remote commander R based on the display information about the remotely controlled autonomous vehicle 2 acquired by the remote instruction request receiving unit 11. The information providing unit 14 displays an image of the front of the remotely controlled autonomous vehicle 2 on the display of the output unit 3a of the commander interface 3, for example.

[0096] For example, when the superimposed image generation unit 13 determines that a superimposed display is required, the information providing unit 14 displays a superimposed image on the display of the output unit 3a, in which a route recognition assistance image is superimposed on an image in front of the remotely self-driving vehicle 2 captured by a camera of the remotely self-driving vehicle 2.

[0097] The information providing unit 14 may display an image seen from near the driver's seat of the remotely controlled autonomous vehicle 2 by viewpoint conversion. The information providing unit 14 may display images of the sides and rear of the remotely controlled autonomous vehicle 2. The information providing unit 14 may display a panoramic image created by combining images captured around the remotely controlled autonomous vehicle 2, or may display an overhead image created by image composition and viewpoint conversion to look down on the remotely controlled autonomous vehicle 2. The information providing unit 14 may highlight objects in the image (for example, by surrounding other vehicles with a frame). If a traffic light is included in the image, the information providing unit 14 may display the recognition result of the traffic light status on the display. The information providing unit 14 may display the types of remote instructions (proceed, wait, etc.) that the remote commander R can select on the display.

[0098] For example, when the superimposed image generation unit 13 determines that superimposed display is not necessary, the information providing unit 14 displays a forward image of the remotely self-driving vehicle 2 captured by the camera of the remotely self-driving vehicle 2 (a forward image without a route recognition auxiliary image superimposed) on the display of the output unit 3a.

[0099] The information providing unit 14 provides sound information to the remote commander R through the speaker of the output unit 3a of the commander interface 3. The information providing unit 14 may output the situation of the remotely automatically driven vehicle 2 (when turning right at an intersection, when offsetting to avoid an obstacle, etc.) as sound from the speaker. The information providing unit 14 may also output the approach of other vehicles around the remotely automatically driven vehicle 2 as sound or voice from the speaker. The information providing unit 14 may also output the sound (noise) around the remotely automatically driven vehicle 2 directly from the speaker. The information providing unit 14 may also output the voices of occupants in the vehicle cabin from the speaker as necessary. Note that providing information through a speaker is not essential.

[0100] Additionally, if the output unit 3a has a vibration actuator, the information providing unit 14 may provide information to the remote commander R by vibration. In this case, the information providing unit 14 can provide information (warning) to the remote commander R by vibrating a vibration actuator at a position corresponding to a direction that requires attention, such as the direction in which another vehicle is approaching the remotely driven autonomous vehicle 2 or the direction in which a pedestrian is present.

[0101] When the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction transmission unit 15 transmits the input remote instruction to the remotely driven vehicle 2. When the information provision unit 14 transmits the remote instruction input by the remote commander R to the remotely driven vehicle 2, it may continue to convey information about the remotely driven vehicle 2 to the remote commander R, or may switch to information about another remotely driven vehicle 2 that requests a remote instruction.

[0102] [Autonomous driving ECU and remote control device processing] Next, an example of processing by the autonomous driving ECU 20 and the remote instruction device 1 will be described with reference to the flowcharts of Fig. 13 and Fig. 14. Fig. 13 is a flowchart showing an example of processing by the autonomous driving ECU 20. The processing shown in Fig. 13 is executed, for example, while the remotely autonomously driven vehicle 2 is autonomously driving.

[0103] 13, in S01, the autonomous driving ECU 20 determines, via the remote instruction determination unit 34, whether or not a remote instruction should be requested. The remote instruction determination unit 34 determines that a remote instruction should be requested, for example, when the remotely controlled autonomous vehicle 2 enters a remote instruction target situation. If the autonomous driving ECU 20 determines that a remote instruction should be requested (S01: YES), the process proceeds to S02.

[0104] In S02, the autonomous driving ECU 20 acquires vehicle-side display information using the display information acquisition unit 35. The display information acquisition unit 35 acquires, as display information for the remotely autonomously driven vehicle 2, information on the position, destination, and driving state (e.g., vehicle speed) of the remotely autonomously driven vehicle 2 when switching from autonomous driving to remote driving, the surrounding conditions of the remotely autonomously driven vehicle 2, and information on the reliability of the autonomous driving information of the remotely autonomously driven vehicle 2. The display information acquisition unit 35 may also acquire information on the vehicle-side route of the remotely autonomously driven vehicle 2 as display information for the remotely autonomously driven vehicle 2.

[0105] In S03, the autonomous driving ECU 20 transmits vehicle-side display information via the display information transmission unit 36. The display information transmission unit 36 ​​transmits, for example, the display information acquired in S02 to the remote instruction device 1 as display information for the remotely driven autonomous vehicle 2. The display information transmission unit 36 ​​transmits a remote instruction request together with the display information to the remote instruction device 1. Note that if the load on communication between the remotely driven autonomous vehicle 2 and the remote instruction device 1 is less than a predetermined load, the display information transmission unit 36 ​​may transmit vehicle-side route information to the remote instruction device 1 as display information for the remotely driven autonomous vehicle 2. Thereafter, the autonomous driving ECU 20 ends the processing of FIG. 13.

[0106] On the other hand, if the autonomous driving ECU 20 determines that a remote instruction should not be requested (S01: NO), it ends the processing of FIG.

[0107] Fig. 14 is a flowchart showing an example of processing by the remote instruction device 1. The processing shown in Fig. 14 is executed, for example, when a remote instruction request transmitted from the autonomous driving ECU 20 is received by the remote instruction request receiving unit 11 of the remote instruction device 1.

[0108] 14, in S11, the remote instruction device 1 receives vehicle-side display information via the route acquisition unit 12. The route acquisition unit 12 receives, as display information for the remotely automatically driven vehicle 2, information on the position, destination, and driving conditions (e.g., vehicle speed) of the remotely automatically driven vehicle 2 when switching from autonomous driving to remote driving, information on the surrounding conditions of the remotely automatically driven vehicle 2, and information on the reliability of the autonomous driving information of the remotely automatically driven vehicle 2. For example, if the load on communication between the remotely automatically driven vehicle 2 and the remote instruction device 1 is less than a predetermined load and vehicle-side route information is transmitted by the display information transmission unit 36, the route acquisition unit 12 may receive the vehicle-side route for the remotely automatically driven vehicle 2 as display information.

[0109] In S12, the remote instruction device 1 determines whether or not superimposed display is required using the superimposed image generation unit 13. The superimposed image generation unit 13 determines whether or not superimposed display of the route recognition auxiliary image on the forward image is required, for example, based on the surrounding conditions of the remotely controlled autonomously driven vehicle 2 when switching from autonomous driving to remote driving and information on the reliability of the autonomous driving information of the remotely controlled autonomously driven vehicle 2. If the remote instruction device 1 determines that superimposed display is required (S12: YES), it proceeds to processing of S13. On the other hand, if the remote instruction device 1 determines that superimposed display is not required (S12: NO), it proceeds to processing of S18.

[0110] In S13, the remote instruction device 1 determines whether or not it is necessary to generate a superimposed route using the route acquisition unit 12. The route acquisition unit 12 determines that it is necessary to generate a superimposed route, for example, when information on the vehicle route has not been transmitted. If the remote instruction device 1 determines that it is necessary to generate a superimposed route (S13: YES), it proceeds to processing in S14.

[0111] In S14, the remote instruction device 1 generates a superimposed route using the route acquisition unit 12. The route acquisition unit 12 generates, for example, a route for the remotely controlled autonomous vehicle 2 from the position of the remotely controlled autonomous vehicle 2 that requested remote instruction to the destination. The route acquisition unit 12 generates, for example, a superimposed route such as those shown in FIGS. 5, 6, 11, and 12 as a superimposed route corresponding to the route of the remotely controlled autonomous vehicle 2 when switching from autonomous driving to remote driving from the generated routes. The process then proceeds to S16. Note that the generation of the route for the remotely controlled autonomous vehicle 2 by the route acquisition unit 12 may be omitted if, after the route has been generated, there is no change in the route based on a change in the position or destination of the remotely controlled autonomous vehicle 2 that requested remote instruction.

[0112] On the other hand, if the remote instruction device 1 determines that generation of a superimposed route is not necessary (S13: NO), the remote instruction device 1 proceeds to processing in S15. In S15, the remote instruction device 1 causes the route acquisition unit 12 to acquire a route on the side of the remotely controlled autonomous vehicle 2. The route acquisition unit 12, for example, uses (repurposes) the vehicle-side route received in S11 to acquire a superimposed route that corresponds to the route of the remotely controlled autonomous vehicle 2 when switching from autonomous driving to remote driving, from among the vehicle-side routes. Thereafter, the process proceeds to processing in S16.

[0113] In S16, the remote instruction device 1 generates a superimposed image using the superimposed image generation unit 13. The superimposed image generation unit 13 generates a route recognition auxiliary image and generates a superimposed image including the route recognition auxiliary image, for example, as shown in Figures 5, 6, 11, and 12. In S17, the remote instruction device 1 displays the superimposed image using the information provision unit 14. For example, the information provision unit 14 displays the superimposed image generated in S16 on the display of the output unit 3a. Thereafter, the processing of Figure 14 ends.

[0114] Meanwhile, in S18, the remote instruction device 1 displays a forward image using the information providing unit 14. The information providing unit 14 displays an image of the forward view of the remotely controlled automatically driven vehicle 2 captured by a camera of the remotely controlled automatically driven vehicle 2 (an image of the forward scenery not including a route recognition auxiliary image) on the display of the output unit 3a, for example, as shown in Figures 7 to 10. Thereafter, the processing of Figure 14 ends.

[0115] As described above, in the vehicle remote instruction system 100, the remote instruction device 1 has a superimposed image generation unit 13 that generates a superimposed image based on the surrounding conditions of the remotely driven autonomous vehicle 2 when switching from autonomous driving to remote driving. The superimposed image is a forward image onto which a route recognition auxiliary image is superimposed to allow the remote commander R to recognize the route of the remotely driven autonomous vehicle 2. The information provision unit 14 displays the superimposed image on the display of the output unit 3a to provide information to the remote commander R. This makes it easier for the remote commander R to recognize the route of the remotely driven autonomous vehicle 2 according to the surrounding conditions of the remotely driven vehicle 2 by referring to the route recognition auxiliary image superimposed on the forward image when switching from autonomous driving to remote driving. Therefore, the vehicle remote instruction system 100 makes it possible to appropriately provide the remote commander R with information about the route of the remotely driven autonomous vehicle 2, which is used by the remote commander R to make remote instruction decisions when switching from autonomous driving to remote driving.

[0116] In the vehicle remote instruction system 100, the superimposed image generation unit 13 determines whether or not it is necessary to superimpose a route recognition auxiliary image on a forward image based on the surrounding conditions of the remotely controlled automatically driven vehicle 2 when switching from automatic driving to remote driving, and if the superimposed image generation unit 13 determines that superimposition is not necessary, the information provision unit 14 displays a forward image without the route recognition auxiliary image superimposed on it on the output unit to provide information to the remote commander R. This prevents, for example, interference between the route recognition auxiliary image and traffic information objects in the forward image, thereby preventing the route recognition auxiliary image from interfering with the remote commander R's recognition of traffic information objects.

[0117] In vehicle remote instruction system 100, remote instruction device 1 acquires display information, including information on the position and destination of remotely driven autonomous vehicle 2, used to display a route recognition auxiliary image, in response to transmission from remotely driven autonomous vehicle 2, and has route acquisition unit 12 that generates a route for remotely driven autonomous vehicle 2 based on the acquired display information. This makes it possible to generate a route recognition auxiliary image without using a route generated by remotely driven autonomous vehicle 2, thereby suppressing an increase in communication burden when transmitting information on the route generated by remotely driven vehicle 2 would result in a communication burden.

[0118] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments.

[0119] In the above embodiment, in Fig. 14, a determination is made as to whether or not superimposed display is required in S12, but this determination is not essential and may be omitted. For example, if the determination as to whether or not superimposed display is required is omitted, the information providing unit 14 may display, on the display of the output unit 3a, a superimposed image in which a route recognition assistance image is superimposed on an image of the road ahead of the remotely driven autonomous vehicle 2 captured by a camera of the remotely driven autonomous vehicle 2 each time autonomous driving is switched to remote driving.

[0120] In the above embodiment, in FIG. 14 , a determination is made as to whether or not to generate a superimposed route in S13, but this determination is not essential and may be omitted. For example, the determination in S13 may be omitted, and either S14 or S15 may be performed. For example, the route acquisition unit 12 may generate a route for the remotely controlled vehicle 2 from the position of the remotely controlled vehicle 2 that requested remote instructions to the destination each time autonomous driving switches from autonomous driving to remote driving. Alternatively, the route acquisition unit 12 may use (repurpose) the vehicle-side route generated by the route generation unit 37 each time autonomous driving switches from autonomous driving to remote driving to acquire a superimposed route from the vehicle-side route that corresponds to the route of the remotely controlled vehicle 2 when autonomous driving switches to remote driving.

[0121] In short, by executing at least the processes of S11, S14, S16, and S17 in Figure 14, the above-mentioned effect of the vehicle remote instruction system 100 can be achieved, that is, when switching from automatic driving to remote driving, information on the route of the remotely controlled automatic driving vehicle 2 can be appropriately provided to the remote commander R so that the remote commander R can make remote instruction decisions.

[0122] The remote instruction device 1 may be mounted on the remotely controlled automatically driven vehicle 2. In this case, the remote commander R is also mounted on the remotely controlled automatically driven vehicle 2. The remote instruction server 10 may be a cloud server configured from the ECUs of multiple remotely controlled automatically driven vehicles 2. [Explanation of symbols]

[0123] 1...remote instruction device, 2...remotely operated autonomous vehicle, 3a...output unit, 12...route acquisition unit, 13...superimposed image generation unit, 14...information provision unit, 100...vehicle remote instruction system, R...remote commander.

Claims

1. a remotely controlled autonomous vehicle capable of autonomous driving and remote driving based on remote instructions from a remote commander; a remote instruction device having an output unit that outputs a forward image of the remotely driven autonomous vehicle, and in which the remote commander uses the forward image to give remote instructions regarding the remote driving, The remote instruction device a superimposed image generation unit that generates a superimposed image, which is the forward image on which a route recognition auxiliary image is superimposed to allow the remote commander to recognize the route of the remotely driven autonomous vehicle, based on the surrounding conditions of the remotely driven autonomous vehicle when switching from the autonomous driving to the remote driving; an information providing unit that displays the superimposed image on the output unit and provides information to the remote commander; and A vehicle remote instruction system in which the superimposed image generation unit generates the superimposed image so that the type of the route recognition auxiliary image is variable depending on the surrounding conditions of the remotely driven autonomous vehicle when switching from autonomous driving to remote driving.

2. a remotely controlled autonomous vehicle capable of autonomous driving and remote driving based on remote instructions from a remote commander; a remote instruction device having an output unit that outputs a forward image of the remotely driven autonomous vehicle, and in which the remote commander uses the forward image to give remote instructions regarding the remote driving, The remote instruction device a superimposed image generation unit that generates a superimposed image, which is the forward image on which a route recognition auxiliary image is superimposed to allow the remote commander to recognize the route of the remotely driven autonomous vehicle, based on the surrounding conditions of the remotely driven autonomous vehicle when switching from the autonomous driving to the remote driving; an information providing unit that displays the superimposed image on the output unit and provides information to the remote commander; and the superimposed image generation unit determines whether or not it is necessary to superimpose the route recognition assistance image on the forward image based on a surrounding situation of the remotely driven autonomous vehicle when switching from the autonomous driving to the remote driving, When the superimposed image generation unit determines that the superimposed display is unnecessary, the information providing unit displays the forward image without the route recognition auxiliary image superimposed on the output unit to provide information to the remote commander.

3. 2. The vehicle remote instruction system according to claim 1, wherein the remote instruction device acquires display information, including information on the position and destination of the remotely driven autonomous vehicle, used to display the route recognition assistance image in response to transmission from the remotely driven autonomous vehicle, and has a route acquisition unit that acquires a route for the remotely driven autonomous vehicle based on the acquired display information.

4. 4. The vehicle remote instruction system according to claim 3, wherein the route acquisition unit acquires the route of the remotely driven autonomous vehicle as the display information based on information about the destination outside an operating range of an autonomous driving system of the remotely driven autonomous vehicle.

5. The vehicle remote instruction system according to claim 3 , wherein the route acquisition unit acquires the route of the remotely driven autonomous vehicle based on vehicle route information generated by the remotely driven autonomous vehicle as the display information.

6. 2. The vehicle remote instruction system according to claim 1, wherein the superimposed image generation unit changes the type of the route recognition assistance image when other vehicles around the remotely driven autonomous vehicle make it difficult to see markings on the road surface in the forward image, based on the surrounding conditions of the remotely driven autonomous vehicle when switching from the autonomous driving to the remote driving.

7. 2. The vehicle remote instruction system of claim 1, wherein, when there are multiple possible routes that the remotely driven autonomous vehicle can take based on the surrounding conditions of the remotely driven autonomous vehicle when switching from autonomous driving to remote driving, the superimposed image generation unit superimposes the route recognition auxiliary image indicating the planned route that the remotely driven autonomous vehicle will take onto the forward image.

8. 2. The vehicle remote instruction system according to claim 1, wherein the superimposed image generation unit generates the superimposed image by changing the type of the route recognition auxiliary image from a type that requires a starting point to a type that does not require a starting point.

Images