Remote support device, remote support method, and remote support program

Abstract

To assist in improving situational judgment ability of a staff involved in a remote operation for assisting traveling of an automatic driving vehicle.SOLUTION: A remote support device comprises: a front image transmission unit that transmits a front image including at least a front part of an automatic driving vehicle captured by the automatic driving vehicle to a remote operation device that performs remote operation on the automatic driving vehicle when remote support at a support point is requested from the automatic driving vehicle; a reception unit that receives an operation signal corresponding to the remote operation at the support point from the remote operation device; an operation signal transmission unit that transmits the received operation signal to the automatic driving vehicle; and a rear image transmission unit that transmits, to the remote control device, a rear image captured so as to include at least a rear part of the automatic drive vehicle and emphasizing the rear part after the automatic drive vehicle has passed through a switching point set so as to switch display on the remote control device based on the support point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a remote assistance device, a remote assistance method, and a remote assistance program. [Background technology]

[0002] Patent Document 1 discloses a remote assistance device that includes a task allocation unit (40) that assigns a task of remotely operating a vehicle (10) to an operator selected from a plurality of operators whose operating ability is affirmed by the latest evaluation result stored in a memory unit (35), an evaluation acquisition unit (42) that acquires an evaluation result of the operating ability of the selected operator when the selected operator performs the task assigned by the task allocation unit using a remote operation unit (41) that causes the vehicle to run by the operator's remote operation, and a memory control unit (43) that controls the storage of the latest evaluation result acquired by the evaluation acquisition unit in the memory unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-170191 A Summary of the Invention [Problem to be solved by the invention]

[0004] Remote operators who perform remote operations to support the driving of autonomous vehicles, as well as supervisors who supervise the operators, are required to have the ability to make accurate and rapid situational judgments.

[0005] The technology described in the above Patent Document 1 presents an evaluation result of the tasks performed by an operator in order to guarantee the operator's remote operation capabilities.

[0006] However, since the traffic conditions are different each time even at the same support point, the evaluation may change each time. Therefore, it is difficult to judge how to improve the remote operation. For this reason, the technology described in the above Patent Document 1 has a problem that it is difficult to support the improvement of the operator's situation judgment ability.

[0007] The present disclosure aims to provide a remote assistance device, a remote assistance method, and a remote assistance program that can help improve the situational judgment capabilities of staff involved in remote operation to assist the driving of autonomous vehicles. [Means for solving the problem]

[0008] A remote assistance device according to a first aspect of the present disclosure includes a front image transmitting unit that, when remote assistance at an assistance point is requested by an autonomous vehicle, transmits a forward image captured by the autonomous vehicle and including at least the front of the autonomous vehicle to a remote operation device that remotely operates the autonomous vehicle, a receiving unit that receives an operation signal from the remote operation device corresponding to remote operation at the assistance point, an operation signal transmitting unit that transmits the received operation signal to the autonomous vehicle, and a rear image transmitting unit that transmits a rear image captured to include at least the rear of the autonomous vehicle and highlights the rear, after the autonomous vehicle passes a switching point set to switch the display on the remote operation device based on the assistance point.

[0009] A remote support device according to a second aspect is the remote support device according to the first aspect, wherein the rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote operation.

[0010] A remote support device according to a third aspect is a remote support device according to the first or second aspect, wherein the receiving unit receives gaze position information regarding the gaze position of an operator of the remote control device in a forward image displayed on the remote control device, and the rear image transmitting unit transmits to the remote control device a rear image including the operator's gaze range identified based on the gaze position information.

[0011] A fourth aspect of the remote support device is the remote support device of the third aspect, and when there are multiple gaze ranges, the rear image transmission unit transmits to the remote control device a rear image including the gaze range with the longest gaze time among the multiple gaze ranges.

[0012] A remote support device according to a fifth aspect is a remote support device according to the third aspect, wherein, when there are multiple gaze ranges, the rear image transmission unit transmits a rear image including all of the multiple gaze ranges to the remote operation device.

[0013] A remote support device according to a sixth aspect is a remote support device according to any one of the first to fifth aspects, wherein the rear image transmission unit transmits to the remote control device a rear image that highlights an object that is likely to be overlooked by the operator of the remote control device.

[0014] A remote support device according to a seventh aspect is a remote support device according to any one of the third to fifth aspects, wherein the rear image transmission unit transmits to the remote control device a rear image that highlights an object that is likely to be overlooked by the operator, which is identified based on the gaze range.

[0015] A remote support device according to an eighth aspect is the remote support device according to the first aspect, wherein the rear image transmission unit transmits to the remote operation device a rear image in which a newly detected object in the rear image is highlighted.

[0016] A remote support device according to a ninth aspect is the remote support device according to the first aspect, wherein the rear image transmission unit transmits to the remote operation device a rear image in which speed information related to a speed change of an oncoming vehicle detected by the autonomous driving vehicle is displayed.

[0017] A remote support device according to a tenth aspect is the remote support device according to the first aspect, wherein the rear image transmission unit acquires an overhead image taken from an overhead view of an area including the switching point and transmits the image to the remote operation device.

[0018] A remote support device according to an eleventh aspect is an image in which, in the remote support device according to the first aspect, the rear image has a display area displaying the rear of the autonomously driven vehicle that is larger than the display area displaying the front of the autonomously driven vehicle.

[0019] A remote assistance method according to a twelfth aspect includes, when an autonomous vehicle requests remote assistance at an assistance point, at least one processor performs processing including: transmitting a forward image captured by the autonomous vehicle and including at least the front of the autonomous vehicle to a remote control device that remotely controls the autonomous vehicle; receiving an operation signal corresponding to remote operation at the assistance point from the remote control device; transmitting the received operation signal to the autonomous vehicle; and, after the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, transmitting a rear image captured to include at least the rear of the autonomous vehicle and emphasizing the rear to the remote control device.

[0020] The remote assistance program of the thirteenth aspect causes at least one processor to execute processing including, when remote assistance at an assistance point is requested by an autonomous vehicle, transmitting a forward image taken by the autonomous vehicle and including at least the front of the autonomous vehicle to a remote control device that remotely controls the autonomous vehicle, receiving an operation signal corresponding to remote operation at the assistance point from the remote control device, transmitting the received operation signal to the autonomous vehicle, and, after the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, transmitting a rear image taken to include at least the rear of the autonomous vehicle and emphasizing the rear to the remote control device. Effect of the Invention

[0021] According to the present disclosure, the number of staff involved in remote operation to support the driving of an autonomous vehicle is This has the effect of helping to improve situational judgment capabilities. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a configuration diagram of an autonomous driving system. [Diagram 2] FIG. 2 is a diagram for explaining a shooting range. [Diagram 3] FIG. 2 is a functional block diagram of a remote support unit. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of a remote support device. [Diagram 5] 13 is a flowchart of a remote support process. [Figure 6A] FIG. 2 is a diagram showing the state of a vehicle at an intersection before remote assistance. [Figure 6B] FIG. 13 is a diagram showing the state of a vehicle after remote assistance at an intersection. [Figure 7A] FIG. 2 is a diagram showing the state of a vehicle at an intersection before remote assistance. [Figure 7B] FIG. 13 is a diagram showing the state of a vehicle after remote assistance at an intersection. [Figure 8A] FIG. 2 is a diagram showing the state of a vehicle at a bus stop before remote assistance. [Figure 8B] FIG. 13 is a diagram showing the state of a vehicle after remote assistance at a bus stop. [Figure 9A] FIG. 13 is a diagram showing the state of a vehicle before remote assistance when an obstacle is present; [Figure 9B] FIG. 13 is a diagram showing the state of the vehicle after remote assistance in the presence of an obstacle. [Figure 10A] FIG. 13 is a diagram showing the state of a vehicle at a pedestrian crossing before remote assistance. [Figure 10B] FIG. 13 is a diagram showing the state of a vehicle after remote assistance at a pedestrian crossing. [Figure 11] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. [Figure 12] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. [Figure 13] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. [Figure 14]FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. [Figure 15] FIG. 11 is a functional block diagram of an operator terminal according to the second embodiment. [Figure 16] FIG. 13 is a diagram for explaining a rear image including an operator's gaze range. [Figure 17] FIG. 13 is a diagram for explaining an object to be overlooked. [Figure 18] FIG. 11 is a diagram for explaining a change in speed of an oncoming vehicle. [Figure 19] FIG. 1 is a diagram showing an intersection where an infrastructure camera is installed. [Figure 20] FIG. 13 is a diagram showing an example of an overhead image. [Figure 21] FIG. 13 is a diagram for explaining a shooting range according to the third embodiment. [Figure 22] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the third embodiment. [Figure 23] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the third embodiment. [Figure 24] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the fourth embodiment. [Diagram 25] FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0024] First Embodiment

[0025] As shown in Fig. 1, an autonomous driving system 100 of this embodiment includes a vehicle 10, a remote assistance device 50 arranged at an autonomous driving center, and an operator terminal 80 as an example of a remote operation device operated by an operator. The vehicle 10 in this embodiment is an autonomous vehicle that is operated automatically. The autonomous driving system 100 is a system that allows an operator at a remote location to remotely assist the vehicle 10 when the vehicle 10 (autonomous vehicle) arrives at an assistance point.

[0026] In this embodiment, remote support refers to an operator operating the vehicle 10 from a remote location, or an operator monitoring the vehicle 10 from a remote location. A support point refers to a point where it is difficult for the vehicle 10 to drive autonomously, or a point where the operator can remotely support the vehicle. Examples of support points include, but are not limited to, intersections, construction sites, parking lots, expressways, motorways, service areas, parking areas, baggage collection points, toll booths, and interchanges.

[0027] The autonomous driving system 100 of this embodiment includes one or more vehicles 10. Each vehicle 10 includes a vehicle-side communication device 12, a camera 14, a positioning device 16, and a vehicle-side control unit 18.

[0028] The vehicle-side communication device 12 performs wireless communication with the remote assistance device 50 .

[0029] As shown in FIG. 2, the camera 14 photographs the front range 30F, the right side range 30RS, the left side range 30LS, and the rear range 30R of the vehicle 10. The camera 14 may be configured with four cameras that photograph each range individually, or may be configured with a 360-degree camera that can photograph the entire range. In this embodiment, a case where four cameras that photograph each range are provided will be described. In the following, for convenience, a camera that photographs a range including at least the front range 30F will be referred to as camera 14F, a camera that photographs a range including at least the right side range 30RS will be referred to as camera 14RS, a camera that photographs a range including at least the left side range 30LS will be referred to as camera 14LS, and a camera that photographs a range including at least the rear range 30R will be referred to as camera 14R. In addition, when there is no need to distinguish between the cameras, they will be referred to as cameras 14.

[0030] As shown in FIG. 2, the forward range 30F is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the forward direction (traveling direction) F of the vehicle 10, when the vehicle 10 is viewed in a plan view. The right side range 30RS is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the right direction RS of the vehicle 10, when the vehicle 10 is viewed in a plan view. The left side range 30LS is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the left direction LS of the vehicle 10, when the vehicle 10 is viewed in a plan view. The rear range 30R is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the rear direction (reverse direction) R of the vehicle 10, when the vehicle 10 is viewed in a plan view.

[0031] The positioning device 16 includes a Global Positioning System (GPS) receiver or a Global Navigation Satellite System (GNSS) receiver, and determines the current position of the vehicle 10.

[0032] The vehicle-side control unit 18 controls the vehicle-side communication device 12, the camera 14, and the positioning device 16. The vehicle-side control unit 18 is also configured as a computer equipped with a CPU and a memory, and functions as an operation control unit 20 by the CPU executing a program stored in the memory.

[0033] The operation control unit 20 controls the operation (acceleration, deceleration, braking, steering) of the vehicle based on remote operation by the operator. The operation control unit 20 also repeatedly transmits vehicle information including identification information (vehicle ID) and current position information of the vehicle 10 to the remote support device 50 via the vehicle-side communication device 12 at a predetermined cycle. The vehicle information may include information for identifying the vehicle type and occupants of the vehicle 10, and information indicating the location of the destination. When remote support of the vehicle 10 is performed by the operator, the operation control unit 20 also transmits images captured by the camera 14 to the remote support device 50 via the vehicle-side communication device 12. In this embodiment, the captured images are, as an example, moving images (video).

[0034] The remote support device 50 includes a communication unit 52, a control unit 60, and a storage device 70.

[0035] The communication unit 52 performs wireless communication with the vehicle 10. The communication unit 52 also performs wired or wireless communication with the operator terminal 80.

[0036] The storage device 70 stores a map database 72, an operator information database 74, and a remote assistance program 76.

[0037] The positions of the support points are registered in the map database 72. In addition, road data showing the connection state of intersections and roads, congestion information, traffic regulation information, and traffic information including the traffic volume for each support point are recorded in the map database 72. The traffic information is obtained from another server via the communication unit 52, for example, and stored.

[0038] The operator information database 74 records, for example, the schedule of each operator providing remote support to vehicles, the number of operators currently available, the content of the remote support, the traffic volume history at each support point, and the history of the time required for remote support.

[0039] The control unit 60 controls the communication unit 52 and the storage device 70. The control unit 60 functionally includes a route planning unit 62, a remote support unit 64, a prediction unit 66, a recording unit 68, and an allocation unit 69.

[0040] The route planning unit 62 specifies a route along which each vehicle 10 will travel. More specifically, the route planning unit 62 specifies a route along which each vehicle 10 will travel from a departure point to a destination point while passing through support points, based on road data, traffic information, and the positions of the support points recorded in the map database 72. The departure point, destination, and support points to be passed through are set, for example, by an operator or an administrator, or are set in response to a request from the vehicle 10. The route planning unit 62 distributes the specified route to each vehicle 10 via the communication unit 52. Note that the route may be specified in the vehicle-side control unit 18 of each vehicle 10. Also, the route may be specified in response to a request from each vehicle 10.

[0041] The remote support unit 64 performs remote support for the vehicle 10 at a support point based on remote operation by an operator. As shown in FIG. 3, the remote support unit 64 includes a front image transmission unit 40, a receiving unit 42, an operation signal transmission unit 44, and a rear image transmission unit 46.

[0042] When remote assistance at an assistance point is requested by vehicle 10, the forward image transmission unit 40 transmits a forward image including at least the area in front of the vehicle 10 captured by vehicle 10, for example a forward image including the forward range 30F, to an operator terminal 80 that remotely controls vehicle 10.

[0043] The receiving unit 42 receives an operation signal corresponding to a remote operation at the support point from the operator terminal 80.

[0044] The operation signal transmitting unit 44 transmits the operation signal received from the operator terminal 80 to the vehicle 10 .

[0045] After the vehicle 10 passes a switching point that is set to switch the display on the operator terminal 80 based on the support point, the rear image transmission unit 46 transmits to the operator terminal 80 a rear image that is captured by the vehicle 10 to include at least the rear of the vehicle 10 and emphasizes the rear, for example a rear image including the rear range 30R.

[0046] The prediction unit 66 predicts the timing when remote support of the vehicle 10 will be started by the operator based on the estimated time when the vehicle 10 will arrive at a support point included in the route specified by the route planning unit 62, and also predicts the time for which the remote support will be performed (hereinafter referred to as "required time"). The prediction unit 66 can, for example, learn the relationship between the past required time and traffic volume for each support point, and predict the required time from the learning result and the current traffic volume. The prediction unit 66 in this embodiment has a function of predicting the number of operators required to simultaneously support multiple vehicles based on the estimated time when the vehicle 10 will arrive at a support point included in the route specified for each vehicle 10.

[0047] The recording unit 68 records in the operator information database 74 the history of the number of operators who provided remote support for each time period. The recording unit 68 also records in the operator information database 74 the history of the time required for each operator to remotely support the vehicle 10 at a support point for each support point. The recording unit 68 also records the traffic volume history for each support point in the operator information database 74. The recording unit 68 also records the support content in the operator information database 74. The support content includes, for example, right turn support and left turn support at intersections, door opening / closing support and starting support at bus stops, obstacle avoidance, and handling of pedestrians crossing the road.

[0048] The allocation unit 69 allocates operators to support points included in the route of each vehicle 10 at the start timing of remote support predicted by the prediction unit 66 or at the timing of receiving a remote support request from the vehicle 10, and determines a schedule for each operator. The allocation unit 69 records the determined schedule in the operator information database 74. In the schedule, for example, the time when remote support is to be performed, the support point, and the vehicle are registered for each operator. Note that in this embodiment, a case will be described where an operator is allocated when a remote support request is received from the vehicle 10.

[0049] 4 is a diagram showing the hardware configuration of the control unit 60. The control unit 60 is configured with a device including a general computer.

[0050] 4, the control unit 60 includes a central processing unit (CPU) 60A, a read only memory (ROM) 60B, a random access memory (RAM) 60C, a non-volatile memory 60D, and an input / output interface (I / O) 60E. The CPU 60A, the ROM 60B, the RAM 60C, the non-volatile memory 60D, and the I / O 60E are connected to each other via a bus 60F. The communication unit 52 and the storage device 70 are connected to the I / O 60E.

[0051] The CPU 60A is an example of a computer. The term "computer" used here refers to a processor in a broad sense, including a general-purpose processor (e.g., a CPU) and a dedicated processor (e.g., a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, etc.).

[0052] The storage device 70 is configured with a non-volatile storage device such as a hard disk. The CPU 60A reads and executes a remote assistance program 76 stored in the storage device 70. This allows the CPU 60A to function as each functional unit constituting the control unit 60 described above. The remote assistance program 76 may be realized by storing it in a non-volatile non-transitory recording medium or distributing it via a network and installing it appropriately in the remote assistance device 50.

[0053] Examples of non-volatile non-transient recording media include CD-ROMs (Compact Disc Read Only Memory), magneto-optical disks, HDDs (Hard Disk Drives), DVD-ROMs (Digital Versatile Disc Read Only Memory), flash memories, memory cards, etc.

[0054] The operator terminal 80 is configured as a computer or a mobile terminal equipped with a CPU and a memory. The autonomous driving system 100 in this embodiment includes one or more operator terminals 80. Each operator terminal 80 includes an operation unit 82 and a display unit 84. The operation unit 82 accepts operations for remotely supporting the vehicle 10 from the operator. The accepted operations are transmitted as operation signals to the vehicle 10 through the remote support device 50. The display unit 84 displays a user interface used for the operator's operation. In addition, the display unit 84 displays an image of the camera 14 transmitted from the vehicle 10.

[0055] The remote support process executed by the CPU 60A of the control unit 60 of the autonomous driving system 100 will be described with reference to Fig. 5. The remote support process shown in Fig. 5 is a process that is repeatedly executed at a predetermined cycle by the remote support device 50 while the autonomous driving system 100 is in operation. Note that, in the following description, it is assumed that the route of each vehicle 10 has been specified by the route planning unit 62.

[0056] When the vehicle 10 is performing autonomous driving along a set route, if the vehicle 10 reaches an assistance point and determines that remote assistance from an operator is required, the vehicle 10 transmits a remote assistance request to the remote assistance device 50.

[0057] Therefore, in step S100, the CPU 60A determines whether or not a remote support request has been received from the vehicle 10. If a remote support request has been received, the process proceeds to step S102. On the other hand, if a remote support request has not been received from the vehicle 10, the process waits until a remote support request is received.

[0058] In step S102, the CPU 60A executes an operator allocation process. Specifically, for example, based on the schedule recorded in the operator information database 74, an operator is allocated to the vehicle 10 that has received the remote assistance request.

[0059] In step S104, the CPU 60A receives the vehicle information and captured images transmitted from the vehicle 10. The captured images are four captured images (moving images) of the respective ranges captured by the cameras 14F, 14RS, 14LS, and 14R. The vehicle information and captured images are periodically transmitted from the vehicle 10, and are successively received from step S104 onwards until this routine is terminated.

[0060] In step S106, the CPU 60A calculates the position of a switching point at which the type of captured image to be transmitted to the operator terminal 80 is switched after the remote assistance is completed. Specifically, the position of the switching point is calculated based on the current position of the vehicle 10 included in the vehicle information received in step S104, i.e., the position of the assistance point where remote assistance was requested, and the route of the vehicle 10. Specifically, the position of the switching point is the position of a point where it can be considered that the operation corresponding to the remote operation has ended when the vehicle 10 has passed the switching point.

[0061] For example, as shown in Fig. 6A, assume that the vehicle 10 is making a right turn at an intersection Cr according to a route Rt, and requests remote assistance at the intersection Cr, and is instructed to turn right as a remote operation. In this case, the point where the vehicle 10 turns right is set as a switching point Cg.

[0062] 7A, for example, a case is assumed in which the vehicle 10 travels along the non-priority lane S1 according to the route Rt, turns left at an intersection Cr, and travels along a priority road S2, and then requests remote assistance just before the intersection Cr, and is instructed to turn left as a remote operation. In this case, the point where the vehicle 10 turns left is set as the switching point Cg.

[0063] 8A, for example, a case is assumed in which the vehicle 10 stops at a bus stop BS according to the route Rt, then departs and travels along a lane S1, the vehicle requests remote assistance at the bus stop BS, and is instructed to merge onto a road S1 by remote operation. In this case, the point where the vehicle 10 merges into the lane S1 is set as the switching point Cg.

[0064] 9A, for example, a situation is assumed in which the vehicle 10 encounters an obstacle OB while traveling in the left lane S1 of two lanes on one side according to the route Rt, and the vehicle requests remote assistance just before the obstacle OB and is instructed by remote control to travel in the adjacent lane S2 to the right. In this case, the point where the vehicle 10 changes lanes to the lane S2 is set as the switching point Cg.

[0065] 10A, for example, a case is assumed in which the vehicle 10 travels along the lane S1 following the route Rt, stops in front of the crosswalk PC to request remote assistance, and is instructed to start moving by remote control after the pedestrian PD crosses the crosswalk PC. In this case, the point where the vehicle 10 passes the center of the crosswalk is set as the switching point Cg.

[0066] In step S108, the CPU 60A transmits, as a forward image, an image including a captured image 32F of the forward range 30F, a captured image 32RS of the right side range 30RS, and a captured image 32LS of the left side range 30LS, as shown in Fig. 6A, among the captured images of each range received in step S104, to the operator terminal 80. As a result, on the display unit 84 of the operator terminal 80, as shown in Fig. 11, the captured image 32F is displayed in the center, the captured image 32RS is displayed on the right side, and the captured image 32LS is displayed on the left side.

[0067] 6A, 7A, and 10A, the forward image is a captured image of the range 30 including the forward range 30F, the right side range 30RS, and the left side range 30LS, but for example, as shown in Figures 8A and 9A, in a scene where the vehicle 10 travels diagonally forward to the right, a captured image of the range 30 including the forward range 30F, the right side range 30RS, and the rear range 30R may be transmitted as a forward image to the operator terminal 80. The process of step S108 is repeatedly executed every time a captured image is received from the vehicle 10 until this routine is terminated.

[0068] The operator of the operator terminal 80 checks the captured image displayed on the display unit 84 and operates the operation unit 82 to remotely control the vehicle 10. For example, as shown in Fig. 6A, if the vehicle 10 is in a situation where it is to turn right at an intersection Cr, the operator checks for oncoming vehicles and pedestrians, etc., and issues an instruction to turn right when it is determined that it is OK to turn right. When the operator operates the operation unit 82 to remotely control the vehicle 10, an operation signal corresponding to the remote operation is transmitted to the remote support device 50.

[0069] Therefore, in step S110, the CPU 60A determines whether or not an operation signal has been received from the operator terminal 80. If an operation signal has been received, the process proceeds to step S112. On the other hand, if an operation signal has not been received, the process waits until an operation signal is received.

[0070] In step S112, the CPU 60A transmits the operation signal received in step S110 to the vehicle 10. As a result, the vehicle 10 performs an operation according to the operation signal. For example, as in the example of Fig. 6A, if the operation signal is an operation signal indicating an instruction to turn right, the vehicle 10 performs an operation to turn right. As a result, the vehicle 10 turns right at the intersection Cr as shown in Fig. 6B.

[0071] In step S114, it is determined whether the vehicle 10 has passed the switching point Cg set in step S106. Specifically, it is determined whether the vehicle 10 has passed the switching point Cg based on the history of the current position of the vehicle 10 and the route contained in the vehicle information successively received from the vehicle 10.

[0072] Then, if the vehicle 10 has passed the switching point Cg, the process proceeds to step S116. On the other hand, if the vehicle 10 has not passed the switching point Cg, the process waits until the vehicle 10 passes the switching point Cg.

[0073] In step S116, the CPU 60A transmits, as rear images, images including the photographed image 32R of the rear range 30R, the photographed image 32RS of the right side range 30RS, and the photographed image 32LS of the left side range 30LS among the photographed images of each range received from the vehicle 10 to the operator terminal 80. As a result, on the display unit 84 of the operator terminal 80, as shown in FIG. 12, the photographed image 32R is displayed in the center, the photographed image 32RS is displayed on the right side, and the photographed image 32LS is displayed on the left side. That is, when the vehicle 10 passes the switching point Cg, the photographed image 32F that was displayed in the center of the display unit 84 until then is switched to the photographed image 32R. This allows the operator to check the rear of the vehicle 10 after remote assistance. The process of step S116 is repeatedly executed every time a photographed image is received from the vehicle 10 until this routine is terminated.

[0074] Further, in addition to the rear image, a photographed image 32F of the forward range 30F may be transmitted to the operator terminal 80. As a result, as shown in FIG. 13, the photographed image 32F is displayed at the top of the display unit 84, and the operator can check not only the rear but also the front. Furthermore, a single photographed image obtained by combining the photographed images 32R, 32RS, and 32LS may be generated as a rear image and transmitted to the operator terminal 80. As a result, as shown in FIG. 14, the rear image is displayed on the display unit 84 of the operator terminal 80 as a photographed image 32P like a single panoramic image.

[0075] In the example of Figure 6A, until the vehicle 10 passes the switching point Cg, where the vehicle 10 turns right, a forward image captured of the range 30 including the forward range 30F, the right side range 30RS, and the left side range 30LS is transmitted to the operator terminal 80 and displayed on the display unit 84.

[0076] Then, when the vehicle 10 passes the switching point Cg, as shown in Fig. 6B, a rear image captured of the range 30 including the rear range 30R, the right side range 30RS, and the left side range 30LS is transmitted to the operator terminal 80 and displayed on the display unit 84. The same applies to the cases of Figs. 7B, 8B, 9B, and 10B.

[0077] In this way, when the switching point Cg is passed, the image displayed on the display unit 84 of the operator terminal 80 is switched from a forward image to a rearward image. This allows the operator to easily evaluate whether the remote operation he or she instructed the vehicle 10 to perform was appropriate. For example, in the example of Figs. 6A and 6B, when the vehicle 10 turns right at the intersection Cr and passes the switching point Cg, the operator checks the rearward image. If the operator can see a pedestrian crossing the crosswalk in the rearward image, the operator can recognize that the pedestrian may have been overlooked even though the pedestrian is about to cross the crosswalk PC. Therefore, the operator is prompted to pay special attention to whether there is a pedestrian about to cross the crosswalk from the next time, which can help improve the operator's situation judgment ability.

[0078] In the example of Fig. 6B, the rear image is an image capturing the range 30 including the rear range 30R, the right side range 30RS, and the left side range 30LS, but a rear image of a capturing range determined according to the content of the remote operation may be transmitted to the operator terminal 80. For example, in the example of Fig. 6B, if the important range that the operator should check after turning right of the vehicle 10 is the rear range 30R and the left side range 30LS, a captured image of the range 30 including the rear range 30R and the left side range 30LS may be transmitted to the operator terminal 80. In this way, the capturing range of the rear image to be transmitted to the operator terminal 80 may be determined in advance according to the content of the remote operation.

[0079] In step S118, the CPU 60A determines whether the vehicle 10 has resumed autonomous driving. For example, after the vehicle 10 performs an operation corresponding to the operation signal, the vehicle 10 transmits a withdrawal request to the remote assistance device 50 to withdraw the remote assistance request, and resumes autonomous driving. Therefore, whether autonomous driving has resumed can be determined, for example, by whether a withdrawal request to withdraw the remote assistance request has been received from the vehicle 10.

[0080] If the vehicle 10 has resumed autonomous driving, the process proceeds to step S120. On the other hand, if the vehicle 10 has not resumed autonomous driving, the process waits until autonomous driving is resumed.

[0081] In step S120, CPU 60A stops transmission of captured images to operator terminal 80 and releases the allocation of the operator to the remote support of vehicle 10.

[0082] Thus, in this embodiment, until the vehicle 10 passes the switching point Cg, a forward image that includes at least the area in front of the vehicle 10, for example a forward image including the forward range 30F, is displayed on the display unit 84 of the operator terminal 80, and after the vehicle 10 passes the switching point Cg, a rear image that includes at least the area behind the vehicle 10, for example a rear image including the rear range 30R, is displayed on the display unit 84 of the operator terminal 80.

[0083] This allows the operator to easily and quickly evaluate whether the remote operation of the vehicle 10 that he or she performed was appropriate or not, thereby helping to improve the operator's situational judgment ability.

[0084] <Second embodiment>

[0085] A second embodiment will be described below. In the second embodiment, a gaze range is detected based on the line of sight of the operator, and a rear image including the detected gaze range is displayed on the display unit 84 of the operator terminal 80.

[0086] As shown in FIG. 15, an operator terminal 80 according to this embodiment includes an operation unit 82, a display unit 84, a camera 86, and a line-of-sight detection unit 88.

[0087] The camera 86 is provided at a position where it can photograph the operator's eyes, and photographs the operator's eyes.

[0088] The gaze detection unit 88 detects the gaze position indicating at what point in the forward image displayed on the display unit 84 the operator is looking based on the captured image including the operator's eyes captured by the camera 86, and sequentially transmits gaze position information indicating the detected gaze position to the remote support device 50.

[0089] The receiver 42 of the remote support device 50 receives the gaze position information transmitted from the operator terminal 80, and when the vehicle 10 passes the switching point Cg and transmits a rear image to the operator terminal 80, the rear image transmitter 46 specifies the gaze range based on the gaze position information received from the operator terminal 80 and transmits a rear image including the specified gaze range to the operator terminal 80. Here, the gaze range refers to a range in which the gaze position remains for a predetermined time (e.g., 2 seconds) or more.

[0090] If there are multiple gaze ranges, for example, three gaze ranges M1, M2, and M3 as shown in FIG. 16, the remote support device 50 transmits a rear image including all of the gaze ranges M1, M2, and M3 to the operator terminal 80.

[0091] In addition, a rear image including the gaze range with the longest gaze time among the multiple gaze ranges may be transmitted to the operator terminal 80. For example, when the gaze range with the longest gaze time among the gaze ranges M1, M2, and M3 is gaze range M1, a rear image including at least gaze range M1 may be transmitted to the operator terminal 80.

[0092] The rear image transmission unit 46 may transmit to the operator terminal 80 a rear image in which an object to be overlooked by the operator, which is identified based on the gaze range, is highlighted. For example, a known object extraction process is performed on the rear image to extract an object. Here, the object is a moving object such as a person or a vehicle. Then, among the objects extracted from the rear image, an object outside the gaze range is set as an object to be overlooked by the operator. Then, the rear image in which the object to be overlooked is highlighted is transmitted to the operator terminal 80. For example, as shown in FIG. 17, the object to be overlooked is highlighted by surrounding it with a frame W in the captured image 32R of the rear range 30R, and transmitted to the operator terminal 80. This allows the operator to easily recognize an object that may have been overlooked by the operator.

[0093] The rear image transmission unit 46 may transmit to the operator terminal 80 a rear image in which an object that was not detected in the front image but was newly detected in the rear image is highlighted. This allows the operator to easily check, for example, vehicles and pedestrians in the blind spot in the front image that the operator was unable to recognize. In this case, too, as shown in FIG. 17, the newly detected object in the rear image is highlighted, for example, by surrounding it with a frame W, and transmitted to the operator terminal 80. This allows the operator to easily recognize an object that was in the blind spot and that the operator was unable to notice.

[0094] The rear image transmission unit 46 may transmit to the operator terminal 80 a rear image on which speed information on the speed change of the oncoming vehicle detected by the vehicle 10 is displayed. In this case, the vehicle 10 is configured to include a radar device. The vehicle 10 detects speed information on the speed change of the oncoming vehicle and transmits it to the remote support device 50. The rear image transmission unit 46 transmits to the operator terminal 80 a rear image on which the speed information of the oncoming vehicle is displayed. For example, as shown in FIG. 18, when the speed information received from the vehicle 10 indicates that the oncoming vehicle has decelerated by 10 km / h, a rear image on which the oncoming vehicle is surrounded by a frame W and the text "10 km / h deceleration" is displayed is transmitted to the operator terminal 80. This allows the operator to recognize that the oncoming vehicle has decelerated by 10 km / h due to his / her remote operation, and to easily recognize that the remote operation may not have been appropriate.

[0095] 19, when an infrastructure camera 34 is installed that captures an overhead image of the area including the switching point Cg, the rear image transmission unit 46 may obtain the overhead image from the infrastructure camera 34 and transmit it to the operator terminal 80. As a result, the display unit 84 of the operator terminal 80 displays the overhead image 36 as shown in FIG. 20, making it possible to easily check the situation around the vehicle 10.

[0096] <Third embodiment>

[0097] A third embodiment will be described below. Note that the same parts as those in the above embodiments are given the same reference numerals, and detailed description thereof will be omitted.

[0098] In the third embodiment, a modified example of a forward image transmitted to the operator terminal 80 in step S108 in FIG. 5 and a modified example of a rearward image transmitted to the operator terminal 80 in step S116 in FIG. 5 will be described.

[0099] In the third embodiment, as shown in Fig. 21, the right lateral area 30RS shown in Fig. 2 is divided into a right front area 30FRS and a right rear area 30RRS, and the left lateral area 30LS shown in Fig. 2 is divided into two areas, a left front area 30FLS and a left rear area 30RLS.

[0100] In the third embodiment, in step S108 of FIG. 5, the CPU 60A transmits an image including the captured image 32F of the forward range 30F, the captured image 32FRS of the right forward range 30FRS, the captured image 32FLS of the left forward range 30FLS, the captured image 32R of the rear range 30R, the captured image 32RRS of the right rear range 30RRS, and the captured image 32RLS of the left rear range 30RLS to the operator terminal 80 as a forward image.

[0101] As a result, on the display unit 84 of the operator terminal 80, as shown in Fig. 22, the captured images 32F and 32R are displayed in the center, the captured images 32FRS and 32RRS are displayed on the right side, and the captured images 32FLS and 32RLS are displayed on the left side. Note that in the example of Fig. 21, the display area of ​​the captured image 32F is larger than that of the captured image 32R, the display area of ​​the captured image 32FRS is larger than that of the captured image 32RRS, and the display area of ​​the captured image 32FLS is larger than that of the captured image 32RLS. That is, the display unit 84 displays the front of the vehicle 10 more emphatically than the rear.

[0102] Also, in step S116 of Fig. 5, the CPU 60A transmits an image in which the rear is emphasized as a rear image to the operator terminal 80 as shown in Fig. 23. In the rear image shown in Fig. 23, compared to the front image shown in Fig. 22, the display area of ​​the captured image 32R is larger than that of the captured image 32F, the display area of ​​the captured image 32RRS is larger than that of the captured image 32FRS, and the display area of ​​the captured image 32RLS is larger than that of the captured image 32FLS. That is, the rear of the vehicle 10 is displayed on the display unit 84 with emphasis on the front of the vehicle 10.

[0103] <Fourth embodiment>

[0104] A fourth embodiment will be described below. Note that the same parts as those in the above embodiments are given the same reference numerals and detailed description will be omitted.

[0105] In the fourth embodiment, a modified example in which a rear image in which an object that is overlooked by the operator is emphasized is transmitted to the operator terminal 80 will be described.

[0106] FIG. 24 shows an example of a forward image transmitted by the CPU 60A to the operator terminal 80 in step S108 of FIG. 5. In the example of FIG. 24, the forward image includes a photographed image 32F of the forward range 30F, a photographed image 32FRS of the right forward range 30FRS, a photographed image 32FLS of the left forward range 30FLS, a photographed image 32R of the rear range 30R, a photographed image 32RRS of the right rear range 30RRS, and a photographed image 32RLS of the left rear range 30RLS. In the example of FIG. 24, the sizes of the display areas of the photographed images 32F and 32R are the same, the sizes of the display areas of the photographed images 32FRS and 32RRS are the same, and the sizes of the display areas of the photographed images 32FLS and RLS are the same.

[0107] Moreover, Fig. 25 shows an example of a rearward image transmitted by CPU 60A to operator terminal 80 in step S116 in Fig. 5. In the example of Fig. 25, the rearward image is an image in which an object that is to be overlooked by the operator is highlighted by surrounding it with frames W1 and W2 in the frontward image shown in Fig. 24.

[0108] The method of extracting objects that may have been overlooked by the operator is the same as that described in the second embodiment, and for example, a known object extraction process is executed on the front image and the rear image to extract objects. Then, among the objects extracted from the rear image, objects that were not included in the front image are set as objects that may have been overlooked by the operator. Then, the objects that may have been overlooked are highlighted by surrounding them with frames W1 and W2, and transmitted to the operator terminal 80. This allows the operator to easily recognize objects that may have been overlooked by the operator.

[0109] The present disclosure is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0110] For example, in the above embodiment, a case has been described in which an operator remotely controls the vehicle 10 by operating the operator terminal 80, but the disclosed technology can also be applied to a case in which the vehicle 10 is remotely controlled by operating a remote control device equipped with a steering wheel, an accelerator pedal, and a brake pedal.

[0111] In the above embodiment, the case of supporting the improvement of the situation judgment ability of the operator has been described, but the disclosed technology can also be applied to a case of supporting the improvement of the situation judgment ability of a supervisor who supervises an operator. The disclosed technology can also be applied to a case where a supervisor evaluates the remote support for an operator and provides feedback on improvements to the operator. The disclosed technology can also be applied to a case where the result of remote support is presented to another operator to support the improvement of the situation judgment ability of the other operator. For example, the disclosed technology can also be applied to a case where it is desired to train new employees or to match the levels of operators.

[0112] In addition, the configuration of the autonomous driving system 100 described in the above embodiment (see FIG. 1) is just one example, and it goes without saying that unnecessary parts may be deleted or new parts may be added without departing from the spirit of the present invention.

[0113] Furthermore, the processing flow of the remote assistance program 76 described in the above embodiment (see FIG. 5) is also one example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0114] The control unit and the method described herein may be implemented by a special-purpose computer having a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the device and the method described herein may be implemented by a special-purpose computer having a processor configured by dedicated hardware logic circuits. Alternatively, the device and the method described herein may be implemented by one or more special-purpose computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. The computer program may also be stored in a computer-readable non-transitory tangible recording medium as instructions to be executed by the computer.

[0115] The following notes are disclosed regarding the technology of the present disclosure. <Additional Notes> (Appendix 1) a forward image transmission unit (40) that transmits, when a remote assistance at an assistance point is requested from an autonomous vehicle (10), a forward image including at least a front of the autonomous vehicle, captured by the autonomous vehicle, to a remote operation device (80) that remotely operates the autonomous vehicle; a receiving unit (42) for receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; an operation signal transmission unit (44) that transmits the received operation signal to the autonomous driving vehicle; a rear image transmission unit (46) that transmits to the remote control device a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear after the autonomous vehicle passes a switching point that is set to switch the display on the remote control device based on the assistance point; and A remote support device (50) comprising: (Appendix 2) The rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote control. 2. A remote assistance device as described in claim 1. (Appendix 3) the receiving unit receives gaze position information regarding a gaze position of an operator of the remote control device in a forward image displayed on the remote control device, The rear image transmission unit transmits a rear image including the gaze range of the operator specified based on the gaze position information to the remote control device. 3. The remote assistance device according to claim 1 or 2. (Appendix 4) When there are a plurality of gaze ranges, the rear image transmission unit transmits to the remote control device a rear image including the gaze range having the longest gaze time among the plurality of gaze ranges. 4. A remote assistance device as described in claim 3. (Appendix 5) When there are a plurality of gaze ranges, the rear image transmission unit transmits a rear image including all of the plurality of gaze ranges to the remote control device. 4. A remote assistance device as described in claim 3. (Appendix 6) The rear image transmission unit transmits, to the remote control device, a rear image in which an object that is overlooked by an operator of the remote control device is highlighted. 6. A remote support device according to any one of claims 1 to 5. (Appendix 7) The rear image transmission unit transmits to the remote control device a rear image in which an object to be overlooked by the operator, which object is identified based on the gaze range, is highlighted. 6. A remote support device according to any one of claims 3 to 5. (Appendix 8) The rear image transmission unit transmits to the remote control device a rear image in which a newly detected object is highlighted in the rear image. 8. A remote support device according to any one of appendix 1 to 7. (Appendix 9) The rear image transmission unit transmits to the remote control device a rear image in which speed information related to a speed change of an oncoming vehicle detected by the autonomous driving vehicle is displayed. 9. A remote support device according to any one of appendix 1 to 8. (Appendix 10) The rear image transmission unit acquires an overhead image of an area including the switching point and transmits the overhead image to the remote control device. 10. A remote support device according to any one of appendix 1 to 9. (Appendix 11) The rear image is an image in which a display area displaying the rear of the autonomous vehicle is larger than a display area displaying the front of the autonomous vehicle. 2. The remote assistance device of claim 1. 11. A remote support device according to any one of claims 1 to 10. (Appendix 12) At least one processor (60A) When an autonomous vehicle requests remote assistance at an assistance point, the autonomous vehicle a remote control device that remotely controls the autonomous vehicle; and receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; Transmitting the received operation signal to the autonomous vehicle; After the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear is transmitted to the remote control device. A remote support method for performing a process including the steps of: (Appendix 13) At least one processor has When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle; receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; Transmitting the received operation signal to the autonomous vehicle; After the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear is transmitted to the remote control device. A remote support program (76) for executing a process including the steps of: [Explanation of symbols]

[0116] 10 vehicle, 40 forward image transmission unit, 42 receiving unit, 44 operation signal transmission unit, 46 rear image transmission unit, 50 remote support device, 52 communication unit, 60 control unit, 62 route planning unit, 64 remote support unit, 66 prediction unit, 68 recording unit, 69 allocation unit, 70 storage device, 72 map database, 74 operator information database, 76 remote support program, 80 operator terminal, 100 automatic driving system

Claims

1. a forward image transmission unit (40) that transmits, when a remote assistance at an assistance point is requested from an autonomous vehicle (10), a forward image including at least a front of the autonomous vehicle, captured by the autonomous vehicle, to a remote operation device (80) that remotely operates the autonomous vehicle; a receiving unit (42) for receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; an operation signal transmission unit (44) that transmits the received operation signal to the autonomous driving vehicle; a rear image transmission unit (46) that transmits to the remote control device a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear after the autonomous vehicle passes a switching point that is set to switch the display on the remote control device based on the assistance point; and A remote support device (50) comprising:

2. The rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote control. The remote assistance device according to claim 1.

3. the receiving unit receives gaze position information regarding a gaze position of an operator of the remote control device in a forward image displayed on the remote control device, The rear image transmission unit transmits a rear image including the gaze range of the operator specified based on the gaze position information to the remote control device. The remote assistance device according to claim 1.

4. When there are a plurality of gaze ranges, the rear image transmission unit transmits to the remote control device a rear image including the gaze range having the longest gaze time among the plurality of gaze ranges. The remote assistance device according to claim 3.

5. When there are a plurality of gaze ranges, the rear image transmission unit transmits a rear image including all of the plurality of gaze ranges to the remote control device. The remote assistance device according to claim 3.

6. The rear image transmission unit transmits, to the remote control device, a rear image in which an object that is overlooked by an operator of the remote control device is highlighted. A remote support device according to any one of claims 1 to 5.

7. The rear image transmission unit transmits to the remote control device a rear image in which an object to be overlooked by the operator, which object is identified based on the gaze range, is highlighted. A remote support device according to any one of claims 3 to 5.

8. The rear image transmission unit transmits to the remote control device a rear image in which a newly detected object is highlighted in the rear image. The remote assistance device according to claim 1.

9. The rear image transmission unit transmits to the remote control device a rear image in which speed information related to a speed change of an oncoming vehicle detected by the autonomous driving vehicle is displayed. The remote assistance device according to claim 1.

10. The rear image transmission unit acquires an overhead image of an area including the switching point and transmits the overhead image to the remote control device. The remote assistance device according to claim 1.

11. The rear image is an image in which a display area displaying the rear of the autonomous vehicle is larger than a display area displaying the front of the autonomous vehicle. The remote assistance device according to claim 1.

12. At least one processor (60A) When an autonomous vehicle requests remote assistance at an assistance point, the autonomous vehicle a remote control device that remotely controls the autonomous vehicle; and receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; Transmitting the received operation signal to the autonomous vehicle; After the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear is transmitted to the remote control device. A remote support method for performing a process including the steps of:

13. At least one processor, When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle; receiving an operation signal corresponding to a remote operation at the support point from the remote operation device; Transmitting the received operation signal to the autonomous vehicle; After the autonomous vehicle passes a switching point set to switch the display on the remote control device based on the assistance point, a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear is transmitted to the remote control device. A remote support program (76) for executing a process including the steps of:

Images