Remote assistance method, remote assistance system, and computer-readable recording medium

Abstract

The present disclosure relates to a remote assistance method, a remote assistance system, and a computer-readable recording medium. Provided is a remote assistance technology capable of causing a vehicle to act at a timing desired by a remote operator while excluding the influence of deviation of the assistance timing caused by, for example, communication delay. According to the remote assistance method of the present disclosure, first, a position relationship of the vehicle and an object at a future time point further ahead than a current time point, which is predicted on the basis of a path plan for autonomous travel created by the vehicle and information on the object existing in the surroundings of the vehicle, is spatially displayed on a display device. Next, an assistance content input from the operator is transmitted to the vehicle. Then, after the vehicle receives the assistance content, confirmation of the position relationship of the vehicle and the object displayed on the display device at the time when the assistance content is input is accepted, and remote assistance corresponding to the assistance content is executed in the vehicle.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a remote assistance technology that remotely assists a vehicle having an autonomous travel function. BACKGROUND

[0002] A remote assistance technology is known in which an automatic driving vehicle is remotely monitored, and in the case where assistance is needed, a remote operator conveys a judgment and an instruction to the vehicle, thereby assisting the automatic travel of the automatic driving vehicle. Patent Literature 1 discloses an example of such a remote assistance technology.

[0003] According to the prior art disclosed in Patent Literature 1, the remote driving device acquires, from the vehicle, information related to the vehicle and information related to an object in the surroundings of the vehicle. The information related to the vehicle includes the current geographical position of the vehicle, the current speed and acceleration of the vehicle, the identification information of the vehicle in the remote driving service, the future trajectory recommended by the autonomous driving function, and the like. The information related to the object includes the kind of the object, the current geographical position of the object, the speed and acceleration of the object, the future predicted trajectory of the object, and the like. The future predicted trajectory of the object is generated, for example, based on the geographical position, speed, and acceleration of the object, and the like.

[0004] According to the prior art disclosed in Patent Literature 1, the remote driving device displays a virtual object for representing the object on the display device. When deciding the display position of the virtual object, the remote driving device takes into account the delay that occurs when acquiring information from the vehicle and the delay that occurs when providing an operation instruction from the remote driving device to the vehicle. That is, according to the prior art disclosed in Patent Literature 1, not the current state of the surroundings of the vehicle, but the future state taking into account the delay time is displayed on the display device.

[0005] However, the above-described prior art, although taking into account the delay time to display the future state, can have an error between the assumed delay time and the actual delay time. In this case, it is not possible to make the vehicle act at the timing desired by the remote operator.

[0006] Note that, as a document indicating the technical level of the technical field associated with the present disclosure, in addition to the above-described Patent Literature 1, Patent Literature 2, Patent Literature 3, and Patent Literature 4 described below can be exemplified.

[0007] PRIOR ART DOCUMENTS

[0008] PATENT LITERATURE

[0009] Patent Literature 1: Japanese Patent Application Publication No. 2020-167551

[0010] Patent Literature 2: Japanese Patent Application Publication No. 2021-018744

[0011] Patent Literature 3: Japanese Patent Application Laid-Open No. 2021-033612

[0012] Patent Literature 4: Japanese Patent Application Laid-Open No. 2020-003890 SUMMARY

[0013] PROBLEMS TO BE SOLVED BY THE INVENTION

[0014] The present disclosure was completed in view of the above-described problems, and aims to provide a remote assistance technology capable of excluding the influence of a deviation in an assistance timing caused by a delay generated between a vehicle and a remote assistance terminal, and causing the vehicle to act at a timing desired by a remote operator.

[0015] SOLUTION TO THE PROBLEM

[0016] The present disclosure provides a remote assistance method and a remote assistance system as a remote assistance technology for achieving the above-described aim.

[0017] The remote assistance method of the present disclosure is a method of remotely assisting a vehicle having an autonomous travel function. The remote assistance method of the present disclosure includes: spatially displaying, on a display device, a positional relationship between the vehicle and an object at a future time point further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle and information related to the object existing in the surroundings of the vehicle; and transmitting assistance content input from a remote operator to the vehicle. Further, the remote assistance method of the present disclosure includes: after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device at the time when the assistance content is input, and executing remote assistance corresponding to the assistance content in the vehicle.

[0018] The remote assistance system of the present disclosure is a system of remotely assisting a vehicle having an autonomous travel function. The remote assistance system of the present disclosure includes: at least one memory including at least one program; and at least one processor combined with the at least one memory. The at least one program is configured to cause the at least one processor to execute the following processing. The processing executed by the processor includes: spatially displaying, on a display device, a positional relationship between the vehicle and an object at a future time point further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle and information related to the object existing in the surroundings of the vehicle; and transmitting assistance content input from a remote operator to the vehicle. Further, the processing executed by the processor includes: after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device at the time when the assistance content is input, and executing remote assistance corresponding to the assistance content in the vehicle.

[0019] According to the remote assistance technology of the present disclosure, remote assistance corresponding to the assistance content input by the remote operator is executed in the vehicle on the condition that the confirmation of the positional relationship of the vehicle and the surrounding object displayed on the display device at the time when the assistance content is input by the remote operator is achieved. That is, before the above-mentioned confirmation is made, even if the vehicle receives the assistance content, remote assistance corresponding to the assistance content is not executed in the vehicle. Thus, the influence of the deviation of the assistance timing caused by the delay generated between the vehicle and the remote assistance terminal can be eliminated, the vehicle is caused to act at the timing desired by the remote operator, and thus the reliability of the judgment of the remote operator can be improved.

[0020] In the remote assistance technology of the present disclosure, the above-mentioned confirmation can also include confirmation that the degree of matching of the positional relationship of the vehicle and the object displayed on the display device at the time when the assistance content is input and the actual positional relationship of the vehicle and the object acquired by the sensor of the vehicle is within an allowable range. In a case where an error is included in the prediction based on the route plan and the information related to the surrounding object, the predicted positional relationship of the vehicle and the object and the actual positional relationship of the vehicle and the surrounding object can deviate. By using the degree of matching of the predicted positional relationship of the vehicle and the object and the actual positional relationship of the vehicle and the surrounding object as a judgment material, it is possible to accurately judge whether the positional relationship of the vehicle and the surrounding object displayed on the display device at the time when the assistance content is input by the remote operator is achieved.

[0021] In addition, in the remote assistance technology of the present disclosure, the information related to the object existing around the vehicle can be information acquired by a sensor, and at least a part of the information related to the object acquired by the sensor can be used for creation of the route plan. Thus, the actual positional relationship of the vehicle and the surrounding object predicted based on the information acquired by the sensor of the vehicle and the actual positional relationship of the vehicle and the surrounding object acquired by the same sensor are compared, and thus the accuracy of the judgment based on the degree of matching can be improved.

[0022] In addition, in the remote assistance technology of the present disclosure, the above-mentioned confirmation can include confirmation that the timing at which remote assistance corresponding to the assistance content can be executed in the vehicle does not exceed the future timing of the positional relationship of the vehicle and the object displayed on the display device at the time when the assistance content is input. If the timing at which remote assistance can be executed in the vehicle does not exceed the future timing of the display device at the time when the assistance content is input, it can be judged that the positional relationship of the vehicle and the surrounding object displayed on the display device is achieved or will be achieved in the future. However, in a case where the timing at which remote assistance can be executed in the vehicle exceeds the above-mentioned future timing, it can be judged that the positional relationship of the vehicle and the surrounding object displayed on the display device cannot be achieved.

[0023] The present disclosure provides a program for achieving the above object. The program of the present disclosure is configured to cause a computer to execute the following processing. The processing executed by the computer includes: spatially displaying, on a display device, a positional relationship between a vehicle and an object at a future time point further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle and information related to the object existing in the surroundings of the vehicle; and transmitting assistance content input from a remote operator to the vehicle. Further, the processing executed by the computer includes: after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device at the time when the assistance content is input, and executing remote assistance corresponding to the assistance content in the vehicle. The program of the present disclosure can be stored in a recording medium readable by a computer or provided via a network.

[0024] Effects of the Invention

[0025] As described above, according to the remote assistance technology of the present disclosure, it is possible to eliminate the influence of deviation of the timing of assistance caused by a delay generated between the vehicle and the remote assistance terminal, and to cause the vehicle to act at the timing desired by the remote operator. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a configuration diagram of a remote assistance system of an autonomous vehicle to which a remote assistance method of the present disclosure is applied.

[0027] Figure 2 is a block diagram showing an example of the configuration of an autonomous vehicle.

[0028] Figure 3 is a block diagram showing an example of the configuration of a remote assistance terminal.

[0029] Figure 4 is a diagram showing an example of an image acquired by a camera of an autonomous vehicle and an example of a display screen of a remote assistance terminal.

[0030] Figure 5 is a diagram for explaining the outline of a remote assistance method of an embodiment of the present disclosure.

[0031] Figure 6 is a diagram for explaining the outline of a remote assistance method of an embodiment of the present disclosure.

[0032] Figure 7 is a diagram for explaining the outline of a remote assistance method of an embodiment of the present disclosure.

[0033] Figure 8 is a diagram for explaining a first example of a method of calculating a matching degree.

[0034] Figure 9 is a diagram for explaining a second example of a method of calculating a matching degree.

[0035] Figure 10 is a diagram for explaining the second example of the calculation method of the matching degree.

[0036] Figure 11 is a block diagram showing the structure of a remote assistance system of an embodiment of the present disclosure.

[0037] Figure 12 is a sequence chart showing the flow of processing between a vehicle, a remote assistance terminal, and an operator in a remote assistance system of an embodiment of the present disclosure.

[0038] Figure 13 is a flowchart showing an example of processing of a remote assistance terminal of an embodiment of the present disclosure.

[0039] Figure 14 is a flowchart showing an example of processing of an automated driving vehicle of an embodiment of the present disclosure.

[0040] Figure 15 is a flowchart showing another example of processing of an automated driving vehicle of an embodiment of the present disclosure.

[0041] Explanation of Reference Numerals

[0042] 20, automated driving vehicle; 21, computer; 21a, processor; 21b, memory; 21c, program; 30, remote assistance terminal; 31, computer; 31a, processor; 31b, memory; 31c, program; 32, display device; 33, input device; 40, remote operator; 50, other vehicle; 100, remote assistance system. DETAILED DESCRIPTION

[0043] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. However, in the case where the number, the quantity, the amount, the range, and the like of each element are mentioned in the embodiment shown below, the technical idea of the present disclosure is not limited to the mentioned number, unless it is specifically mentioned, or it is clearly determined as the number in principle. In addition, the configuration and the like described in the embodiment shown below is not necessarily essential to the technical idea of the present disclosure, unless it is specifically mentioned, or it is clearly determined as the configuration and the like in principle.

[0044] 1. Structure of Remote Assistance System

[0045] Figure 1is a configuration diagram of a remote assistance system for an automated driving vehicle. The remote assistance system 100 is a system that assists the automated driving of an automated driving vehicle 20 having an autonomous driving function through remote operation by an operator 40. The operator 40 who will remotely assist the automated driving vehicle 20 is referred to as a remote operator. As the level of automated driving of the automated driving vehicle 20 that is the target of remote assistance, for example, Level 4 or Level 5 is assumed. Hereinafter, the automated driving vehicle 20 will be simply referred to as the vehicle 20.

[0046] The remote assistance of the operator 40 to the vehicle 20 uses a remote assistance terminal 30. The remote assistance terminal 30 is connected to the server 10 of the management center via a communication network including the Internet. The vehicle 20 is also connected to the server 10 of the management center via a communication network including 4G, 5G. A plurality of remote assistance terminals 30 and a plurality of vehicles 20 are connected to the server 10. The server 10 that has received an assistance request from the vehicle 20 selects a person in charge from among the idle operators 40, and connects the remote assistance terminal 30 of the operator 40 who will be the person in charge to the vehicle 20 that has requested assistance.

[0047] In a scenario in which the vehicle 20 requests assistance, for example, a case including overtaking a preceding vehicle, a case passing through a pedestrian crossing, a scenario of right turn at an intersection, a scenario of detouring from a lane to avoid an obstacle, and the like. In remote assistance, at least a part of the judgment for the automated driving of the vehicle 20 is performed by the operator 40. Basic calculations related to the cognition, judgment, and operation required for driving are performed in the vehicle 20. The operator 40 judges the action that the vehicle 20 should take based on various vehicle information transmitted from the vehicle 20, and transmits an instruction to the vehicle 20 based on the judgment result. Among the instructions of remote assistance transmitted from the operator 40 to the vehicle 20, there are included an instruction of the progress of the vehicle 20 and an instruction of the stop of the vehicle 20. In addition, the instructions of remote assistance include an instruction of avoidance of deviation with respect to an obstacle ahead, an indication of overtaking a preceding vehicle, an indication of emergency avoidance, and the like.

[0048] Figure 2 is a block diagram showing an example of the configuration of the vehicle 20. The vehicle 20 is provided with a computer 21. The computer 21 is a collection of a plurality of ECUs (Electronic Control Units) mounted on the vehicle 20. In addition, the vehicle 20 is provided with an external sensor 22, an internal sensor 23, an actuator 24, and a communication device 25. They are connected to the computer 21 using a vehicle-mounted network such as CAN (Controller Area Network).

[0049] The computer 21 is provided with one or a plurality of processors 21a (hereinafter, simply referred to as processors 21a) and one or a plurality of memories 21b (hereinafter, simply referred to as memories 21b) coupled to the processors 21a. In the memories 21b, one or a plurality of programs 21c (hereinafter, simply referred to as programs 21c) executable by the processors 21a and various information associated therewith are stored. The programs 21c include a plurality of instructions executable. In addition, in the memories 21b, high-precision three-dimensional map information for automatic driving is stored.

[0050] Various processes of the processors 21a are implemented by executing the programs 21c by the processors 21a. The programs 21c include an automatic driving program for implementing automatic driving. By executing the automatic driving program by the processors 21a, the computer 21 functions as an automatic driving system that automatically drives the vehicle 20. Hereinafter, the computer 21 as the automatic driving system will be simply referred to as an automatic driving system. In addition, the programs 21c include a remote assistance program for accepting remote assistance. By executing the remote assistance program by the processors 21a, a process for requesting assistance to the server 10 and executing the assistance content acquired from the server 10 by the vehicle 20 is performed.

[0051] The external sensor 22 includes a camera that captures the surroundings of the vehicle 20, particularly, the front of the vehicle 20. The camera can also be provided in multiple, and in addition to capturing the front of the vehicle 20, can also capture the side and the rear. In addition, the camera can be shared in common in automatic driving and remote assistance for the operator 40, or a camera for automatic driving and a camera for remote assistance can be provided separately. The external sensor 22 includes an identification sensor other than the camera. The identification sensor is a sensor that acquires information for identifying the situation of the surroundings of the vehicle 20. As the identification sensor other than the camera, a LiDAR (Laser Imaging Detection and Ranging) and a millimeter wave radar are exemplified. In addition, the external sensor 22 includes a position sensor that detects the position and orientation of the vehicle 20. As the position sensor, a GPS (Global Positioning System) sensor is exemplified. The information obtained by the external sensor 22 is transmitted to the computer 21.

[0052] The internal sensor 23 includes a state sensor that acquires information related to the motion of the vehicle 20. As the state sensor, for example, a wheel speed sensor, an acceleration sensor, an angular velocity sensor, and a steering angle sensor are exemplified. The acceleration sensor and the angular velocity sensor can also be an IMU. The information obtained by the internal sensor 23 is transmitted to the computer 21.

[0053] Hereinafter, information obtained by the internal sensor 23 and information obtained by the external sensor 22 are collectively referred to as vehicle information. Among them, in addition to information acquired by the sensors of the vehicle 20, the vehicle information also includes a path plan of the vehicle 20 created by the automated driving system and prediction information of the surrounding environment of the vehicle 20. The automated driving system predicts a change in the surrounding environment of the vehicle 20 in a short time to come using information obtained by the external sensor 22 regarding the situation around the vehicle 20, and creates a path plan based on the prediction information and a target path decided by, for example, a navigation system.

[0054] The actuator 24 includes a steering manipulation device that performs steering manipulation of the vehicle 20, a drive device that performs driving of the vehicle 20, and a brake device that performs braking of the vehicle 20. The steering manipulation device includes, for example, a power steering system, a steer-by-wire steering manipulation system, and a rear wheel steering manipulation system. The drive device includes, for example, an engine, an EV system, and a hybrid system. The brake device includes, for example, a hydraulic brake and an electric power regenerative brake. The actuator 24 acts in accordance with a control signal transmitted from the computer 21.

[0055] The communication device 25 is a device that controls wireless communication of the vehicle 20 with the outside. The communication device 25 communicates with the server 10 via a communication network. Information processed by the computer 21 is transmitted to the server 10 using the communication device 25. Information processed by the server 10 is taken into the computer 21 using the communication device 25. In addition, in a case where vehicle-to-vehicle communication with other vehicles and road-to-vehicle communication with infrastructure are required for automated driving, communication with these external devices is also performed by the communication device 25.

[0056] Figure 3 is a block diagram that shows an example of the structure of the remote assistance terminal 30. The remote assistance terminal 30 is provided with a computer 31, a display device 32, an input device 33, and a communication device 35. The display device 32, the input device 33, and the communication device 35 are connected to the computer 31. Note that the remote assistance terminal 30 can be provided in the management center, or outside the management center, for example, in the home of the operator 40.

[0057] The computer 31 is provided with one or a plurality of processors 31a (hereinafter, simply referred to as processors 31a) and one or a plurality of memories 31b (hereinafter, simply referred to as memories 31b) associated with the processors 31a. In the memories 31b, one or a plurality of programs 31c (hereinafter, simply referred to as programs 31c) executable by the processors 31a and various information associated therewith are stored. The programs 31c include a plurality of instructions executable. In addition, in the memories 31b, high-precision three-dimensional map information is stored.

[0058] Processor 31a executes program 31c, thereby performing various processes. Program 31c includes an operator UI management program for managing the user interface used for remote assistance of vehicle 20. By executing the operator UI management program by processor 31a, computer 31 functions as an operator UI management system, performing processes for displaying information required for remote assistance on display device 32, described later.

[0059] Display device 32 is a device for displaying information required by operator 40 for remote assistance. Specifically, the information displayed by display device 32 is equivalent to an image of the front of vehicle 20. However, the information displayed on display device 32 is not an image captured by the camera of vehicle 20, i.e., information used by the autonomous driving system of vehicle 20. Figure 4 This represents an example of an image captured by a camera on vehicle 20. Figure 4 A) and an example of the screen on display device 32 ( Figure 4 B). The information displayed on display device 32 is an image based on a 3D model created by computer 31. In this image, the surrounding environment of vehicle 20 is reproduced in a 3D generated space. Regarding stationary objects in the surrounding environment, high-precision 3D map information and the position information of vehicle 20 can be spatially reproduced. The content of the information displayed on display device 32 and its generation method will be explained in detail later.

[0060] Input device 33 is a device for remotely assisting operator 40 in operation. Information input by input device 33 is processed by computer 31 and sent to vehicle 20. Specific examples of input device 33 include buttons, levers, and touch panels. For example, the direction of pushing the lever can instruct vehicle 20 to move / stop or to move laterally. Lateral movement includes, for example, avoiding obstacles in front, lane changing, and overtaking of vehicles ahead.

[0061] The communication device 35 is a device for controlling communication between the remote auxiliary terminal 30 and external entities. The communication device 35 communicates with the server 10 via a communication network. Information processed by the computer 31 is sent to the server 10 using the communication device 35. Information processed by the server 10 is retrieved into the computer 31 using the communication device 35.

[0062] 2. Overview of Remote Assistance Methods

[0063] The purpose of the remote assistance method disclosed herein is to eliminate the influence of deviations in assistance timing caused by delays between vehicle 20 and remote assistance terminal 30, and to enable vehicle 20 to act at the timing desired by operator 40. To achieve this purpose, in the remote assistance method of this disclosure, a future image generated based on vehicle information is displayed on the display device 32 of the remote assistance terminal 30, rather than the image of the front of vehicle 20 captured by a camera mounted on vehicle 20 itself. The future image displayed on the display device 32 refers to an image predicted to occur at a time further into the future than the current time. The vehicle information used to generate the future image includes a path plan for vehicle 20 generated by the autonomous driving system and prediction information of the surrounding environment of vehicle 20 used to generate the path plan. The generation of the path plan and the prediction of the surrounding environment are basic and well-known functions for autonomous driving systems, therefore, descriptions related to them are omitted. Vehicle 20 sends this vehicle information to remote assistance terminal 30 at least from the time assistance is requested until assistance is completed.

[0064] In the remote assistance method of this embodiment, the remote assistance terminal 30 sends assistance content input by the operator 40 and future information about the vehicle 20 generated by the computer 31 to the vehicle 20 via the server 10. The future information about the vehicle 20 generated by the computer 31 includes information related to the positional relationship between the vehicle 20 and objects existing around the vehicle 20 at a future time, taking into account the delay time described later. Based on the received assistance content and future information, the computer 21 of the vehicle 20 determines whether the assistance can be performed when it is actually needed.

[0065] The following uses Figures 5-7 The general outline of the remote assistance method of this embodiment will be described in detail. Here, we imagine a situation where vehicle 20 wants to turn right at an intersection, and other vehicles 50 (hereinafter referred to as other vehicles) enter the intersection from the opposite lane.

[0066] First of all, Figures 5-7 This provides an overall overview. Figures 5-7 The upper part of each diagram depicts a schematic diagram showing the positional relationship between vehicle 20 and other vehicles 50, as identified from images taken by the camera on vehicle 20. These schematic diagrams are equivalent to... Figure 4 The camera image shown in Figure A is transformed into a top-down view. The five schematic diagrams shown above are all at different times, arranged from left to right in time sequence: T0, T11, T21, T31, and T41. The image captured by the camera on vehicle 20 is a moving image, and the positional relationship between vehicle 20 and other vehicles 50 is also identified within a continuous time stream. Here, only the positional relationships at specific moments within a continuous time frame are shown.

[0067] The automatic driving system of the vehicle 20 creates a path plan of the vehicle 20 based on the images of the cameras, and calculates a predicted path of the other vehicle 50. In each diagram, the arrow lines of solid lines and dotted lines extending from the vehicle 20 are the path plan created by the automatic driving system. The path plan includes a target position of the vehicle 20 in a coordinate system centered on the vehicle 20 or an absolute coordinate system and a target speed or a target acceleration at the position. The solid lines are the determined path plan, and the dotted lines are the undetermined path plan. The arrow lines of dotted lines extending from the other vehicle 50 are the predicted path of the other vehicle 50 calculated by the automatic driving system.

[0068] In Figures 5-7 The lower rows of each of the diagrams depict diagrams representing the positional relationship of the vehicle 20 and the other vehicle 50 as projected in the image of the display device 32 of the remote assistance terminal 30. These diagrams correspond to the image displayed on the display device 32. Figure 4 The 3D image shown in B is converted into an image of a top view. The three diagrams shown in the lower rows are different in time, and are arranged in time series from left to right in the order of times T12, T22, and T32. Among them, the image displayed on the display device 32 is a dynamic image, and the positional relationship of the vehicle 20 and the other vehicle 50 in the image changes continuously according to time. Here, only the positional relationship at a specific time in the continuous time is picked up to be shown.

[0069] The computer 31 of the remote assistance terminal 30 predicts the positional relationship of the vehicle 20 and the other vehicle 50 at a time further in the future than the current time point, based on the path plan of the vehicle 20 received from the vehicle 20 and the prediction information of the surrounding environment including the predicted path of the other vehicle 50. In each diagram, the arrow lines of solid lines and dotted lines extending from the vehicle 20 are the path plan received from the vehicle 20. The solid lines are the path plan approved by the operator 40, and the dotted lines are the path plan not approved by the operator 40. The arrow lines of dotted lines extending from the other vehicle 50 are the predicted path of the other vehicle 50 at the future time predicted by the computer 31. The computer 31, for example, predicts the probability of the other vehicle 50 going straight on the current lane, the probability of the other vehicle 50 making a right turn, and the probability of the other vehicle 50 making a left turn, based on the map information and the speed and position of the other vehicle 50. Note that in the case where the object in the periphery of the vehicle 20 is a pedestrian, the computer 31, for example, divides the road into grids, uses the speed and position of the pedestrian to calculate the probability of the pedestrian passing through each grid, and thereby predicts the travel route of the pedestrian.

[0070] Figures 5-7 Each of the diagrams represents an example of remote assistance achieved by the remote assistance method of the present embodiment. From the example shown in A, Figure 5 The examples shown in each of the diagrams will be described in order from the example shown in A.

[0071] In Figure 5 In the example shown, a request for assistance is made from the vehicle 20 at time T0. In this example, the assistance requested by the vehicle 20 is a judgment as to whether or not a right turn can be performed at the intersection located ahead. The vehicle 20 requests assistance not after entering the intersection but at a predetermined distance from the intersection, near the front. At the time point of time T0 when the vehicle 20 requests assistance, it is not clear whether or not the other vehicle 50 on the opposite lane is straight, left, or right. In such a situation, the automatic driving system of the vehicle 20 does not make a judgment by itself but entrusts the judgment to the operator 40 and waits for the judgment result to be sent. However, during the period of time when assistance is requested, the vehicle 20 does not stop completely at the site but slowly travels at a low speed to the vicinity of the center of the intersection in order to suppress the influence on the following vehicle.

[0072] In response to the assistance request of the vehicle 20, the server 10 assigns an operator 40, and transmits vehicle information including the route plan of the vehicle 20 and the predicted route of the other vehicle 50 to the remote assistance terminal 30 of the assigned operator 40 together with the assistance request. The computer 31 of the remote assistance terminal 30 generates a future image taking into account the delay time on the basis of the vehicle information, and displays the generated future image on the display device 32 at time T12. Time is required from when the image including the positional relationship of the vehicle 20 and the other vehicle 50 is acquired by the camera of the vehicle 20, to when the vehicle information transmitted from the vehicle 20 is received by the remote assistance terminal 30 and the future image can be displayed by the remote assistance terminal 30. This time is the delay time in the uplink direction.

[0073] The total of the delay time in the uplink direction and the delay time in the downlink direction described later is the delay time taken into account in the generation of the future image. The delay time includes the communication delay and the operation time of each computer 21, 31 of the vehicle 20 and the remote assistance terminal 30. The operation time can be set to a fixed value, and the delay time can be determined on the basis of the communication delay (for example, RTT (Round Trip Time)) measured immediately before. Alternatively, the delay time can be given as a fixed value on the basis of the actual results up to this point. In the case where the delay time is given as a fixed value, the value of the delay time can be changed according to the time period, taking into account that the communication delay differs according to the time period.

[0074] The positional relationship between the vehicle 20 and the other vehicle 50 at a future time point further ahead than the current time point (time T12) is included in the future image displayed on the display device 32. In addition, the display device 32 displays what kind of assistance the vehicle 20 has requested. The operator 40 judges whether the vehicle 20 can make a right turn at the intersection based on the positional relationship between the vehicle 20 and the other vehicle 50 and the surrounding environment displayed on the display device 32 of the remote assistance terminal 30. The operator 40 operates the input device 33 of the remote assistance terminal 30 in accordance with the result of his or her own judgment. In Figure 5 In the example shown, the direction of travel of the other vehicle 50 cannot be determined from the image displayed on the display device 32 at time T12. Therefore, the operator 40 inputs a No-Go for execution of the right turn as the content of assistance to the vehicle 50.

[0075] The content of assistance input to the remote assistance terminal 30 is transmitted from the remote assistance terminal 30 to the vehicle 20. In addition, the future information, that is, information about the positional relationship between the vehicle 20 and the other vehicle 50 at a future time point, created by the remote assistance terminal 30 is transmitted to the vehicle 20 together with the content of assistance. Here, for simplicity, the time required for the judgment of the operator 40 is ignored, and the content of assistance and the future information are transmitted from the remote assistance terminal 30 to the vehicle 20 at time T12. In the remote assistance system, the judgment time of the operator 40 is not included in the delay time in the uplink direction nor in the delay time in the downlink direction, and therefore the judgment time of the operator 40 can be ignored for the study.

[0076] A time is required from the transmission of the content of assistance and the future information from the remote assistance terminal 30 until the content of assistance and the future information are received by the vehicle 20 and can be executed by the vehicle 20. This time is the delay time in the downlink direction. The time point in the future corresponding to the amount of the total delay time in the uplink direction and the downlink direction from the time point at which the vehicle information is transmitted by the vehicle 20 is time T21 at which the vehicle 20 can execute the content of assistance. As described above, the future image displayed on the display device 32 of the remote assistance terminal 30 is created taking the delay time into account. That is, the future image displayed on the display device 32 at time T12 is an image created by predicting the actual positional relationship between the vehicle 20 and the other vehicle 50 at time T21. Then, the positional relationship between the vehicle 20 and the other vehicle 50 at time T21 predicted at time T12 is transmitted to the vehicle 20 as the future information.

[0077] Since the assistance content performed by the vehicle 20 at the time T21 is No-Go, the vehicle 20 does not perform the right turn and continues the standby state. The future information received by the vehicle 20 along with the assistance content is used for the final decision of whether to perform the remote assistance corresponding to the assistance content in the case where the assistance content is Go, that is, in the case where the decision is to positively move the vehicle 20. Not limited to the scenario of the right turn at the intersection, in the case where the assistance content of the assistance request by the operator 40 is No-Go, the vehicle 20 continues the standby state regardless of the future information. However, in the case where the decision by the vehicle 20 coincides with the assistance content of the remote assistance, there is also a case where the future information is not used in the decision of whether to perform the remote assistance corresponding to the assistance content. In addition, the vehicle 20 transmits the latest vehicle information at the time T21 to the remote assistance terminal 30.

[0078] After the assistance request is issued at the time T0, the vehicle 20 continuously transmits the vehicle information to the remote assistance terminal 30. Therefore, at the time T11 during the period from when the assistance request is issued at the time T0 to when the assistance content is performed at the time T21, the vehicle 20 also transmits the vehicle information at the time T11 to the remote assistance terminal 30.

[0079] The computer 31 of the remote assistance terminal 30 generates a future image taking into account the delay time based on the vehicle information transmitted from the vehicle 20 at the time T11, and displays the generated future image on the display device 32 at the time T22. The future image displayed on the display device 32 at the time T22 is an image of the positional relationship of the vehicle 20 and the other vehicle 50 at a future time (time T31) further ahead than the current time point (time T22) predicted. According to the image displayed at the time T22, it can be judged that the other vehicle 50 is going to make a right turn at the intersection. If the other vehicle 50 makes a right turn at the intersection, the vehicle 20 can be made to safely make a right turn without causing interference with the other vehicle 50. Therefore, the operator 40 inputs the execution (Go) of the right turn as the assistance content to the vehicle 50.

[0080] Next, the vehicle information transmitted from the vehicle 20 is input to the remote assistance terminal 30. At the time T32, a future image generated based on the vehicle information transmitted from the vehicle 20 at the time T21 is displayed on the display device 32. The future image displayed on the display device 32 at the time T32 is an image of the positional relationship of the vehicle 20 and the other vehicle 50 at a future time (time T41) further ahead than the current time point (time T32) predicted. According to the image displayed at the time T32, it can be known that the other vehicle 50 is in the middle of a right turn at the intersection. If the other vehicle 50 is also in the middle of a right turn, the right turn of the vehicle 20 can be made without any problem directly. Therefore, the operator 40 inputs the execution (Go) of the right turn as the assistance content to the vehicle 50.

[0081] The assistance content input by the operator 40 to the remote assistance terminal 30 at time T22 is transmitted to the vehicle 20 together with the future information created by the remote assistance terminal 30 at time T22. The future information received by the vehicle 20 contains the positional relationship between the vehicle 20 and the other vehicle 50 at time T31 predicted at time T22. Since the judgment of Go transmitted from the operator 40 is the assistance content, the vehicle 20 refers to the future information to decide whether to perform the remote assistance corresponding to the assistance content.

[0082] In the judgment of whether to perform the remote assistance of the vehicle 20, it is confirmed whether the positional relationship between the vehicle 20 and the other vehicle 50 displayed on the display device 32 when the operator 40 inputs the assistance content to the remote assistance terminal 30 is realized in reality. As a method of confirmation, for example, the degree of matching of the positional relationship between the vehicle 20 and the other vehicle 50 in the future (hereinafter, referred to as the future positional relationship) displayed on the display device 32 at time T22 and the actual positional relationship between the vehicle 20 and the other vehicle 50 realized at time T31 is determined. The future positional relationship is contained in the future information transmitted from the remote assistance terminal 30. The actual positional relationship between the vehicle 20 and the other vehicle 50 can be acquired by the camera of the vehicle 20. The method of calculating the degree of matching will be described later in detail. If the degree of matching is equal to or more than a threshold value, the vehicle 20 judges that the future positional relationship predicted at time T22 is confirmed to be realized at time T31.

[0083] The vehicle 20 performs the remote assistance corresponding to the assistance content upon the above-mentioned confirmation. In the example shown in FIG. 10, the remote assistance corresponding to the judgment of Go of the assistance content is the execution of the right turn. At time T31, the automatic driving system of the vehicle 20 determines the path plan of the right turn at the intersection and executes the right turn. Note that the delay time in the downward direction further contains the time for confirming that the predicted future positional relationship is realized. Figure 5

[0084] Next, the assistance content and the future information transmitted from the remote assistance terminal 30 are input to the vehicle 20. The future information transmitted from the remote assistance terminal 30 at time T32 contains the positional relationship between the vehicle 20 and the other vehicle 50 at time T41 predicted at time T32. The assistance content transmitted from the remote assistance terminal 30 at time T32 is the judgment of Go, so the vehicle 20 determines the degree of matching of the future positional relationship predicted by the remote assistance terminal 30 at time T32 and the actual positional relationship between the vehicle 20 and the other vehicle 50 realized at time T41. In the example shown in FIG. 11, the degree of matching is equal to or more than a threshold value, so the vehicle 20 continues the right turn in accordance with the judgment of Go by the operator 40. Figure 5

[0085] ​​Next, the processing in the case where the remote assistance corresponding to the assistance content input by the operator 40 cannot be executed because the future positional relationship predicted by the remote assistance terminal 30 is not realized in reality will be described. In Figure 6 In the example shown, the future positional relationship predicted by the remote assistance terminal 30 at time T22 and the actual positional relationship of the vehicle 20 and the other vehicle 50 realized at time T31 do not match. In this case, the vehicle 20 does not follow the Go judgment of the assistance content sent from the remote assistance terminal 30 at time T22. That is, the vehicle 20 does not execute the right turn and continues the standby state. Further, at time T31, the vehicle 20 requests assistance from the operator 40 again. In this example, the assistance requested by the vehicle 20 again is the judgment of whether the right turn can be executed next.

[0086] Next, the assistance content and the future information sent from the remote assistance terminal 30 are input to the vehicle 20. In Figure 6 In the example shown, the future positional relationship predicted by the remote assistance terminal 30 at time T32 and the actual positional relationship of the vehicle 20 and the other vehicle 50 realized at time T41 do not match. Thus, the vehicle 20 does not follow the Go judgment of the operator 40 and continues the standby state next. By requesting assistance from the operator 40 again from the vehicle 20 at time T31, new assistance content and future information are input to the vehicle 20 from the remote assistance terminal 30 after time T41. Then, upon confirmation that the future positional relationship predicted by the remote assistance terminal 30 is realized in reality, the vehicle 20 executes the right turn according to the Go judgment of the operator 40.

[0087] In the case where the remote assistance corresponding to the assistance content input by the operator 40 cannot be executed because the future positional relationship predicted by the remote assistance terminal 30 is not realized in reality, in Figure 6 In the example shown, assistance is requested from the operator 40 again from the vehicle 20. However, depending on the situation in which the vehicle 20 is located, assistance cannot be requested from the operator 40 again. For example, in the situation where the color of the signal light in the traveling direction is to be changed from green to red, it is not possible to continue the standby state within the intersection.

[0088] In Figure 7 In the example shown, the future positional relationship predicted by the remote assistance terminal 30 at time T22 and the actual positional relationship of the vehicle 20 and the other vehicle 50 realized at time T31 do not match. In this case, the vehicle 20 does not follow the Go judgment of the assistance content sent from the remote assistance terminal 30 at time T22. However, it is also not a situation in which the standby state can be continued within the intersection while requesting assistance from the operator 40 again. In this case, the vehicle 20 selects and executes a route that can ensure safety. In Figure 7In the example shown, at time T31, the vehicle 20 abandons the right turn and executes the path that straight goes at the intersection.

[0089] Next, the assistance content and the future information transmitted from the remote assistance terminal 30 are input to the vehicle 20. In this case, the vehicle 20 has already selected and executed the path that can ensure safety, and thus the assistance content transmitted from the remote assistance terminal 30 is not executed. The vehicle 20 also continues to execute the path that can ensure safety at time T41. Figure 7 In the example shown, the assistance content and the future information transmitted from the remote assistance terminal 30 are input to the vehicle 20 at time T32. However, since the vehicle 20 has already selected and executed the path that can ensure safety, the assistance content transmitted from the remote assistance terminal 30 is not executed. The vehicle 20 also continues to execute the path that can ensure safety at time T41.

[0090] As described above, according to the remote assistance method of the present embodiment, the remote assistance corresponding to the assistance content input by the operator 40 is executed in the vehicle 20 on the condition that it is confirmed that the positional relationship between the vehicle 20 and the other vehicle 50 displayed on the display device 32 at the time when the assistance content is input by the operator 40 is achieved. That is, until the above-described confirmation is obtained, even if the vehicle 20 receives the assistance content, the remote assistance corresponding to the assistance content is not executed in the vehicle 20. Thereby, the vehicle 20 can be caused to act at the timing desired by the operator 40 without being affected by the deviation of the assistance timing caused by the delay time including the communication delay, and thus the reliability of the judgment of the operator 40 can be improved.

[0091] 3. Matching degree calculation method

[0092] 3-1. First example

[0093] Three examples are given as the matching degree calculation method. Figure 8 is a diagram for explaining the first example of the matching degree calculation method. In Figure 8 The vehicle 20 and the objects 50-1, 50-2,..., 50-N in the vicinity thereof are schematically depicted in FIG. 12. Although not shown, it is assumed that N objects exist in the vicinity of the vehicle 20. The objects in the vicinity include all moving bodies such as vehicles, pedestrians, bicycles, and the like. The positions of the vehicle 20 and the objects 50-1, 50-2,..., 50-N shown by the broken lines are the predicted positions referred to in the judgment of the assistance content by the operator 40, and the positions shown by the solid lines are the actual positions at the time when the vehicle 20 executes the assistance content.

[0094] In the first example, the matching degree is expressed using the following function. The following function is configured such that the smaller the error between the predicted position and the actual position is, the closer the value is to zero. For example, the matching degree calculation formula can be created in such a manner that the matching degree becomes the largest when the value of the following function is zero.

[0095] [Num. 1]

[0096] {(α e x De +β e ×|ΔV e |)+Σ i＝1～N (α a，i ×D a，i +β a，i ×|ΔV a，i |)} / (1+N)

[0097] In the function described above, D e This represents the change between the current position and the future position of vehicle 20. ΔV e This represents the change between the current speed of vehicle 20 and its predicted future speed based on the route plan. Additionally, D... a,i ΔV represents the change between the current position and the future position of object 50-i. a,i α represents the change between the current velocity of object 50-i and its future velocity predicted by the autonomous driving system. e β e α a,i and β a,i These are the coefficients. It should be noted that D, related to vehicle 20, can be omitted from the above function. e terms and ΔV e In this case, the denominator 1+N can be set to N.

[0098] Based on the judgment and context made by operator 40, the objects 50-1, 50-2, ..., 50-N that should be included in the calculation of the above function are determined. For example, in Figure 8 In the example shown, only objects 50-1 moving in front of vehicle 20 in its direction of travel can be included in the calculation. The calculation objects can also be limited to specific objects, such as moving objects on the road, vehicles in the opposite lane, vehicles approaching the vehicle, or simply vehicles. Furthermore, specific objects, such as following vehicles or pedestrians, can be excluded from the calculation objects.

[0099] exist Figures 5-7 In the example shown, there may be other objects besides other vehicles 50 around vehicle 20, but in the right turn at the intersection, only other vehicles 50 affect the judgment of vehicle 20's action. Therefore, in Figures 5-7 In the example shown, if the above function is used to calculate the matching degree, then only other vehicles 50 are considered in the calculation.

[0100] It should be noted that, based on adding all objects existing within a certain range starting from vehicle 20 to the calculation objects, the coefficient α can be changed according to the magnitude of its influence on the judgment of vehicle 20's actions. a,i and coefficient β a,iFor example, the coefficient assigned to an object in the direction approaching the vehicle 20 can be made larger than the coefficient assigned to an object in the direction away from the vehicle 20.

[0101] 3-2. Case 2

[0102] Case 2 is an example in which TTC (Time to Collision) is used for the determination of the matching degree. In Case 2, for an object interfering with the path of the vehicle 20, a predicted TTC in the future positional relationship shown on the display device 32 and an actual TTC in the actual positional relationship when the vehicle 20 executes the assist content are calculated. In the case where a plurality of objects exist on the path of the vehicle 20, the objects closest in the front and the rear are regarded as the objects interfering with the vehicle 20. Also, in the case where the error between the predicted TTC and the actual TTC is within a threshold value, it is determined that the matching degree is high. Note that, in Case 2, the relative distance or the relative speed can be used instead of the TTC.

[0103] Figure 9 is a diagram for a second example for explaining the method of calculating the matching degree in more detail. In Case 2, the threshold value for the matching degree determination in which the assist content is adopted is changed in the case where the vehicle interfering with the path of the vehicle 20 is a preceding vehicle 50F with respect to the vehicle 20 and in the case where it is a following vehicle 50R. The position shown by the broken line of the preceding vehicle 50F and the following vehicle 50R is the predicted position referred to by the operator 40 in the judgment of the assist content, and the position shown by the solid line is the actual position when the vehicle 20 executes the assist content. As shown in the diagram, the threshold value for the matching degree determination is increased with respect to the preceding vehicle 50F, and is decreased with respect to the following vehicle 50R. Figure 9 As shown in the diagram and the schematic diagram, the threshold value for the matching degree determination is increased with respect to the preceding vehicle 50F, and is decreased with respect to the following vehicle 50R. Note that the relationship between the error of the predicted TTC and the actual TTC and the matching degree can be set to be linear as shown in the diagram, or can be set to be non-linear. Even in the case where the relative distance or the relative speed is used instead of the TTC, as long as it is set that the smaller the error, the higher the matching degree, the relationship between the error and the matching degree can be set arbitrarily.

[0104] Here, the "object interfering with the path of the vehicle 20" which is the calculation target of the matching degree in Case 2 is explained in more detail. Figure 9 The preceding vehicle 50F and the following vehicle 50R shown in Figure 10 the schematic diagram shown in

[0105] In the case where an object crosses the path of vehicle 20, whether the object interferes with the path of vehicle 20 is determined by the actual position of the object when the auxiliary content is executed. For example, as shown by the dashed line, the predicted position referenced by operator 40 in the determination of auxiliary content interferes with the path, and as shown by the solid line, the actual position when executing auxiliary content also interferes with the path. In the second example, the object crossing the path, vehicle 50C1, is set as the object for calculating the matching degree. That is, the error between the predicted TTC and the actual TTC is calculated for vehicle 50C1. Figure 10 The chart shown calculates the matching degree based on the error. Furthermore, if the matching degree is above a threshold, vehicle 20 adopts and executes the auxiliary content. It should be noted that even when an object crosses the path of vehicle 20, relative distance or relative speed can be used instead of TTC.

[0106] On the other hand, in the second example, if the predicted position referenced by operator 40 in determining the auxiliary content interferes with the path, but the actual position when executing the auxiliary content is a vehicle 50C2 crossing the path, then in this case, vehicle 20 is excluded from the matching degree calculation. In this case, vehicle 20 does not interfere with vehicle 50C2, therefore vehicle 20 does not calculate the matching degree but adopts and executes the auxiliary content. On the other hand, in the second example, if the predicted position referenced by operator 40 in determining the auxiliary content interferes with the path, but the actual position when executing the auxiliary content is not yet interfering with the path, then vehicle 50C3 crossing the path is also excluded from the matching degree calculation. In this case, vehicle 20 does not calculate the matching degree and does not adopt the auxiliary content.

[0107] 3-3.Example 3

[0108] The third example demonstrates the use of Mahalanobis distance for determining the degree of matching. When vehicle 20 predicts the positions and velocities of surrounding objects and considers the uncertainty of these predictions to generate a path plan, it is appropriate to calculate the discrepancy between the predicted and measured values ​​based on this uncertainty. Mahalanobis distance can be used as the method for calculating this discrepancy. The Mahalanobis distance M for each object i surrounding vehicle 20 is given by the following formula.

[0109] [Number 2]

[0110]

[0111] Here, S i X is the covariance matrix representing the uncertainty of the predicted value of object i. → i It is D i With ΔV i The XOR expression gives the deviation between the predicted and measured values. Even if the deviation between the predicted and measured values ​​is X... → i In slightly larger cases, under Si In a case where the uncertainty of the given prediction value is sufficiently large and the uncertainty is incorporated into the path plan of the vehicle 20, it is also possible to expect that no big problem will occur in the operation of the automated driving system. It can be said that the definition of M(X → i ) is an index that reflects how much the uncertainty of the prediction value is incorporated into the matching degree of the path plan of the vehicle 20.

[0112] 4. Structure of Remote Assistance System

[0113] The structure of the remote assistance system of the embodiment of the present disclosure will be described. Figure 11 is a block diagram that shows the structure of the remote assistance system of the embodiment. The remote assistance system 100 is constituted by the vehicle 20 and the remote assistance terminal 30. However, the remote assistance system 100 can include the server 10 that relays the vehicle 20 and the remote assistance terminal 30, and can further include a communication network that connects them.

[0114] First, the functions that the remote assistance terminal 30 has will be described. The remote assistance terminal 30 has a vehicle information receiving section 301, a future information generating section 302, a remote assistance display section 303, a remote assistance operation section 304, and an instruction signal transmitting section 305. They are realized as the functions of the operator UI management system, that is, the remote assistance terminal 30 when a program 31c stored in a memory 31b of a computer 31 that physically constitutes the remote assistance terminal 30 is executed by a processor 31a.

[0115] The vehicle information receiving section 301 communicates with the vehicle 20 through the communication device 35. The vehicle information receiving section 301 acquires the assistance request and the vehicle information for remote assistance that are issued by the vehicle 20.

[0116] The future information generating section 302 generates future information related to the state of the vehicle 20 and the objects (vehicles, pedestrians, and the like) around the vehicle 20 at a future time that is more forward than the current time point, on the basis of the vehicle information transmitted from the vehicle 20. In the generation of the future information, specifically, the transmission time at which the vehicle 20 transmits the vehicle information, the path plan, the prediction information, and the recognition information included in the vehicle information, and the reception time at which the vehicle information receiving section 301 receives the vehicle information are used. The future time can be a future time corresponding to the amount of delay time of the total of the uplink direction and the downlink direction, or can be a future time corresponding to the amount of time set in advance. Note that the time used in the judgment by the operator 40 is not included in the delay time, but the average judgment time can be added as a buffer to the future time.

[0117] The remote assistance display section 303 displays the future information generated by the future information generation section 302 on the display device 32. The future information contains the positional relationship of the vehicle 20 and the objects around it at a future time point further ahead than the current time point, which is spatially displayed on the screen of the display device 32. With regard to the display on the display device 32, the operator 40 can arbitrarily select Figure 4 B such a display based on a 3D model and a display based on a 2D image. If it is a display based on a 3D model, in the case where it is necessary to grasp the objects around a specific place such as a pedestrian crossing, for example, the viewpoint on the three-dimensional space can be moved and rotated so that the operator 40 can easily observe around it. Also, in the case where there is a possibility that an occlusion exists around the place where assistance is requested, the viewpoint can also be moved and rotated to a place where an occlusion is likely to exist. If a display based on a 2D image is used, it is possible to adopt a method of increasing the resolution of the image, changing the quality and the number of pixels, enlarging, switching to a panoramic display, switching the display destination to another display device, and the like.

[0118] The remote assistance operation section 304 accepts the operation input by the operator 40 to the input device 33. The operator judges the assistance content for the vehicle 20 based on the future information displayed in the display device 32 by the remote assistance display section 303, and inputs the permission judgment of Go or No-Go or the details thereof to the input device 33. The remote assistance operation section 304 transmits the assistance content input to the input device 33 to the instruction signal transmission section 305.

[0119] The instruction signal transmission section 305 communicates with the vehicle 20 through the communication device 35. The instruction signal transmission section 305 transmits the future information generated by the future information generation section 302 and the assistance content judged by the operator 40 using the future information to the vehicle 20.

[0120] Next, the functions possessed by the vehicle 20 will be described. The vehicle 20 is provided with an instruction signal reception section 201, a vehicle instruction change section 202, an automatic driving system section 203, and a vehicle information transmission section 204. They are realized as the functions of the computer 21 when the program 21c stored in the memory 21b of the computer 21 is executed by the processor 21a.

[0121] The instruction signal reception section 201 communicates with the remote assistance terminal 30 through the communication device 25. The instruction signal reception section 201 receives the assistance content and the future information transmitted from the remote assistance terminal 30.

[0122] The vehicle instruction change unit 202 extracts information that needs to be changed, based on the assistance content and the future information obtained by the instruction signal receiving unit 201, and converts the signal information that the automatic driving system unit 203 can accept. Specifically, in a case where the assistance content obtained by the instruction signal receiving unit 201 is a judgment (Go) of permission of the action, the vehicle instruction change unit 202 compares the positional relationship between the vehicle 20 and the surrounding objects included in the future information and the positional relationship between the vehicle 20 and the surrounding objects at the time of actual execution of the assistance content. In a case where the result of the comparison is that the two do not match, that is, in a case where the degree of matching of the two is less than a threshold value, the vehicle instruction change unit 202 judges that the assistance content is not executable. If the degree of matching of the two is the threshold value or more, the vehicle instruction change unit 202 permits execution of the assistance content, and transmits the assistance content to the automatic driving system unit 203.

[0123] The automatic driving system unit 203 is a function of the automatic driving system of the computer 31. The automatic driving system unit 203, when traveling by the usual automatic driving, recognizes the surrounding objects based on the information related to the surrounding environment obtained by the external sensor 22, predicts the action of the recognized surrounding objects, and calculates the path in which the vehicle 20 advances. Also, when becoming or about to become a situation that is difficult to judge by only the automatic driving system unit 203, the automatic driving system unit 203 outputs a request for remote assistance. When accepting the remote assistance, the automatic driving system unit 203 performs a calculation for executing the assistance content transmitted from the vehicle instruction change unit 202.

[0124] The vehicle information transmitting unit 204 communicates with the remote assistance terminal 30 through the communication device 25. The vehicle information transmitting unit 204 transmits the assistance request and the vehicle information required for the remote assistance to the server 10 when the request for the remote assistance is made. After the operator 40 who is in charge of the assistance of the vehicle 20 is assigned by the server 10, the vehicle information is transmitted to the remote assistance terminal 30 of the operator 40 via the server 10 together with the assistance request.

[0125] 5. Process in the remote assistance system

[0126] The flow of the process in the remote assistance system 100 configured as described above is shown in FIG. 8. Figure 12 Figure 12 is a sequence chart showing the flow of the process of the remote assistance system 100 between the vehicle 20, the remote assistance terminal 30, and the operator 40.

[0127] ​First, the assistance request and the vehicle information are transmitted from the vehicle information transmitting section 204 of the vehicle 20. In detail, the assistance request is temporarily received by the server 10, and after the operator 40 in charge of the assistance of the vehicle 20 is assigned, it is transmitted to the remote assistance terminal 30 of the operator 40 together with the vehicle information. The assistance request and the vehicle information are received by the vehicle information receiving section 301 of the remote assistance terminal 30.

[0128] The future information generating section 302 of the remote assistance terminal 30 that has received the assistance request calculates a delay time for calculating the future information. However, in the case where a fixed value is used as the delay time, this processing is not needed.

[0129] The future information generating section 302 of the remote assistance terminal 30 generates the future information related to the states of the vehicle 20 and the surrounding objects at the future time that is more forward than the current time point, on the basis of the delay time and the vehicle information.

[0130] The remote assistance display section 303 of the remote assistance terminal 30 displays the future information generated by the future information generating section 302 on the display device 32. From the future information displayed on the display device 32, the operator 40 is able to grasp the positional relationship of the vehicle 20 and the surrounding objects at the future time to judge the assistance content for the assistance request.

[0131] The operator 40 inputs the assistance content judged into the input device 33. The remote assistance operation section 304 of the remote assistance terminal 30 accepts the assistance content input into the input device 33.

[0132] The instruction signal transmitting section 305 of the remote assistance terminal 30 transmits the assistance content accepted by the remote assistance operation section 304 to the vehicle 20 together with the future information generated by the future information generating section 302. The assistance content and the future information are received by the instruction signal receiving section 201 of the vehicle 20.

[0133] The vehicle instruction changing section 202 of the vehicle 20 performs confirmation that the positional relationship of the vehicle 20 and the surrounding objects displayed on the display device 32 at the time when the assistance content is input is realized, on the basis of the future information input from the remote assistance terminal 30.

[0134] Upon receiving the situation that the above-described confirmation is made, the autonomous driving system section 203 of the vehicle 20 executes the remote assistance corresponding to the assistance content. In other words, the remote assistance corresponding to the assistance content is not executed in the vehicle 20 until the above-described confirmation is made. By executing the above-described processing by the remote assistance system 100, it is possible to make the vehicle 20 act at the timing desired by the operator 40.

[0135] Next, the processing of the remote assistance terminal 30 and the processing of the vehicle 20 are described respectively.

[0136] Figure 13 FIG. 10 is a flowchart showing an example of the process of the remote assistance terminal 30. The program 31c stored in the memory 31b of the remote assistance terminal 30 causes the processor 31a to execute the series of processes shown in the flowchart, and causes the remote assistance terminal 30 to function as the operator UI management system.

[0137] Upon receiving the fact that the assistance request from the vehicle 20 has been received, step S101 is executed. In step S101, it is determined whether the calculated future time is valid. In the case where the calculated future time is valid, step S103 is executed. In the case where the calculated future time is not valid, step S102 is executed before step S103. In step S102, the future time is calculated based on the latest delay time.

[0138] The reasons for executing steps S101 and S102 are as follows. For example, assume that the moving average of the communication delay measured additionally is used for the calculation of the future time. In this case, if the communication is interrupted for the above time, the amount of time of the interruption is added to the moving average value of the communication delay, and thus the future time calculated after the reconnection becomes an invalid time different from the future time calculated based on the actual delay time. Or, since the communication state is different before the interruption and after the interruption, the delay time before the interruption cannot be referred to sometimes. Thus, when the communication is interrupted for the above time, the future time is calculated based on the latest delay time not affected thereby. Or, the future time is calculated based on the delay time newly calculated after the interruption.

[0139] In step S103, the future information of the future time is generated based on the vehicle information received from the vehicle 20. In step S104, the future information generated in step S103 is displayed on the display device 32.

[0140] In step S105, it is determined whether the assistance content is acquired from the operator 40. Steps S104 and S105 are repeated until the assistance content is acquired from the operator 40. Then, in the case where the assistance content is acquired from the operator 40, step S106 is executed. In step S106, the acquired assistance content is transmitted to the vehicle 20.

[0141] Figure 14 FIG. 11 is a flowchart showing an example of the process executed by the vehicle 20. The program 21c stored in the memory 21b of the vehicle 20 causes the processor 21a to execute the series of processes shown in the flowchart. Figure 14

[0142] ​Upon receiving that the assistance request has been generated, the vehicle 20 executes step S201. The assistance request is programmed to be generated in response to generation of a predefined event. In step S201, it is determined whether the prediction information that was just sent to the remote assistance terminal 30 before is valid.

[0143] One example in which the prediction information is not valid is that a certain time or more has elapsed since the prediction information that was sent last time. Another example in which the prediction information is not valid is that information about a predicted path of an object that interferes with the path of the vehicle 20 has changed significantly from the information that was sent last time. Changes in the information about the predicted path include, for example, a case in which the location of the interference approaches a certain distance or more, a case in which the acceleration or speed of the interfering object changes a certain value or more, a case in which the object that interferes closest to the vehicle 20 changes to a different object from the last time, and the like.

[0144] In a case in which the prediction information that was just sent before is valid, step S203 is executed. In a case in which the prediction information that was just sent before is not valid, step S203 is executed after step S202 is executed. In step S202, new prediction information is calculated. In step S203, the valid prediction information is sent to the remote assistance terminal 30 together with the path plan as vehicle information.

[0145] In step S204, it is determined whether assistance content is received from the remote assistance terminal 30. Step S204 is repeatedly executed until the assistance content is received. In a case in which the assistance content is received, steps S205 and S206 are executed in parallel. In step S205, the received assistance content and future information that is received together with the assistance content are temporarily saved. In step S206, a path that can ensure safety in the current situation of the vehicle 20 is calculated and temporarily saved.

[0146] Next, in step S207, a degree of matching between the future information that is temporarily saved in step S205 and the surrounding information at the time of execution of the assistance content is calculated. The future information includes a future positional relationship between the vehicle 20 and the surrounding objects that is displayed on the display device 32 at the time when the assistance content is input by the operator 40. The surrounding information includes an actual positional relationship between the vehicle 20 and the surrounding objects that is acquired by the external sensor 22. In more detail, the degree of matching that is calculated in step S207 is a degree of matching between the future positional relationship and the actual positional relationship.

[0147] In step S208, it is determined whether the degree of matching that is calculated in step S207 is within an allowable range, that is, whether the degree of matching is a threshold value or more. If the degree of matching is the threshold value or more, step S209 is executed. In step S209, the assistance content that is temporarily saved in step S205 is executed.

[0148] In a case where the degree of matching is less than the threshold value, execution of the assistance content temporarily saved in step S205 is abandoned, and step S210 is executed instead. In step S210, it is determined whether or not the assistance can be requested again from the operator 40. In a case where the assistance can be requested again from the operator 40, the processing returns to the initial processing to generate the assistance request. In a case where the assistance cannot be requested again from the operator 40, step S211 is executed. In step S211, the safe-ensuring path temporarily saved in step S206 is executed.

[0149] Figure 15 is a flowchart showing another example of the processing of the vehicle 20. In a case where this example is employed, the program 21c stored in the memory 21b of the vehicle 20 causes the processor 21a to execute the series of processing shown in the flowchart of Figure 15 . Hereinafter, regarding the processing common to the flowchart of Figure 14 , the explanation will be omitted or simplified, and regarding the processing peculiar to the flowchart of Figure 15 , the explanation will be focused on.

[0150] According to the flowchart of Figure 15 , step S221 is executed next to step S205 and step S206. In step S221, the time at which the assistance content can be executed in the vehicle 20 is compared with the future time of the display device 32 at which the assistance content was input. Then, it is determined whether or not the executable time of the assistance content exceeds the future time.

[0151] In a case where the executable time of the assistance content does not exceed the future time, it can be judged that the positional relationship of the vehicle 20 and the surrounding object displayed on the display device 32 is realized or will be realized in the future. Thus, in a case where the determination result of step S221 is affirmative, step S209 is executed after the vehicle 20 is put on standby until the future time in step S222. In a case where the executable time of the assistance content coincides with the future time, the step S209 is executed immediately without being put on standby.

[0152] On the other hand, in a case where the time at which the remote assistance can be executed in the vehicle 20 exceeds the future time, it can be judged that the positional relationship of the vehicle 20 and the surrounding object displayed on the display device 32 has already been unable to be realized. Thus, in a case where the determination result of step S221 is negative, step S210 is executed instead of step S209.

[0153] However, in a case where the determination result of step S221 is negative, step S223 can be executed before step S210. In step S223, the degree of matching of the future information temporarily saved in step S205 and the surrounding information at the time of execution of the assistance content is calculated, and it is determined whether or not the degree of matching is equal to or greater than the threshold value.

[0154] If the result of the determination is that the degree of matching is above the threshold value, step S209 can also be executed. That is, even in the case where the future time has passed, as long as the positional relationship of the vehicle 20 and the surrounding object displayed on the display device 32 is realized in reality, the auxiliary content temporarily saved in step S205 can also be executed. If the degree of matching is lower than the threshold value, step S209 is not executed and step S210 is executed.

[0155] In the case where the above-described example is adopted as the process of the vehicle 20, the delay time for the calculation of the future time is preferably set to a time longer than the assumed actual delay time. Thereby, the vehicle 20 can wait until the future time after receiving the auxiliary content, and thereafter reliably execute the auxiliary content.

Claims

1. A remote assistance method of performing remote assistance on a vehicle having an autonomous travel function, characterized by, The remote assistance method includes: spatially displaying, on a display device, a positional relationship between the vehicle and the object at a future time point further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle and information about the object existing around the vehicle; sending assistance content input from a remote operator to the vehicle; and after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device when the assistance content is input, and performing remote assistance corresponding to the assistance content in the vehicle.

2. The remote assistance method according to claim 1, wherein the confirmation includes confirming that a degree of matching between the positional relationship between the vehicle and the object displayed on the display device when the assistance content is input and an actual positional relationship between the vehicle and the object acquired by a sensor of the vehicle is within an allowable range.

3. The remote assistance method according to claim 2, wherein the information about the object is information acquired by the sensor, and at least a part of the information about the object acquired by the sensor is used for creation of the path plan.

4. The remote assistance method according to claim 2 or 3, wherein the confirmation includes confirming that a time at which remote assistance corresponding to the assistance content can be performed in the vehicle does not exceed the future time point of the positional relationship between the vehicle and the object displayed on the display device when the assistance content is input.

5. A remote assistance system that performs remote assistance for a vehicle having an autonomous travel function, the remote assistance system comprising: at least one memory including at least one program; and at least one processor in combination with the at least one memory, the at least one program configured to cause the at least one processor to perform: spatially displaying, on a display device, a positional relationship between the vehicle and the object at a future time point further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle and information about the object existing around the vehicle; sending assistance content input from a remote operator to the vehicle; and after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device when the assistance content is input, and performing remote assistance corresponding to the assistance content in the vehicle.

6. The remote assistance system according to claim 5, wherein the confirmation includes confirming that a degree of matching between the positional relationship between the vehicle and the object displayed on the display device when the assistance content is input and an actual positional relationship between the vehicle and the object acquired by a sensor of the vehicle is within an allowable range.

7. The remote assistance system according to claim 6, wherein the information about the object is information acquired by the sensor, and at least a part of the information about the object acquired by the sensor is used for creation of the path plan. The information related to the object is information acquired by the sensor, and at least a part of the information related to the object acquired by the sensor is used for creation of the path plan.

8. The remote assistance system according to claim 6 or 7, characterized in that, the confirmation includes confirmation that a time at which remote assistance corresponding to the assistance content can be executed in the vehicle does not exceed a future time of the positional relationship between the vehicle and the object displayed on the display device at the time when the assistance content is input.

9. A computer-readable recording medium recording at least one program causing a computer to execute a process, characterized by, the process includes: spatially displaying, on a display device, a positional relationship between a vehicle and an object existing in the surroundings of the vehicle at a future time further ahead than a current time point, which is predicted based on a path plan for autonomous travel created by the vehicle having an autonomous travel function and information related to the object; transmitting assistance content input from a remote operator to the vehicle; and after the vehicle receives the assistance content, accepting confirmation of the positional relationship between the vehicle and the object displayed on the display device at the time when the assistance content is input, and executing remote assistance corresponding to the assistance content in the vehicle.

Images