Remote assistance system and remote assistance method

Abstract

A remote assistance system and a remote assistance method that can minimize the constrained time of a remote operator who performs remote assistance for an autonomous driving vehicle. When it is determined that a remote assistance request situation in which it is difficult to continue autonomous driving occurs due to a vehicle in autonomous driving approaching one or more avoidance target vehicles within a predetermined area, the remote assistance system entrusts a remote operator to perform remote assistance for passing through the predetermined area. At this time, the remote assistance system displays information on one or more avoidance target vehicles on the display device of the remote operator, receives information on a starting reference vehicle that is the reference for allowing a start specified by the remote operator from the information on the one or more displayed avoidance target vehicles, determines whether the starting reference vehicle has passed through the predetermined area, and when it is determined that the starting reference vehicle has passed through the predetermined area, causes the vehicle to pass through the predetermined area by autonomous driving.

Description

Technical Field

[0001] The present disclosure relates to a remote assistance system and a remote assistance method for sending a remote assistance request of a vehicle to a remote operator. Background Art

[0002] Patent Document 1 discloses a technology related to a remote operation system for reducing the burden on an operator. The remote operation system of this technology includes: a vehicle that makes a remote operation request to a control center; and a control center provided with a remote operation device that remotely operates the vehicle that has made the remote operation request. The vehicle makes a remote operation request to the control center when entering a remote operation target area. Moreover, when the vehicle that is the remote operation target leaves the remote operation target area where it is currently traveling, a remote operation end notification is sent to the vehicle.

[0003] Prior Art Documents

[0004] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-147626 Summary of the Invention

[0005] Problems to be Solved by the Invention

[0006] In the technology of Patent Document 1 described above, the remote operation continues until the vehicle that is the remote operation target leaves the remote operation target area where it is currently traveling. In this case, the constrained time of the remote operator may become longer.

[0007] The present disclosure has been made in view of the above problems, and an object thereof is to provide a remote assistance system and a remote assistance method that can minimize the constrained time of a remote operator who performs remote assistance for an autonomous driving vehicle.

[0008] Means for Solving the Problems

[0009] To solve the above problems, the present disclosure is applied to a remote assistance system. When the remote assistance system determines that it has become a remote assistance request situation where it is difficult to continue autonomous driving due to a vehicle in autonomous driving approaching one or more avoidance target vehicles within a predetermined area, it entrusts a remote operator to perform remote assistance for passing through the predetermined area. The remote assistance system includes at least one storage device storing at least one program, and at least one processor coupled to the at least one storage device. When it is determined that it has become a remote assistance request situation, the at least one processor is configured to perform the following processes by executing the at least one program: a display process of displaying information on one or more avoidance target vehicles on a display device of the remote operator; a reception process of receiving information on a start reference vehicle that is a reference for allowing a start designated by the remote operator from the information on the one or more displayed avoidance target vehicles; a determination process of determining whether the start reference vehicle has passed through the predetermined area; and a vehicle control process of, when it is determined that the start reference vehicle has passed through the predetermined area, causing the vehicle to pass through the predetermined area by autonomous driving.

[0010] In the present disclosure, the vehicle control process may also be configured to cause the vehicle to wait at a predetermined waiting position by autonomous driving until it is determined that the start reference vehicle has passed through the predetermined area.

[0011] In the present disclosure, the display process may also be configured to display each of the one or more avoidance target vehicles together with identification information on an overhead view, and the reception process may also be configured to receive a selection of the identification information by the remote operator.

[0012] In the present disclosure, the display device may also be a touch panel display. The display process may also be configured to display an overhead view including each of the one or more avoidance target vehicles on the touch panel display, and the reception process may also be configured to receive a designation of the start reference vehicle from the touch panel display.

[0013] In the present disclosure, the display process may also be configured to display a confirmation message designating any one of the one or more avoidance target vehicles as the start reference vehicle. Further, the reception process may also be configured to receive whether the confirmation message is correct or not.

[0014] In the present disclosure, when it is determined that it has become a remote assistance request situation, the at least one processor may also be configured to calculate a predicted passing order of one or more avoidance target vehicles passing through the predetermined area by executing the at least one program, and may also be configured to cancel the received start reference vehicle when the predicted passing order has changed after receiving the start reference vehicle through the reception process.

[0015] In addition, the present disclosure is applied to a remote assistance method. In a case where it is determined that a remote assistance request situation in which it becomes difficult to continue autonomous driving due to a vehicle in autonomous driving approaching one or more avoidance target vehicles in a predetermined area occurs, a remote operator is entrusted to perform remote assistance for passing through the predetermined area. The remote assistance method includes: in a case where it is determined that the remote assistance request situation has occurred, displaying information on one or more avoidance target vehicles on a display device of the remote operator, receiving information on a starting reference vehicle that is designated as a reference for allowing a start from the information on the one or more displayed avoidance target vehicles by the remote operator, determining whether the starting reference vehicle has passed through the predetermined area, and in a case where it is determined that the starting reference vehicle has passed through the predetermined area, causing the vehicle to pass through the predetermined area by autonomous driving.

[0016] Advantages of the Invention

[0017] According to the present disclosure, in a case where it is determined that the remote assistance request situation has occurred, the remote operator designates a starting reference vehicle that is a reference for allowing a start from the information on one or more avoidance target vehicles displayed on the display device. With this configuration, it is not necessary to restrict the remote operator until the vehicle passes through the predetermined area, and thus the restricted time of the remote operator can be shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a block diagram showing a configuration example for explaining an outline of the remote assistance system according to Embodiment 1.

[0019] Figure 2 It is a block diagram showing an example of the configuration of an autonomous driving vehicle.

[0020] Figure 3 It is a diagram showing an example of a situation in which remote assistance is performed in the remote assistance system.

[0021] Figure 4 It is a diagram showing an example of display information displayed on a display device of a remote operator interface.

[0022] Figure 5 It is a diagram showing an example of an input device of a remote operator interface.

[0023] Figure 6 It is a functional block diagram showing a part of the functions of the autonomous driving control device according to Embodiment 1.

[0024] Figure 7 It is a flowchart of a process executed in the autonomous driving control device according to Embodiment 1.

[0025] Figure 8It is a flowchart showing a routine for determining the necessity of remote assistance in the remote assistance determination unit.

[0026] Figure 9 It is a diagram showing the first application example of the remote assistance system.

[0027] Figure 10 It is a diagram showing the second application example of the remote assistance system.

[0028] Figure 11 It is a diagram showing the third application example of the remote assistance system.

[0029] Figure 12 It is a diagram showing the fourth application example of the remote assistance system.

[0030] Figure 13 It is a diagram showing a modified example of the display mode of the display information INF1.

[0031] Figure 14 It is a diagram for explaining the features of the remote assistance system in Embodiment 2.

[0032] Figure 15 It is a diagram for explaining the features of the remote assistance system in Embodiment 2.

[0033] Figure 16 It is a diagram for explaining the features of the remote assistance system in Embodiment 2.

[0034] Figure 17 It is a functional block diagram showing a part of the functions of the automatic driving control device in Embodiment 2.

[0035] Figure 18 It is a flowchart of the process executed in the automatic driving control device in Embodiment 2.

[0036] Reference numeral description

[0037] 2 Remote assistance device; 4 Remote server; 6 Remote operator interface; 10 Autonomous vehicle (vehicle); 30 Information acquisition device; 31 Vehicle position sensor; 32 Surrounding condition sensor; 33 Vehicle state sensor; 40 Autonomous driving control device; 42 Processor; 44 Storage device; 46 Input / output interface; 50 Communication device; 60 Driving device; 62 Display device; 64 Input device; 100 Remote assistance system; 402 Surrounding environment information acquisition unit; 404 Vehicle motion information acquisition unit; 406 Map information acquisition unit; 410 Identification unit; 412 Target (object) detection unit; 414 Target tracking unit; 420 Driving plan unit; 421 First driving plan generation unit; 422 Remote assistance determination unit; 423 Target passing determination unit; 424 Second driving plan generation unit; 425 Target passing order change determination unit; 430 Vehicle control unit; 440 Program; 442 Various information; 444 Map database. Detailed implementation mode

[0038] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, when numbers such as the number, quantity, amount, range, etc. of each element are mentioned in the following embodiments, unless otherwise specifically stated or clearly determined to be that number in principle, the present invention is not limited to the numbers mentioned above. In addition, regarding the structures, steps, etc. described in the following embodiments, unless otherwise specifically stated or clearly determined to be that structure, step, etc. in principle, they are not necessarily essential for the present invention.

[0039] Embodiment 1.

[0040] 1-1. Overall configuration of the remote assistance system in Embodiment 1

[0041] First, the schematic configuration of the remote assistance system in Embodiment 1 will be described. Figure 1 It is a block diagram showing a configuration example for explaining the outline of the remote assistance system in Embodiment 1. Figure 1 The shown remote assistance system 100 is a system for performing remote assistance for assisting the driving of the autonomous vehicle 10. Hereinafter, the autonomous vehicle 10 utilized by the remote assistance system 100 will also be simply denoted as "vehicle 10".

[0042] Remote assistance assists in the recognition or judgment of a part of the surrounding environment of the vehicle 10. The remote assistance is performed by a remote operator on standby remotely. The number of remote operators utilized by the remote assistance system 100 is not limited. In addition, the number of vehicles 10 utilized by the remote assistance system 100 is also not limited.

[0043] As Figure 1As shown, the remote assistance system 100 includes a vehicle 10 and a remote assistance device 2. The remote assistance device 2 includes a remote server 4 and a remote operator interface 6 for input and output of remote assistance by a remote operator. The remote server 4 is communicably connected to the vehicle 10 via a communication network N. The remote server 4 receives various information sent from the vehicle 10. The remote operator interface 6 includes a display device 62 for displaying the information sent from the vehicle 10 during driving assistance and an input device 64 for inputting information. The display device 62 and the input device 64 may be, for example, a touch panel display having a display function and an input function. In addition, regarding the configuration of the remote assistance device 2, a known technique can be adopted, so the detailed description here is omitted.

[0044] In the remote assistance system 100, when the vehicle 10 is about to pass through a predetermined area determined to require remote assistance, a remote assistance request is sent to the remote assistance device 2. The remote assistance here is a judgment on whether it is possible to pass through the predetermined area. Passing through the predetermined area is, for example, making a right turn at an intersection. In the remote assistance system 100, according to the remote assistance request sent from the vehicle 10, the remote operator performs remote assistance via the remote assistance device 2. Typically, the remote operator inputs passing judgment information to the remote operator interface 6. The remote server 4 sends the passing judgment information to the vehicle 10 via the communication network N. The vehicle 10 performs autonomous driving based on the passing judgment information sent from the remote assistance device 2 and passes through the predetermined area. In addition, the content of the passing judgment information input by the remote operator of the remote assistance system 100 in this embodiment has features. The details of the passing judgment information will be described later.

[0045] 1-2. Configuration of the Autonomous Driving Vehicle in Embodiment 1

[0046] Next, an example of the configuration related to autonomous driving of the autonomous driving vehicle 10 to which the remote assistance system 100 in Embodiment 1 is applied will be described. Figure 2 It is a block diagram showing an example of the configuration of the autonomous driving vehicle 10. The vehicle 10 is an autonomous driving vehicle capable of autonomous driving. As the autonomous driving here, autonomous driving at level 3 or above in the level definition of SAE (Society of Automotive Engineers) is assumed. In addition, the power source of the vehicle 10 is not limited.

[0047] The vehicle 10 is equipped with an autonomous driving control device 40. The autonomous driving control device 40 has a function of performing autonomous driving of the vehicle 10 according to the remote assistance information sent from the remote operator. An information acquisition device 30, a communication device 50, and a driving device 60 are connected to the autonomous driving control device 40.

[0048] The information acquisition device 30 is configured to include a vehicle position sensor 31, a surrounding condition sensor 32, and a vehicle state sensor 33.

[0049] The vehicle position sensor 31 detects the position and orientation of the vehicle 10. For example, the vehicle position sensor 31 includes a GPS (Global Positioning System) sensor. The GPS sensor receives signals transmitted from multiple GPS satellites and calculates the position and orientation of the vehicle 10 based on the received signals. The vehicle position sensor 31 may also perform well-known self-position estimation processing (localization) to improve the accuracy of the current position of the vehicle 10. The information detected by the vehicle position sensor 31 is transmitted to the autonomous driving control device 40 at any time as part of the surrounding environment information.

[0050] The surrounding condition sensor 32 identifies the surrounding information of the vehicle 10. For example, as the surrounding condition sensor 32, a camera (imaging device), a lidar (Laser Imaging Detection and Ranging), a radar, etc. may be exemplified. The surrounding information includes target information identified by the surrounding condition sensor 32. As targets, surrounding vehicles, pedestrians, roadside objects, obstacles, white lines, traffic lights, etc. may be exemplified. The target information includes the relative position and relative speed of the target with respect to the vehicle 10. The information identified by the surrounding condition sensor 32 is transmitted to the autonomous driving control device 40 at any time as part of the surrounding environment information.

[0051] The vehicle state sensor 33 detects vehicle information indicating the state of the vehicle 10. As the vehicle state sensor 33, a vehicle speed sensor, a lateral acceleration sensor, a yaw rate sensor, etc. may be exemplified. The information detected by the vehicle state sensor 33 is transmitted to the autonomous driving control device 40 at any time as part of the vehicle motion information.

[0052] The communication device 50 communicates between the vehicle and the outside. For example, the communication device 50 transmits and receives various information to and from the remote assistance device 2 via the communication network N. In addition, the communication device 50 communicates with external devices such as roadside equipment, surrounding vehicles, and surrounding infrastructure. The roadside equipment is, for example, a beacon device that transmits congestion information, traffic information divided by lane, restriction information such as temporary stop, and traffic condition information at blind spots. In addition, when the external device is a surrounding vehicle, the communication device 50 performs vehicle-to-vehicle communication (V2V communication) with the surrounding vehicle. Furthermore, when the external device is surrounding infrastructure, the communication device 50 performs vehicle-to-infrastructure communication (V2I communication) with the surrounding infrastructure.

[0053] The traveling device 60 includes a steering device, a driving device, and a braking device. The steering device steers the wheels of the vehicle 10. The driving device is a driving source that generates the driving force of the vehicle 10. Examples of the driving device include an engine or an electric motor. The braking device generates a braking force for the vehicle 10. The traveling device 60 controls the traveling of the vehicle 10 based on traveling control amounts related to the steering, acceleration, and deceleration of the vehicle 10.

[0054] The automatic driving control device 40 is an information processing device that performs various processes in automatic driving and remote automatic driving. Typically, the automatic driving control device 40 is a microcomputer including at least one processor 42, at least one storage device 44, and at least one input / output interface 46. The automatic driving control device 40 is also referred to as an ECU (Electronic Control Unit, electronic control unit).

[0055] Various information 442 is stored in the storage device 44. For example, the various information 442 includes the above-mentioned surrounding environment information and vehicle motion information. Examples of the storage device 44 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), etc.

[0056] A map database 444 is stored in the storage device 44. The map database 444 is a database that stores map information. The map information includes traffic environment information representing the traffic environment, such as the position information of roads, the information of road shapes, the number of lanes, the lane width, the position information of intersections and bifurcations, and the priority of roads. In addition, the map database 444 may also be stored in a server such as the remote server 4 of the remote assistance device 2 that can communicate with the vehicle 10.

[0057] The processor 42 includes a CPU (Central Processing Unit). The processor 42 is combined with the storage device 44 and the input / output interface 46. At least one program 440 related to automatic driving is stored in the storage device 44. The processor 42 reads and executes the program 440 stored in the storage device 44, thereby implementing various functions of the automatic driving control device 40.

[0058] The input / output interface 46 is an interface for exchanging information with the remote assistance device 2. Various information generated in the automatic driving control device 40 and a remote assistance request described later are output to the remote assistance device 2 via the input / output interface 46.

[0059] 1-3. Features of the remote assistance system of Embodiment 1

[0060] Next, an example of a situation in which a remote operator performs remote assistance in the remote assistance system 100 of Embodiment 1 will be described. Figure 3This is a diagram showing an example of the situation where remote assistance is performed in a remote assistance system.

[0061] In the remote assistance system 100, when it is determined that the vehicle 10 should make a remote assistance request to the remote assistance device 2, that is, in the remote assistance request situation, a remote assistance request is made to the remote assistance device 2. Such a remote assistance request situation is, for example, a situation where the vehicle 10 is about to enter a "position where the own vehicle is not prioritized" or "a position where the priority of the own vehicle is the same as that of other vehicles". The remote assistance request situation can also be preset, for example, by including situation elements that can be determined based on map information. Typically, the remote assistance request situation is a situation where the vehicle 10 passes through a predetermined area that can be determined based on map information along with specific situation elements. As such a remote assistance request situation, for example, the situation where the vehicle 10 turns right at a specific intersection, the situation where the vehicle 10 goes straight or turns left after temporarily stopping at a specific intersection, the situation where the vehicle 10 starts to change lanes, the situation where the vehicle 10 crosses a lane to pass by in order to avoid a parked vehicle, etc. can be exemplified.

[0062] Figure 3 The shown traffic environment is an intersection I1 where lane L1 intersects with lane L2. Lane L2 is a priority lane relative to lane L1. The vehicle 10 is traveling on lane L1 towards the intersection I1. The vehicle V1 is traveling on lane L1 towards the intersection I1 from the opposite direction of the vehicle 10. The vehicle V2 is traveling on lane L2 towards the intersection I1 from the left side of the vehicle 10.

[0063] The automatic driving control device 40 of the vehicle 10 generates a driving plan to turn right at the intersection from lane L1 and travel to lane L2. In this case, when the vehicle 10 approaches the intersection I1, the automatic driving control device 40 recognizes the situation that the vehicle 10 is about to enter a non-priority lane, and determines that it is in the remote assistance request situation. The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2, and also sends the display information INF1 described later.

[0064] The remote assistance device 2 that has received the remote assistance request REQ performs remote assistance implemented by a remote operator. Typically, the remote assistance device 2 displays the received display information INF1 on the display device 62. Hereinafter, this process will be referred to as "display process". The remote operator refers to the display information INF1 displayed on the display device 62 and makes a determination on passing through the intersection I1. Moreover, the remote operator issues a remote instruction to the vehicle 10 by operating the remote operator interface 6.

[0065] Here, for example, in the case where the remote instruction of the remote assistance system includes the operation amount of the traveling device 60, from the time point when the remote assistance request REQ is received, the remote operator is restricted until the vehicle V1 or V2 passes through the intersection I1 and then until the time point when the vehicle 10 actually turns right and passes through the intersection I1.

[0066] In addition, in the case where the remote instruction of the remote assistance system includes a remote instruction of the start timing of an action such as "start turning right", from the time point when the remote assistance request REQ is received, the remote operator is restricted until the vehicle V1 or V2 passes through the intersection I1 and then until the time point when the start instruction of the action is issued. Thus, in the configuration where the remote operator directly remotely instructs the start timing of the traveling of the vehicle 10, there is a problem that the remote operator will be restricted for a long time.

[0067] Then, the remote assistance system 100 of the first embodiment receives from the remote operator the designation of the vehicle that is to be the object of passing judgment with respect to the vehicle 10. Hereinafter, this process will be referred to as the "reception process". Figure 4 It is a diagram showing an example of the display information of the display device displayed on the remote operator interface. In this diagram, on the bird's-eye view of the intersection I1, the vehicle 10, other vehicles with "A" as the identification information attached, and other vehicles with "B" as the identification information attached are shown. In addition, the target path R1 of the vehicle 10 at the intersection I1 is also shown in the bird's-eye view.

[0068] In the reception process, the remote operator refers to the bird's-eye view displayed on the display device 62 and designates the preceding passing vehicle that should be used as the reference for allowing the start when the vehicle 10 turns right at the intersection I1 according to the target path R1. In the following description, this vehicle will be referred to as the "start reference vehicle". Figure 5 It is a diagram showing an example of the input device of the remote operator interface. In the example shown in this diagram, the input device 64 is provided with buttons corresponding to the marks that are the identification information attached to other vehicles displayed on the display device 62. The remote operator presses the button with the mark of the vehicle that should be the start reference. In Figure 4 the example shown, not only the vehicle "B" traveling toward the intersection I1 in the oncoming lane of the lane L1, but also the vehicle "A" traveling from the left side of the vehicle 10 toward the intersection I1 in the lane L2 needs to pass. Therefore, the remote operator presses the button of "A". Hereinafter, the information of the start reference vehicle designated by the remote operator will be referred to as "start reference information INF2". The start reference information INF2 is sent to the vehicle 10 as a remote instruction. The vehicle 10 starts and turns right at the intersection I1 based on the situation that the start reference vehicle included in the received remote instruction has passed through the intersection I1.

[0069] According to this operation, the remote operator can complete remote assistance without waiting for the starting reference vehicle to pass through the intersection. Thus, compared with the case of indicating the starting timing, the constrained time of the operator can be shortened.

[0070] Hereinafter, the functional structure and specific processing of the automatic driving control device 40 of the remote assistance system 100 according to Embodiment 1 will be described.

[0071] 1-4. Functional Structure of Automatic Driving Control Device

[0072] Next, an example of the functional structure of the automatic driving control device 40 will be described. Figure 6 It is a functional block diagram showing a part of the functions of the automatic driving control device 40. The automatic driving control device 40 includes a surrounding environment information acquisition unit 402, a vehicle motion information acquisition unit 404, a map information acquisition unit 406, an identification unit 410, a travel plan unit 420, and a vehicle control unit 430.

[0073] The surrounding environment information acquisition unit 402 and the vehicle motion information acquisition unit 404 are functional blocks for respectively acquiring the surrounding environment information and vehicle motion information detected by the information acquisition device 30. The map information acquisition unit 406 is a functional block for acquiring the map information stored in the map database 444.

[0074] The identification unit 410 is a functional block for identifying targets such as vehicles and pedestrians existing around the vehicle 10. The identification unit 410 includes a target detection unit 412 and a target tracking unit 414. The target detection unit 412 uses the surrounding environment information and vehicle motion information acquired by the surrounding environment information acquisition unit 402 to detect targets around the vehicle 10. The target detection method here is not limited.

[0075] The target tracking unit 414 correlates the previous detection result of the target detection unit 412 with the current detection result. Typically, when the target in the current detection result is the same as the previous detection result, the target tracking unit 414 assigns the same target ID as the previous detection result. In addition, when the target in the current detection result is different from the previous detection result, the target tracking unit 414 assigns a new target ID to the target in the current detection result. The assigned target ID is sent to the travel plan unit 420 together with information such as position and speed as target ID information.

[0076] The travel plan unit 420 is a functional block for generating a travel plan for the automatic driving of the vehicle 10. The travel plan unit 420 includes a first travel plan generation unit 421, a remote assistance determination unit 422, a target passing determination unit 423, and a second travel plan generation unit 424.

[0077] The first travel plan generation unit 421 generates a travel trajectory (path) TR1 for the vehicle 10 to travel according to the map, using the map information obtained by the map information acquisition unit 406 and the vehicle motion information such as the current position and vehicle speed of the vehicle 10 obtained by the vehicle motion information acquisition unit 404. The generation method of the travel trajectory TR1 is not limited.

[0078] The remote assistance determination unit 422 determines whether remote assistance by a remote operator is required. The remote assistance here is for the vehicle 10 to determine passing through a predetermined area. Typically, the remote assistance determination unit 422 uses the target ID information sent from the target tracking unit 414 to generate a predicted trajectory TR2 of one or more avoidance target vehicles that may collide with the vehicle 10.

[0079] The remote assistance determination unit 422 calculates the predicted collision position CP where the predicted trajectory TR2 intersects with the travel trajectory TR1. Based on the predicted collision position CP, the remote assistance determination unit 422 determines whether the vehicle 10 may collide with the avoidance target vehicle. In the case where there is no predicted collision position CP, the remote assistance determination unit 422 determines that no remote assistance request is required. In addition, in the case where there is a predicted collision position CP, the remote assistance determination unit 422 uses the map information or the surrounding environment information to determine the traffic priority status at the predicted collision position CP. Moreover, in the case where the traffic priority status at the predicted collision position CP is a status where the vehicle 10 has priority, the remote assistance determination unit 422 determines that no remote assistance request is required. In addition, in the case where the traffic priority status at the predicted collision position CP is a status where the vehicle 10 does not have priority, the remote assistance determination unit 422 determines that a remote assistance request is required.

[0080] In the case where it is determined that a remote assistance request is required, the remote assistance determination unit 422 sends the remote assistance request REQ and the display information INF1 to the remote assistance device 2. The display information INF1 is information required for the remote operator to give a remote instruction. For example, it is information in which the vehicle 10, the travel trajectory TR1 of the vehicle 10, and the information of the avoidance target vehicle are configured on an overhead view.

[0081] Upon receiving the remote assistance request REQ, the remote operator refers to the display information INF1 displayed on the display device 62 and designates a starting reference vehicle from the input device 64. The remote assistance device 2 sends the target ID information corresponding to the designated starting reference vehicle to the target passing determination unit 423 as the starting reference information INF2.

[0082] The target passing determination unit 423 executes a determination process for determining whether the starting reference vehicle specified by the remote operator has passed a predetermined area. Typically, the target passing determination unit 423 determines the starting reference vehicle based on the starting reference information INF2 received from the remote assistance device 2 and the target ID information sent from the target tracking unit 414, and determines whether the starting reference vehicle has passed the predetermined area. The determination result is sent to the second travel plan generation unit 424.

[0083] When the second travel plan generation unit 424 receives the determination result that the vehicle target of the starting reference has not passed the predetermined area, it generates a travel trajectory TR3 based on the travel plan to stop at the predetermined waiting position SP. Here, the predetermined waiting position SP is the position where the vehicle 10 waits for an instruction from the remote operator, and examples thereof include the position of the stop line in front of an intersection. Alternatively, when the second travel plan generation unit 424 receives the determination result that the vehicle target of the starting reference has passed the predetermined area, it generates a travel trajectory TR4 based on the travel plan to start and pass the predetermined area. The generated travel trajectory TR3 or TR4 is sent to the vehicle control unit 430.

[0084] The vehicle control unit 430 calculates travel control amounts related to the steering, acceleration, and deceleration of the vehicle 10 for realizing the travel trajectory TR1, TR3, or TR4. The calculated travel control amounts are output to the travel device 60. Hereinafter, the processes executed in the second travel plan generation unit 424 and the vehicle control unit 430 are referred to as "vehicle control processes". The travel device 60 controls the travel of the vehicle 10 in accordance with the travel control amounts calculated through the vehicle control process.

[0085] 1-5. Specific processes executed by the automatic driving control device

[0086] Figure 7 is a flowchart of the processes executed in the automatic driving control device 40. Figure 7 The routine shown is executed by the processor 42 of the automatic driving control device 40 executing the program 440 stored in the storage device 44 during the automatic driving of the vehicle 10.

[0087] In Figure 7 In step S100 of the routine shown, first, the target detection unit 412 detects the targets around the vehicle 10. In the next step S102, the target tracking unit 414 assigns target IDs to the targets detected in step S100. After the process of step S102, the process proceeds to step S104.

[0088] In step S104, the remote assistance determination unit 422 determines whether remote assistance is required. Figure 8This is a flowchart showing the routine of the remote assistance necessity determination process performed in the remote assistance determination unit 422. In step S104, the routine shown in Figure 8 is executed.

[0089] In Figure 8 , in step S130 of the routine shown, a predicted trajectory TR2 of the detection target is generated. In the next step S132, a predicted collision position CP is calculated, which is the position where the driving trajectory TR1 of the vehicle 10 generated in the first driving plan generation unit 421 intersects with the predicted trajectory TR2 of the detection target generated in step S130. In the next step S134, it is determined whether the driving trajectory TR1 and the predicted trajectory TR2 intersect. Here, when a valid predicted collision position CP is calculated in step S132, it is determined that the driving trajectory TR1 and the predicted trajectory TR2 intersect.

[0090] In the process of step S134, when it is determined that the driving trajectory TR1 and the predicted trajectory TR2 do not intersect, the process proceeds to step S136, and it is determined that a remote assistance request is not required.

[0091] On the other hand, in the process of step S134, when it is determined that the driving trajectory TR1 and the predicted trajectory TR2 intersect, the process proceeds to step S138. In step S138, based on the map information and the predicted collision position CP, it is determined whether the traffic environment of the vehicle 10 relative to the detection target is non-priority at the predicted collision position CP. As a result, when the vehicle 10 is non-priority at the predicted collision position, the process proceeds to step S140, and it is determined that a remote assistance request is required. On the other hand, when the vehicle 10 is not non-priority, the process proceeds to step S136, and it is determined that a remote assistance request is not required.

[0092] Returning to the Figure 7 shown flowchart, in the process of step S106, it is determined whether remote assistance is required. Here, it is determined whether it is determined that a remote assistance request is required in the process of the Figure 8 shown flowchart. As a result, when it is determined that a remote assistance request is not required, the process proceeds to step S110. In step S110, the driving trajectory TR1 of the vehicle 10 generated in the first driving plan generation unit 421 is adopted, and the autonomous driving of the vehicle 10 is continued. After the process of step S110 is performed, this routine ends.

[0093] On the other hand, in the process of step S106, when it is determined that remote assistance is required, the process proceeds to step S112. In step S112, a remote assistance request REQ and display information INF1 are sent to the remote assistance device 2. In the remote assistance device 2, the display information INF1 is displayed on the display device 62. The remote operator designates the starting reference vehicle from the input device 64 by referring to the display information INF1 displayed on the display device 62. The remote assistance device 2 sends the starting reference information INF2 including the target ID information corresponding to the designated vehicle to the target passing determination unit 423.

[0094] In step S114, it is determined whether the starting reference information INF2 has been received from the remote assistance device 2. As a result, if the starting reference information INF2 has not been received yet, the process proceeds to step S116, and if the starting reference information INF2 has been received, the process proceeds to step S118. In step S116, the vehicle 10 automatically drives according to the driving trajectory TR3 of the driving plan to stop and wait at the predetermined waiting position SP generated by the second driving plan generation unit 424. After the process of step S116 is performed, the process returns to S114 again.

[0095] In step S118, based on the target ID information, it is determined whether the starting reference vehicle has passed through the predetermined area. As a result, if the starting reference vehicle has not passed through the predetermined area yet, the process proceeds to step S120, and if the starting reference vehicle has passed through the predetermined area, the process proceeds to step S122.

[0096] In step S120, similarly to the process of step S116, the vehicle 10 automatically drives according to the driving trajectory TR3 of the driving plan to stop and wait at the predetermined waiting position SP. After the process of step S120 is executed, the process returns to step S118 again.

[0097] In step S122, the vehicle control amount for realizing the driving trajectory TR4 is calculated in the second driving plan generation unit 424. The calculated vehicle control amount is sent to the driving device 60. The driving device 60 performs the automatic driving of the vehicle 10 according to the vehicle control amount. Thereby, the vehicle 10 passes through the predetermined area.

[0098] In this way, according to the remote assistance system 100 of Embodiment 1, when the vehicle 10 is in a situation where remote assistance is required, the remote operator only designates the starting reference vehicle without designating the starting timing of the vehicle 10. As a result, the restricted time of the remote operator can be reduced, and thus the burden on the remote operator is alleviated.

[0099] 1-6. Application examples for specific traffic environment conditions

[0100] The remote assistance system 100 of Embodiment 1 can be applied to various traffic environment conditions. Hereinafter, application examples of a predetermined area and a predetermined waiting position in several traffic environment conditions will be described.

[0101] 1-6-1. First application example

[0102] Figure 9 It is a diagram showing a first application example of the remote assistance system. Figure 9 An intersection I2 where lane L1 intersects with lane L2 and a vehicle 10 traveling on lane L1 are depicted. The vehicle 10 is scheduled to turn right at the intersection I2. In the first application example, the predetermined area is the intersection I2, and the predetermined waiting position SP is the position in front of the intersection I2 on lane L1. The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2 and performs automatic driving according to the driving trajectory TR3 for waiting in front of the intersection. Moreover, after the starting reference vehicle indicated by the remote operator passes through the intersection I2, the vehicle 10 performs automatic driving according to the driving trajectory TR4 for turning right at the intersection I2.

[0103] 1-6-2. Second application example

[0104] Figure 10 It is a diagram showing a second application example of the remote assistance system. Figure 10 An intersection I3 where lane L1 intersects with lane L2 and a vehicle 10 traveling on lane L1 are depicted. Lane L2 is a priority lane with respect to lane L1. Therefore, on lane L1, a stop line is provided in front of the intersection I3. The vehicle 10 is scheduled to turn left or go straight at the intersection I3. In the second application example, the predetermined area is the intersection I3, and the predetermined waiting position SP is the position of the stop line in front of the intersection on lane L1. The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2 and performs automatic driving according to the driving trajectory TR3 for waiting at the temporary stop line of the intersection I3. Moreover, after the starting reference vehicle indicated by the remote operator passes through the intersection I3, the vehicle 10 performs automatic driving according to the driving trajectory TR4 for going straight or turning left at the intersection I3.

[0105] 1-6-3. Third application example

[0106] Figure 11 It is a diagram showing a third application example of the remote assistance system. Figure 11The figure depicts lane L3, lane L4 adjacent to lane L3, parked vehicle V3 parked on lane L3, and vehicle 10 traveling on lane L3 and approaching from behind parked vehicle V3. Vehicle 10 is scheduled to cross the lane from lane L3 to the side of lane L4 to avoid parked vehicle V3. In the third application example, the predetermined area is the parking area of parked vehicle V3, and the predetermined waiting position SP is the position behind parked vehicle V3. The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2 and performs automatic driving according to the driving trajectory TR3 for waiting at the position behind parked vehicle V3. Moreover, after the start reference vehicle indicated by the remote operator passes through the parking area of parked vehicle V3, vehicle 10 performs automatic driving according to the driving trajectory TR4 for crossing the lane to the side of lane L4 to avoid parked vehicle V3.

[0107] 1-6-4. Fourth application example

[0108] Figure 12 It is a diagram showing a fourth application example of the remote assistance system. Figure 12 The figure depicts lane L5, the same-direction lane L6 adjacent to lane L5, vehicle 10 traveling on lane L5, and vehicle V4 traveling on lane L6. Vehicle 10 is scheduled to change lanes to lane L6. In the fourth application example, the predetermined area is the area of vehicle 10, and the predetermined waiting position SP is lane L5. The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2 on lane L5 and performs automatic driving according to the driving trajectory TR3 for continuing to travel on lane L5. Moreover, when the start reference vehicle indicated by the remote operator is vehicle V4, after vehicle V4 overtakes vehicle 10, vehicle 10 performs automatic driving according to the driving trajectory TR4 for changing lanes to the side of lane L6.

[0109] 1-7. Modification example

[0110] The remote assistance system 100 of Embodiment 1 may also adopt a technical solution obtained by modifying as follows. In addition, the following modification examples can also be applied to the remote assistance system of Embodiment 2 described later.

[0111] The functional configuration of the automatic driving control device 40 is not limited. That is, part or all of the functions of the automatic driving control device 40 can be mounted on vehicle 10 or configured in the remote server 4 of the remote assistance device 2. This modification example can also be applied to the remote assistance system of Embodiment 2 described later.

[0112] The method for determining whether remote assistance is required in step S106 is not limited. That is, the remote assistance determination unit 422 is not limited to the method based on Figure 8 the routine shown, and other known methods can also be used to determine whether remote assistance is required.

[0113] In the display process, the display mode of the display information INF1 displayed on the display device 62 is not limited to an overhead view. That is, for example, the display information INF1 may also be information in which the vehicle 10, the travel trajectory TR1 of the vehicle 10, and one or more avoidance target vehicles with numbers or marks attached as identification information are arranged in a camera image of the front of the vehicle 10. In addition, the identification information attached to the avoidance target vehicle is not limited to numbers or marks, and may also be identified by colors or the like.

[0114] Figure 13 This is a diagram showing a modified example of the display mode of the display information INF1. As shown in this diagram, when the display device 62 is a touch panel display integrally formed with the input device 64, the display information INF1 displayed on the display may also be configured to be selectable by a touch operation. In this case, for example, as long as the rectangular regions FA and FB surrounding the avoidance target vehicle are respectively set as selectable regions, there is no need to attach numbers or marks to the avoidance target vehicle.

[0115] In the acceptance process, the input device 64 is not limited to being configured to select numbers or marks attached to the avoidance target vehicle as Figure 5 described above. That is, the display information INF1 may also include a confirmation message such as "Is it okay to pass after A passes?" together with an overhead view or a camera image. In this case, the input device 64 is configured to be able to input an answer of "yes" or "no" to the confirmation message. When the remote operator inputs an answer of "no" to the confirmation message, the automatic driving control device 40 is configured to generate display information INF1 including a confirmation message related to other avoidance target objects. According to this configuration, the remote operator can indicate the starting reference vehicle by answering "yes" or "no" to the confirmation message.

[0116] When the display information INF1 includes a camera image, there may sometimes be undetected objects located in the distance in the camera image. Such undetected objects are configured to be able to be selected by the remote operator by attaching an identification mark to the undetected object at the time when the identification unit 410 of the automatic driving control device 40 identifies it.

[0117] Embodiment 2.

[0118] Next, the remote assistance system 100 of Embodiment 2 will be described. In addition to the functional structure of the automatic driving control device 40, the remote assistance system 100 of Embodiment 2 has a configuration common to the remote assistance system 100 of Embodiment 1. The description of the elements common to the remote assistance system 100 of Embodiment 1 is omitted.

[0119] Features of the automatic driving control device 40 according to Embodiment 2

[0120] Figure 14 and Figure 15 and Figure 16 are diagrams for explaining the features of the remote assistance system according to Embodiment 2 Figure 14 depicts an intersection I5 where lane L1 intersects with lane L2, a vehicle 10 traveling on lane L1 at time t = t1, an avoidance target vehicle V5 traveling on lane L2 from the left side of vehicle 10 towards intersection I5, and an avoidance target vehicle V6 traveling on the oncoming lane of lane L1 towards intersection I5. Vehicle 10 is scheduled to turn right at intersection I5. In this application example, the predetermined area is intersection I5, and the predetermined waiting position SP is the position in front of intersection I5 on lane L1. When vehicle 10 passes through the predetermined area, it needs to wait not only for vehicle V6 to pass but also for vehicle V5 to pass

[0121] The automatic driving control device 40 sends a remote assistance request REQ to the remote assistance device 2 and performs automatic driving according to a driving trajectory TR3 for waiting in front of the intersection. Moreover, when the starting reference vehicle indicated by the remote operator is the avoidance target vehicle V5, vehicle 10 turns right at intersection I5 after the avoidance target vehicle V5 passes through intersection I5

[0122] Figure 15 depicts the appearance of vehicle 10 traveling on lane L1, avoidance target vehicle V5 traveling on lane L2 from the left side of vehicle 10 towards intersection I5, and avoidance target vehicle V6 traveling on the oncoming lane of lane L1 towards intersection I5 at time t = t2, which is several seconds after time t = t1. In Figure 15 the example shown, the appearance of the avoidance target vehicle V6 noticing the avoidance target vehicle V5 and decelerating, and the avoidance target vehicle V5 accelerating is described. In this case, the passing order of the avoidance target vehicles V5 and V6 passing through the intersection I5 as the predetermined area will be swapped, so it is necessary to re-specify the starting reference vehicle

[0123] Figure 16 depicts the situation where at time t = t2, which is several seconds after time t = t1, in addition to vehicle 10, avoidance target vehicle V5, and avoidance target vehicle V6, a new avoidance target vehicle V7 appears behind the avoidance target vehicle V6 and travels on the oncoming lane of lane L1 towards intersection I5. In this case, the avoidance target vehicle V7 is newly added to the passing order of the multiple avoidance target vehicles passing through the intersection I5 as the predetermined area, so it is necessary to re-specify the starting reference vehicle

[0124] Therefore, the remote assistance system according to Embodiment 2 further has a function of determining the predicted passing order of the avoidance target vehicle passing through a predetermined area. Figure 17 FIG. 3 is a functional block diagram showing a part of the functions of the automatic driving control device 40 according to Embodiment 2. The automatic driving control device 40 according to Embodiment 2, in addition to the functions of the automatic driving control device 40 according to Embodiment 1, the travel plan unit 420 further has a target passing order change determination unit 425.

[0125] The target passing order change determination unit 425 calculates the predicted passing order of the avoidance target vehicle that may collide with the vehicle 10 passing through the predetermined area based on the target ID information sent from the target tracking unit 414. Moreover, when the target passing order change determination unit 425 determines that there is a change between the previously calculated predicted passing order and the currently calculated predicted passing order, the travel plan unit 420 discards the start reference information INF2 sent from the target tracking unit 414, and sends the display information INF1 and the remote assistance request REQ based on the new predicted passing order to the remote assistance device 2 again.

[0126] According to this processing, when the predicted passing order of multiple avoidance target vehicles passing through the predetermined area changes, an opportunity to re-specify the passing reference vehicle can be obtained. Thus, even when the conditions of multiple avoidance target vehicles change unexpectedly, an opportunity to specify an appropriate passing reference vehicle corresponding to the conditions can be obtained.

[0127] 2-2. Specific processing executed by the automatic driving control device according to Embodiment 2

[0128] Figure 18 FIG. 4 is a flowchart of the processing executed in the automatic driving control device according to Embodiment 2. Similar to the Figure 7 routine shown, Figure 18 the routine shown is executed by the processor 42 of the automatic driving control device 40 executing the program 440 stored in the storage device 44 during the automatic driving of the vehicle 10.

[0129] In Figure 18 steps S200, S202, S204, S206, S210, S212, S214, and S216 of the routine shown, the same processing as Figure 7 steps S100, S102, S104, S106, S110, S112, S114, and S116 of the routine shown is executed.

[0130] When the start reference information INF2 is received in the determination of step S214, the process proceeds to step S218. In step S218, the predicted passing order of the avoidance target object passing through the predetermined area is calculated in the target passing order change determination unit 425. Further, it is determined whether the calculated predicted passing order has changed from the predicted passing order calculated in the previous process of step S218. As a result, when it is not determined that the determination is established, the process proceeds to step S220. On the other hand, when it is determined that the determination is established in the process of S218, after discarding the start reference information INF2 received in step S214, the process returns to step S204 again.

[0131] In step S220, based on the target ID information, it is determined whether the start reference vehicle has passed through the predetermined area. As a result, when the start reference vehicle has not passed through the predetermined area, the process proceeds to step S222, and when the start reference vehicle has passed through the predetermined area, the process proceeds to step S224.

[0132] In step S222, the vehicle 10 automatically drives along the travel trajectory TR3 based on the travel plan of stopping and waiting at the predetermined waiting position SP. After performing the process of step S222, the process returns to step S218 again.

[0133] In step S224, the vehicle control amount for realizing the travel trajectory TR4 is calculated in the second travel plan generation unit 424. The calculated vehicle control amount is sent to the travel device 60. The travel device 60 automatically drives the vehicle 10 according to the vehicle control amount. Thus, the vehicle 10 passes through the predetermined area.

[0134] As described above, according to the remote assistance system 100 of the second embodiment, when the predicted passing order of a plurality of avoidance target vehicles passing through the predetermined area has changed, it is determined again whether remote assistance is required. Thus, the determination process based on the start reference vehicle indicated before the change in the predicted passing order is not continued, and therefore, it is possible to prevent inappropriate remote assistance from being performed.

Claims

1. A remote assistance system, which, when it is determined that a remote assistance request situation in which it is difficult to continue autonomous driving occurs due to a vehicle in autonomous driving approaching one or more avoidance target vehicles within a predetermined area, entrusts a remote operator to perform remote assistance for passing through the predetermined area. The remote assistance system includes: A first processor provided in the vehicle; A second processor provided in a remote server; The first processor is configured to: Detect a target around the vehicle; Determine the trajectory of a first target among the detected targets; Determine whether the trajectory of the vehicle intersects with the trajectory of the first target; When it is determined that the trajectory of the vehicle intersects with the trajectory of the first target, determine whether the traffic priority situation at the position where the trajectory of the vehicle intersects with the trajectory of the first target is non-priority of the vehicle relative to the first target; When it is determined that the traffic priority situation at the position where the trajectory of the vehicle intersects with the trajectory of the first target is not non-priority of the vehicle relative to the trajectory of the first target, it is determined that a remote assistance request is not required; When it is determined that the traffic priority situation at the position where the trajectory of the vehicle intersects with the trajectory of the first target is a situation of non-priority of the vehicle relative to the trajectory of the first target, it is determined that a request for remote assistance is required, and display information including an overhead view of the vehicle and the first target is sent to the remote server; The second processor is configured to: Receive the display information; Display the display information to the remote operator; Receive only the starting reference vehicle indicated by the remote operator, and the remote operator only indicates the starting reference vehicle without indicating the starting timing of the starting reference vehicle; And Send the remote assistance end after the starting reference vehicle to the vehicle; The first processor is configured to: Receive the starting reference vehicle; Determine whether the starting reference vehicle has passed through the predetermined area; And When it is determined that the starting reference vehicle has passed through the predetermined area, the vehicle is caused to pass through the predetermined area by autonomous driving.

2. The remote assistance system according to claim 1, The first processor is configured to cause the vehicle to wait at a predetermined waiting position by autonomous driving until it is determined that the starting reference vehicle has passed through the predetermined area.

3. The remote assistance system according to claim 1, The second processor is configured to: Display each of the one or more avoidance target vehicles together with identification information in an overhead view, Accept the selection of the identification information by the remote operator.

4. The remote assistance system according to claim 1, The second processor is configured to: Display an overhead view including each of the one or more avoidance target vehicles on a touch panel display, Accept the designation of the starting reference vehicle from the touch panel display.

5. The remote assistance system according to claim 1 or 2, the second processor is configured to: Display a confirmation message designating any one of the one or more avoidance target vehicles as the starting reference vehicle. Accept the correctness of the confirmation message.

6. The remote assistance system according to any one of claims 1 to 4, In the case where it is determined that the remote assistance request condition is satisfied, the first processor is configured to Calculate the predicted passing order of the one or more avoidance target vehicles passing through the predetermined area, When the predicted passing order changes after the starting reference vehicle is received, cancel the received starting reference vehicle.

7. A remote assistance method, which is a remote assistance method for entrusting a remote operator to perform remote assistance for passing through the predetermined area when it is determined that a remote assistance request condition in which it is difficult to continue autonomous driving occurs due to a vehicle in autonomous driving approaching one or more avoidance target vehicles in the predetermined area, including: Detect a target around the vehicle; Determine the trajectory of the first target among the detected targets; Determine whether the trajectory of the vehicle intersects the trajectory of the first target; When it is determined that the trajectory of the vehicle intersects the trajectory of the first target, determine whether the traffic priority status at the position where the trajectory of the vehicle intersects the trajectory of the first target is that the vehicle is non-priority relative to the first target; In the case where it is determined that the traffic priority status at the position where the trajectory of the vehicle intersects the trajectory of the first target is not non-priority of the vehicle relative to the first target trajectory, it is determined that a remote assistance request is not required, In the case where it is determined that the traffic priority status at the position where the trajectory of the vehicle intersects the trajectory of the first target is a status where the vehicle is non-priority relative to the first target trajectory, it is determined that a remote assistance request is required, and display information including an overhead view of the vehicle and the first target is sent to the remote server, Receive the display information; Display the display information to the remote operator; Receive only the starting reference vehicle indicated by the remote operator, and the remote operator only indicates the starting reference vehicle without indicating the starting timing of the starting reference vehicle; Send the starting reference vehicle to the vehicle and then end the remote assistance; Receive the starting reference vehicle; Determine whether the starting reference vehicle has passed through the predetermined area; And In the case where it is determined that the starting reference vehicle has passed through the predetermined area, cause the vehicle to pass through the predetermined area by autonomous driving.

Images