Driving management device, driving management method, and program

Abstract

To provide a technique capable of appropriately coping with delay occurring in remote support of a vehicle by reducing influence of the delay occurring between a terminal on a vehicle side and the terminal on a remote supporter side.SOLUTION: A travel management device manages travel of a vehicle. The travel management device is configured to execute the steps of: sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; sequentially generating prediction information indicating a position of a target present around the vehicle at a future time point based on the sequentially acquired sensor information; transmitting a remote support request to a remote supporter; receiving remote support from the remote supporter in response to the remote support request; and causing the vehicle to autonomously travel in accordance with the remote support in response to confirmation that a position of the target obtained from the sensor information acquired at a determination time point after a time point at which the remote support is received and a position of the target obtained from the prediction information in which the determination time point is the future time point match each other.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a technology for managing the driving of a vehicle having an autonomous driving function. [Background technology]

[0002] In an autonomous vehicle, a driving route is created based on information obtained from sensors mounted on the vehicle, and the vehicle is driven autonomously according to the created driving route. However, in a situation where other vehicles around are moving in a complex manner, there may be a situation where the vehicle cannot determine whether it is possible to operate according to the driving route. In such a situation, a technology is known in which a remote assistant in a different location from the vehicle is requested to make a decision or give instructions, and the vehicle's operation is determined according to the decision or instructions from the remote assistant. The decision or instructions sent from the remote assistant to the vehicle are sometimes called remote assistance.

[0003] Patent Document 1 discloses an example of such a technology related to remote assistance. According to the conventional technology disclosed in Patent Document 1, a driving assistance request device that communicates with a driving assistance device that remotely supports the driving of the vehicle is mounted on an autonomous driving vehicle. The driving assistance request device detects objects around the vehicle. Then, based on a prediction result that predicts the position of the object and the future behavior of the object, it is determined whether or not to request driving assistance from the driving assistance device. If it is determined that driving assistance is requested, a request message and information indicating the object are transmitted to the driving assistance device. The driving assistance request device and the driving assistance device communicate with each other via a communication network.

[0004] In addition to Patent Document 1, Patent Document 2 below can be mentioned as an example of a document that shows the technical level at the time of filing in the technical field of the present disclosure. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-163323 A [Patent Document 2] Patent Publication No. 2021-071783 Summary of the Invention [Problem to be solved by the invention]

[0006] As disclosed in Patent Document 1, in a system for remotely supporting a vehicle, information is transmitted and received between a device on the vehicle side and a device on the remote supporter side by communication via a communication network. Therefore, a delay occurs in the transmission and reception of information when a request for remote support is sent from the vehicle side to the remote supporter side, and when remote support is sent from the remote supporter side to the vehicle side. If the delay is large, the situation around the vehicle may change significantly between the time when the vehicle requests remote support and the time when the remote support in response is received, resulting in a situation that is significantly different from what the remote supporter had grasped. In such a case, the remote support input by the remote supporter may no longer be considered effective support.

[0007] The present disclosure has been made in consideration of the above-mentioned problems. The present disclosure aims to provide a technology that can reduce the effect of delays occurring between a terminal on the vehicle side and a terminal on the remote supporter side and appropriately deal with delays occurring during remote support of a vehicle. [Means for solving the problem]

[0008] The first aspect relates to a driving management device that manages the driving of a vehicle. The driving management device is Sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; Sequentially generating prediction information representing future positions of targets present around the vehicle based on sequentially acquired sensor information; Sending a remote support request to a remote supporter; Receiving remote support from a remote supporter in response to the remote support request; The vehicle is configured to execute the following operation upon confirmation that the target position obtained from the sensor information acquired at a determination time point after the time point at which the remote assistance is received matches the target position obtained from the prediction information whose determination time point is the above-mentioned future time point.

[0009] The second aspect relates to a driving management method for managing the driving of a vehicle having an autonomous driving function. The driving management method is Sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; Sequentially generating prediction information representing future positions of targets present around the vehicle based on sequentially acquired sensor information; Sending a remote support request to a remote supporter; Receiving remote support from a remote supporter in response to the remote support request; The method includes confirming that the target position obtained from sensor information acquired at a determination time point after the time point at which the remote assistance is received matches the target position obtained from prediction information with the determination time point being the above-mentioned future time point, and then causing the vehicle to drive autonomously in accordance with the remote assistance.

[0010] A third aspect relates to a program for managing driving of a vehicle having an autonomous driving function. The program causes a computer to execute the driving management method according to the second aspect. Effect of the Invention

[0011] According to the remote assistance technology disclosed herein, before controlling the vehicle according to the remote assistance, it is confirmed whether the remote assistance is valid or not. This reduces the effect of delays occurring between the vehicle terminal and the remote assistant terminal, and makes it possible to appropriately deal with delays occurring during remote assistance of the vehicle. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of the configuration of a remote support system. [Diagram 2]FIG. 1 is a block diagram showing an example of the configuration of an autonomous driving vehicle. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of a remote support terminal. [Figure 4] FIG. 1 is a diagram for explaining an overview of a scene in which remote support is performed. [Diagram 5] 1 is a diagram for explaining prediction information generated by a driving management device according to an embodiment of the present disclosure. FIG. [Figure 6] FIG. 13 is a diagram showing an example of a scene where traffic conditions match. [Figure 7] FIG. 1 illustrates an example of a scene where traffic conditions are inconsistent. [Figure 8] FIG. 13 illustrates another example of a traffic mismatch scenario. [Figure 9] 1 is a block diagram showing an example of a configuration of a run management device according to an embodiment of the present disclosure. [Figure 10] 1 is a flowchart illustrating an example of processing by a run management device according to an embodiment of the present disclosure. [Figure 11] 13 is a flowchart showing another example of processing by the run management device according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a block diagram showing an example of the configuration of a run management device in a first modified example. [Figure 13] FIG. 11 is a diagram for explaining a determination of a match between traffic conditions in the second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0014] 1. Configuration of the remote support system A driving management device according to an embodiment of the present disclosure manages driving of an autonomous vehicle supported by a remote support system. The configuration of the remote support system is described with reference to FIG. 1. The remote support system 100 is a system that supports the autonomous driving of an autonomous vehicle 20 having an autonomous driving function by remote operation of a remote supporter 40. The remote supporter 40 is an operator who remotely supports the autonomous vehicle 20, and is also called a remote operator. The autonomous driving level of the autonomous vehicle 20 is assumed to be, for example, level 4 or level 5. Hereinafter, the autonomous vehicle 20 will be simply referred to as vehicle 20.

[0015] A remote support terminal 30 is used for remote support of the vehicle 20 by the remote supporter 40. The remote support terminal 30 is connected to the server 10 of the management center via a communication network including the Internet. The vehicle 20 is also connected to the server 10 of the management center via a communication network including 4G and 5G. A plurality of remote support terminals 30 and a plurality of vehicles 20 are connected to the server 10. The server 10, which receives a remote support request from the vehicle 20, selects a person in charge from among the available remote supporters 40, and connects the remote support terminal 30 of the remote supporter 40 who has become the person in charge to the vehicle 20 requesting support. After selecting the remote supporter 40 who will be in charge of the remote support, the remote support terminal 30 of the remote supporter 40 who has become the person in charge may be directly connected to the vehicle 20 without going through the server 10.

[0016] Situations in which the vehicle 20 requests assistance include, for example, overtaking a preceding vehicle, passing a crosswalk, turning right at an intersection, and deviating from a lane to avoid an obstacle. In remote assistance, at least a part of the decisions for automatic driving by the vehicle 20 is made by the remote supporter 40. Basic calculations related to cognition, judgment, and operation required for driving are performed in the vehicle 20. The remote supporter 40 judges the action that the vehicle 20 should take based on information transmitted from the vehicle 20, and transmits commands to the vehicle 20 based on the judgment result. Commands for remote assistance transmitted from the remote supporter 40 to the vehicle 20 include commands for the vehicle 20 to proceed and for the vehicle 20 to stop. Commands for remote assistance also include commands for entering an intersection, passing a crosswalk, overtaking a preceding vehicle, and the like.

[0017] 2 is a block diagram showing an example of a configuration of a vehicle 20. The vehicle 20 is equipped with a driving management device 21 according to an embodiment of the present disclosure. The driving management device 21 is a device that manages driving by the autonomous driving function of the vehicle 20, and includes a plurality of ECUs (Electronic Control Units). The vehicle 20 also includes an external sensor 22, an internal sensor 23, an actuator 24, and a communication device 25. These are connected to the driving management device 21 using an in-vehicle network such as a CAN (Controller Area Network).

[0018] The running management device 21 includes one or more processors 21a (hereinafter simply referred to as processor 21a) and one or more memories 21b (hereinafter simply referred to as memory 21b) coupled to the processor 21a. The processor 21a may be, for example, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other processing unit, or may be a combination of two or more of the CPU, FPGA, ASIC, or other processing unit. The memory 21b stores a plurality of programs 21c (hereinafter simply referred to as programs 21c) executable by the processor 21a and various information related thereto. The programs 21c include a plurality of executable instructions.

[0019] The processor 21a executes the program 21c, thereby realizing various processes by the processor 21a. The program 21c includes an automatic driving program for realizing automatic driving. The automatic driving program is executed by the processor 21a, thereby performing processing for automatic driving of the vehicle 20. The program 21c also includes a remote assistance program for receiving remote assistance. The processor 21a executes the remote assistance program, thereby performing processing for requesting assistance from the server 10 and executing assistance content acquired from the remote assistance terminal 30 directly or via the server 10 in the vehicle 20.

[0020] The external sensor 22 includes a recognition sensor and a position sensor. The recognition sensor is a sensor that acquires information for recognizing the situation around the vehicle 20. The recognition sensor includes a camera that captures images of the surroundings of the vehicle 20, particularly the area in front of the vehicle 20. Examples of recognition sensors other than cameras include LiDAR (Laser Imaging Detection and Ranging) and millimeter wave radar. The position sensor is a sensor that detects the position and orientation of the vehicle 20. An example of a position sensor is a GPS (Global Positioning System) sensor.

[0021] The sensor information obtained from the recognition sensor and the position information obtained from the position sensor are sequentially acquired by the driving management device 21. The driving management device 21 predicts future short-term changes in the environment surrounding the vehicle 20 using information on the situation around the vehicle 20 obtained by these external sensors 22, and generates a driving plan for the vehicle 20 based on the predicted information and a target route determined by, for example, a navigation system. The driving plan includes the target position of the vehicle 20 and a target speed or target acceleration at that position. The driving management device 21 controls the vehicle 20 so that the vehicle 20 drives in accordance with the driving plan, thereby performing automatic driving of the vehicle 20.

[0022] The internal sensor 23 includes a status sensor that acquires information regarding the motion of the vehicle 20. Examples of the status sensor include a wheel speed sensor, an acceleration sensor, an angular velocity sensor, and a steering angle sensor. The acceleration sensor and the angular velocity sensor may be an IMU. The information obtained by the internal sensor 23 is sequentially acquired by the driving management device 21.

[0023] The actuator 24 includes a steering device that steers the vehicle 20, a drive device that drives the vehicle 20, and a braking device that brakes the vehicle 20. The steering device includes, for example, a power steering system, a steer-by-wire steering system, and a rear-wheel steering system. The drive device includes, for example, an engine, an EV system, and a hybrid system. The braking device includes, for example, a hydraulic brake and a regenerative brake. The actuator 24 is operated by a control signal transmitted from the driving management device 21.

[0024] The communication device 25 is a device that controls wireless communication between the vehicle 20 and the outside. The communication device 25 communicates with the server 10 and the remote support terminal 30 via a communication network. Information processed by the driving management device 21 is transmitted to the server 10 or the remote support terminal 30 using the communication device 25. Information processed by the server 10 or the remote support terminal 30 is imported into the driving management device 21 using the communication device 25. In addition, when vehicle-to-vehicle communication with other vehicles or road-to-vehicle communication with infrastructure facilities is required for autonomous driving, communication with these external devices is also performed by the communication device 25.

[0025] 3 is a block diagram showing an example of the configuration of the remote support terminal 30. The remote support terminal 30 includes a computer 31, a display device 32, an input device 33, and a communication device 35. The display device 32, the input device 33, and the communication device 35 are connected to the computer 31. The remote support terminal 30 may be installed in a management center, or may be installed outside the management center, for example, in the home of a remote supporter 40.

[0026] The computer 31 includes one or more processors 31a (hereinafter simply referred to as processor 31a) and one or more memories 31b (hereinafter simply referred to as memory 31b) coupled to the processor 31a. The processor 31a may be, for example, a CPU, an FPGA, an ASIC, or other processing unit, or may be a combination of two or more of the CPU, the FPGA, the ASIC, or other processing units. The memory 31b stores one or more programs 31c (hereinafter simply referred to as program 31c) executable by the processor 31a and various information related thereto. The program 31c includes a plurality of executable instructions. The processor 31a executes the program 31c to realize various processes by the processor 31a.

[0027] The display device 32 is a device that displays information for the remote supporter 40 to make a decision on remote support. The information for making a decision on remote support displayed by the display device 32 includes information acquired by a recognition sensor. Typically, the information includes a camera image obtained by a camera capturing an image of the surroundings of the vehicle 20, particularly the front. The information displayed on the display device 32 may include information other than the camera image acquired by the external sensor 22, or may include information acquired by the internal sensor 23. The information is transmitted from the communication device 25 of the vehicle 20 and acquired by the remote support terminal 30.

[0028] The input device 33 is a device through which the remote supporter 40 inputs operations for remote support. Information input through the input device 33 is processed by the computer 31 and transmitted to the vehicle 20. Specific examples of the input device 33 include a button, a lever, and a touch panel. For example, the remote supporter 40 can instruct the vehicle 20 to proceed / stop or enter / wait at an intersection by pressing a button.

[0029] The communication device 35 is a device that controls communication between the remote assistance terminal 30 and the outside. The communication device 35 communicates with the server 10 and the vehicle 20 via a communication network. Information processed by the computer 31 is transmitted to the server 10 or the vehicle 20 using the communication device 35. Information processed by the server 10 or the vehicle 20 is input to the computer 31 using the communication device 35.

[0030] 2. Overview of remote support An overview of the remote assistance will be described using an example of a situation in which the vehicle 20 turns right at an intersection. Fig. 4 illustrates the state of the vehicle 20 at time tx and time ty. As moving objects around the vehicle 20, there are a vehicle 50 going straight in the intersecting lane around the intersection, a vehicle 60 turning right at the intersection, and a pedestrian 70.

[0031] When the vehicle 20 approaches an intersection, the driving management device 21 judges whether or not it is possible to enter the intersection, and if it is judged that entry is possible, the driving management device 21 allows the vehicle 20 to enter the intersection. If the driving management device 21 cannot autonomously judge that entry into the intersection is possible and judges that remote support is necessary, the driving management device 21 requests remote support. At time tx, the driving management device 21 that judges that remote support is necessary transmits a remote support request. The remote support request is once received by the server 10, and after the remote supporter 40 in charge of supporting the vehicle 20 is assigned, it is transmitted to the remote support terminal 30. The remote support requested here is an instruction to enter or wait at the intersection. At this time, information for the remote supporter 40 to judge remote support, including sensor information acquired from the recognition sensor, is also transmitted to the remote support terminal 30 and displayed on the display device 32.

[0032] The remote supporter 40 responds to the remote support request, and inputs the remote support to the input device 33 by referring to the information displayed on the display device 32. The input remote support is transmitted from the remote support terminal 30 via the communication network and received by the driving management device 21. Here, remote support instructing entry into an intersection is transmitted. Having received the remote support, the driving management device 21 creates a driving plan for the vehicle 20 to enter the intersection, and controls the vehicle 20 so that it drives in accordance with the driving plan. In this way, the vehicle 20 can turn right at the intersection.

[0033] However, delays occur because the transmission of the remote support request and the sensor information from the driving management device 21 to the remote support terminal 30, and the transmission of the remote support from the remote support terminal 30 to the driving management device 21 are performed via a communication network. Therefore, a time lag occurs between the time when the driving management device 21 transmits the remote support request and the time when it receives the remote support in response to that request.

[0034] Time ty is the time when the vehicle 20 receives the remote assistance. Since there is a time lag between time tx and time ty, the positions of the vehicle 50, vehicle 60, and pedestrian 70, which are moving bodies around the vehicle 20, are different between the two times. Even if the positions of the moving bodies change in this way, if the situation around the vehicle 20 is not significantly changed, it can be said that the remote assistance is effective at time ty. However, if the delay is large and the situation around the vehicle 20 is significantly different, the control of the vehicle 20 according to the remote assistance may differ from the control intended by the remote assistant 40. Therefore, the remote assistance cannot be said to be effective. It is undesirable from the viewpoint of ensuring smooth traffic and safety that the vehicle 20 is operated according to the remote assistance that is no longer effective.

[0035] Therefore, when the driving management device 21 receives remote assistance, it checks whether the remote assistance is still valid at the time of reception before driving in accordance with the remote assistance. Then, after checking that the remote assistance is valid, it controls the vehicle 20 so that the vehicle 20 drives in accordance with the remote assistance. In order to check that the remote assistance is valid, the driving management device 21 checks whether the "traffic conditions" match when the remote assistance is received and when the remote assistance request is sent. Then, upon checking that the traffic conditions match, it controls the vehicle 20 so that the vehicle 20 drives in accordance with the remote assistance.

[0036] 3.Generating forecast information Prediction information is used to determine whether traffic conditions match or not. Prediction information is information generated by the driving management device that indicates the future positions of moving objects around the vehicle 20. Prediction information is generated sequentially based on sensor information sequentially acquired from the recognition sensor. Moving objects detected by the recognition sensor and for which prediction information is generated by the driving management device 21 are also called targets. Any moving object around the vehicle 20, such as a vehicle, a pedestrian, or a bicycle, can be a target.

[0037] The generated prediction information will be explained with reference to FIG. 5. Here, a pedestrian is detected as a target, and prediction information indicating the future position of the pedestrian is generated. Times t1, t2, t3, and t4 indicate the timing at which the prediction information is generated. Also, in FIG. 5, the prediction information generated at time t1 is represented in a coordinate system including a time axis. The driving management device 21 can detect the position and movement speed of the pedestrian from, for example, a camera or LiDAR, and calculate the position where the pedestrian is predicted to be located in the future based on the information.

[0038] In Fig. 5, the gray ovals are prediction information, and the positions where the target is predicted to exist are expressed as a probability distribution. The darker the parts, the higher the probability that the target exists. The prediction information may be calculated as a probability distribution in this way, or may be calculated as a prediction range where the target is predicted to exist.

[0039] Times u1, u2, u3, and u4 indicate elapsed times in the prediction information. That is, the time points when the elapsed times are u1, u2, u3, and u4 correspond to times t1+u1, t1+u2, t1+u3, and t1+u4, respectively. In this way, the prediction information is calculated as a series of prediction information that indicates the position of the target at each time after a predetermined time has elapsed from the time when the prediction information is generated. The same applies to prediction information generated at times other than time t1, and prediction information generated at times t2, t3, and t4 is also calculated as a series of prediction information. The intervals at which the prediction information is generated and the intervals of the times in the series of prediction information may be the same or different. That is, for example, the time from time t1 to t2 and the time from time u1 to u2 may be the same or different.

[0040] In the example of Fig. 5, prediction information for four frames from elapsed time u1 to u4 is calculated, but the number of frames of information calculated as prediction information may be more or less than that. The maximum number of frames to be calculated is determined, for example, taking into consideration the expected delay time and prediction accuracy. For example, if the maximum time that is considered to be sufficient for prediction accuracy is 50 seconds, prediction information including predictions up to 50 seconds ahead may be generated. Alternatively, for example, if the average delay time obtained from past data is 30 seconds, prediction information including predictions up to 30 seconds ahead may be generated.

[0041] The prediction information may include predictions of not only the future position of the target but also the speed of the target. The calculated prediction information is temporarily stored in the memory 21b.

[0042] 4. Traffic conditions match or mismatch 4-1. Overview Next, a determination of whether the traffic conditions at the time of receiving the remote support and the time of transmitting the remote support request match or do not match will be described. When remote support is received, the driving management device 21 obtains the position of the target at the time of receiving the remote support from the sensor information. Hereinafter, the position of the target at the time of receiving the remote support is referred to as the actual position. In addition, the driving management device 21 acquires the position of the target at the time of receiving the remote support predicted at the time of transmitting the remote support request from the prediction information temporarily stored in the memory 21b. Hereinafter, the position of the target obtained from the prediction information is referred to as the predicted position. When the actual position of the target and the predicted position of the target match, the driving management device 21 determines that the traffic conditions match, and when the actual position of the target and the predicted position of the target do not match, the traffic conditions do not match, that is, the traffic conditions change between the time of transmitting the remote support request and the time of receiving the remote support.

[0043] The actual position of the target is expressed by a point, whereas the prediction information represents the predicted position of the target by a probability density or a range, so the predicted position of the target obtained from the prediction information is also expressed by a probability density or a range. Therefore, if the predicted position of the target is expressed by a probability density, the match between the actual position of the target and the predicted position of the target means that the predicted presence probability at the actual position of the target is equal to or greater than a predetermined value. If the predicted position of the target is expressed by a range, the match between the actual position of the target and the predicted position of the target means that the actual position of the target is included in the range representing the predicted position of the target. In addition, the comparison of the target positions is performed for all targets detected at the time of receiving the remote assistance, and if the actual position of any target does not match the predicted position of the target, or if prediction information is not generated for any target, it is determined that the traffic conditions do not match.

[0044] Fig. 6 shows an example of a state in which the traffic conditions match between the time when remote assistance is received (right) and the time when a remote assistance request is sent (left). Fig. 6 shows vehicle 20, and targets vehicle 50, vehicle 60, and pedestrian 70. In both the left and right figures, the ellipses filled with diagonal lines indicate predicted positions of the targets at the time when remote assistance is received, among the predicted future positions of the targets detected at the time when the remote assistance request is sent. Here, the predicted positions of the targets are shown as ranges.

[0045] The target positions shown in the right diagram are the actual positions of the targets at the time of receiving remote assistance acquired from the sensor information of the recognition sensor, and the shaded ellipses are the predicted positions of the targets predicted at the time of sending the remote assistance request. The driving management device 21 compares the actual positions of the targets in the right diagram with the predicted positions represented by the ellipses. In the right diagram of FIG. 6, the targets are located within the shaded ellipses, so it is determined that the traffic conditions match. Since it is considered that the situation at the time of sending the remote assistance request continues at the time of receiving remote assistance, the driving management device 21 can employ remote assistance.

[0046] 7 and 8 show examples of a state in which the traffic conditions do not match between the time of receiving remote assistance and the time of sending a remote assistance request. As in Fig. 6, the ellipses filled with diagonal lines indicate the predicted positions of the targets at the time of receiving the remote assistance, among the future predicted positions of each target detected at the time of sending the remote assistance request. Here, too, the predicted positions of the targets are shown as ranges.

[0047] In the example of FIG. 7, the pedestrian 70 has moved to a position different from the predicted range. Therefore, it is determined that the traffic conditions do not match. In the example of FIG. 8, a pedestrian 80 that is not included in the predicted information has newly appeared. Therefore, it is determined that the traffic conditions do not match. Since the situation when the remote assistance is received has changed from the situation at the time of the request, the driving management device 21 does not treat the remote assistance as valid assistance.

[0048] In this way, by checking whether the traffic conditions match between when remote support is received and when it is requested, it is possible to prevent errors in remote support caused by changes in conditions due to communication delays. If the traffic conditions do not match, the driving management device 21 prioritizes control of the vehicle 20 to ensure safety over remote support. For example, even if the remote support instructs the vehicle to turn right, it is expected that the driving management device 21 will stop or slow down the vehicle 20 without making the vehicle turn right. In addition, at this time, the driving management device 21 may reject the received remote support and request remote support from the remote supporter 40 again.

[0049] 4-2. How to calculate consistency The determination of whether the traffic conditions match or not may be performed as follows. The driving management device 21 compares the actual position of the target at the time of receiving remote assistance with the predicted position of the target predicted at the time of sending the remote assistance request, and calculates a value indicating the degree of match. If the value indicating the degree of match is equal to or greater than a threshold, it is determined that the traffic conditions match, and if it is less than the threshold, it is determined that the traffic conditions do not match. Here, the value indicating the degree of match between the traffic conditions at the time of receiving remote assistance and the time of sending the remote assistance request is defined as match X. An example of a method of calculating match X will be described.

[0050] The predicted information for each target i is i It can be expressed as (x,v,t). i is a function that outputs a fixed value when the position x and velocity v of the target i at time t are given, and the larger the output value, the greater the degree of agreement with the predicted information. i may be a probability distribution or a function determined according to the predicted range of the position of the target i. When the predicted information for the target i has not been generated, φ i = 0. Consistency X is calculated using the following formula:

number

[0051] where x i (t) is the position of target i at time t, v i (t) is the velocity of target i at time t, and F is φ i , x, and v as inputs, and outputs a discrete or continuous value.

[0052] The function F can be exemplified as follows. The first function is the position x of each target i. i (t) and velocity v i φ calculated for (t) i (t,x i (t),v i If any one or more of the values ​​of (t)) is smaller than a certain value, output 0, and φ i (t,x i (t),v i (t) is a function that outputs 1 if it is greater than or equal to a certain value.

[0053] The second is φ calculated for each target i. i (t,x i (t),v i (t)) is a function that outputs the minimum value. In this case, the equation for calculating the consistency X is expressed as follows:

number

[0054] In the first and second examples, the traffic situation is determined to be consistent or inconsistent based on the position and speed of the target i that is the most different from the prediction. This makes it possible to more accurately determine the situation in which an unexpected event is occurring, even when there are a large number of targets.

[0055] As a third example, F may be a function that calculates the P value. The P value indicates how likely it is that the chi-squared value will occur statistically at the time of receiving remote assistance. The chi-squared value is calculated using the actual position (observed value) and predicted position (expected value) of the target. In this case, φ i is a probability distribution. As described above, by calculating the degree of agreement X as a fixed value, it is possible to quantitatively evaluate the degree of agreement between the traffic conditions at the time of receiving the remote assistance request and the time of sending the remote assistance request.

[0056] 5. Functional configuration The functions of the run management device 21 will be described with reference to Fig. 9. Each functional unit is realized by the processor 21a executing the program 21c.

[0057] The target detection unit 201 acquires sensor data from a recognition sensor and detects targets. Target detection is performed, for example, by acquiring a camera image captured by a camera and detecting vehicles, pedestrians, etc. through image recognition of the camera image. Alternatively, target detection is performed by acquiring point cloud data by LiDAR and clustering the acquired point cloud data. Information on targets detected by the target detection unit 201 is input to the prediction unit 202, the determination unit 204, and the traffic condition agreement determination unit 207.

[0058] The prediction unit 202 predicts the future movement of the target detected by the target detection unit 201, and outputs the prediction result to the determination unit 204. The prediction unit 202 also generates prediction information and temporarily stores it in the prediction result storage unit 203. The prediction information includes the time when the prediction is made as a time code.

[0059] The determination unit 204 determines whether or not autonomous driving is possible based on information about the target detected by the target detection unit 201 and the prediction result output from the prediction unit 202. The determination made by the determination unit 204 includes determining whether or not starting is possible and whether or not entering an intersection is possible. If it is determined that the vehicle 20 can be driven autonomously, for example, if it is determined that entering an intersection or starting is possible, the determination unit 204 inputs the determination to the driving plan generation unit 208. If it is not determined that the vehicle 20 can be driven autonomously and it is determined that remote assistance is necessary, the determination unit 204 inputs the determination result to the remote assistance request unit 205.

[0060] When the judgment result that remote support is necessary is input from the judgment unit 204, the remote support request unit 205 transmits a remote support request to the remote support terminal 30. The remote support request unit 205 also transmits information for the remote supporter 40 to perform remote support to the remote support terminal 30. The transmitted information for performing remote support contains the time when the information was transmitted as a time code. The transmitted information is displayed on the display device 32, and the remote supporter 40 can make a judgment on remote support by referring to the displayed information.

[0061] The remote support receiving unit 206 receives remote support input by the remote supporter 40 to the input device 33. The received remote support includes the time code of the information displayed on the display device 32 when the remote supporter 40 input the remote support to the input device 33. From the time code, it is possible to know when the remote supporter 40 input the remote support based on the acquired information.

[0062] When the remote assistance receiving unit 206 receives the remote assistance, the traffic condition matching determination unit 207 acquires, from the prediction result storage unit 203, prediction information having a time code that matches the time code included in the remote assistance, among prediction information that predicts the position of the target at the time of receiving the remote assistance. The predicted position of the target is obtained from the prediction information. In addition, the traffic condition matching determination unit 207 acquires the actual position of the target by acquiring sensor information from the recognition sensor of the external sensor 22. Then, based on a comparison between the actual position of the target and the predicted position, it is determined whether the traffic conditions match. If the traffic conditions do not match, the traffic condition matching determination unit 207 inputs the determination result to the remote assistance request unit 205 so that another remote assistance request can be made. If the traffic conditions match, the traffic condition matching determination unit 207 inputs the determination result to the driving plan generation unit 208 in order to control the vehicle 20 according to the remote assistance.

[0063] The driving plan generating unit 208 generates a driving plan for the vehicle 20 based on map information, the current position, and vehicle speed information. The map information is stored in advance in the memory 21b, for example. The current position is acquired from a position sensor of the external sensor 22. The vehicle speed information is acquired from the internal sensor 23. The driving plan is a plan for driving the vehicle 20 according to the map, and includes information necessary for automatic driving of the vehicle 20, such as a target driving route and a target vehicle speed. When a traffic condition match determination input is received from the traffic condition match determination unit 207, the driving plan generating unit 208 generates a driving plan so that driving is performed according to remote assistance.

[0064] The vehicle control unit 209 controls the vehicle 20 so that the vehicle 20 travels in accordance with the travel plan generated by the travel plan generating unit 208 .

[0065] 6. Flowchart 6-1. First example As an example of a situation where remote support is required, the process performed by the run management device 21 when entering an intersection will be described. Fig. 10 is a flowchart showing an example of the process performed by the run management device 21. The program 21c stored in the memory 21b of the run management device 21 causes the processor 21a to execute a series of processes shown in the flowchart of Fig. 10.

[0066] When the vehicle 20 approaches an intersection, the processor 21a of the driving management device 21 starts a series of processes. The driving management device 21 may start the series of processes with the vehicle 20 stopped before the intersection, or may start the series of processes after slowing down the vehicle 20 as it approaches the intersection. In step S211, the processor 21a acquires sensor information from the recognition sensor and detects the position and speed of the target. When the position and speed of the target are detected, the process proceeds to step S212.

[0067] In step S212, the processor 21a predicts the future motion of the target based on the information detected in step S211. The prediction of the motion of the target performed by the processor 21a in step S212 includes the generation of prediction information.

[0068] In step S213, the processor 21a determines whether the vehicle 20 is allowed to start or enter the intersection. This determination is made based on the sensor information acquired from the recognition sensor and the result of the prediction generated in step S212. If it is determined that the vehicle 20 is allowed to start or enter the intersection (step S213; Yes), the process proceeds to step S222. If the vehicle 20 is not allowed to start or enter the intersection (step S213; No), the process proceeds to step S214.

[0069] In step S214, the processor 21a determines whether or not to request remote assistance. If the vehicle cannot autonomously determine whether to start or enter an intersection, remote assistance is requested. If remote assistance is requested (step S214; Yes), the process proceeds to step S215. If remote assistance is not requested (step S214; No), the process returns to step S211.

[0070] In step S215, the processor 21a transmits a remote support request and temporarily stores the prediction information generated in step S211. After the process of step S215 is performed, the process proceeds to step S216.

[0071] In step S216, the processor 21a transmits information for the remote supporter 40 to make a decision on remote support to the remote support terminal 30. The transmitted information includes sensor information acquired from the recognition sensor. The transmitted information is displayed on the display device 32. After the information is transmitted, the process proceeds to step S217.

[0072] In step S217, the processor 21a receives the remote support input by the remote supporter 40 to the input device 33 from the remote support terminal 30. When the remote support is received, the process proceeds to step S218.

[0073] In step S218, the processor 21a determines whether the received remote assistance instructs starting or entering. If the remote assistance instructs starting or entering (step S218; Yes), the process proceeds to step S219. If the remote assistance does not permit starting or entering (step S218; No), the process returns to step S215.

[0074] In step S219, the processor 21a determines whether the traffic conditions at the time of receiving the remote assistance match those at the time of transmitting the remote assistance request. The determination is made by comparing the sensor information acquired by the recognition sensor with the prediction information temporarily stored in step S215. If the traffic conditions match (step S219; Yes), the process proceeds to step S222. If the traffic conditions do not match (step S219; No), the process proceeds to step S220.

[0075] In step S220, the same process as in step S211 is performed. After the position and speed of the target are detected, the process proceeds to step S221.

[0076] In step S221, the same process as in step S212 is performed. Once the prediction is performed, the process returns to step S215.

[0077] In step S222, the processor 21a generates a driving plan for starting or entering an intersection. The driving plan includes a target position and a target speed of the vehicle 20. When the driving plan is generated, the process proceeds to step S223.

[0078] In step S223, the processor 21a controls the vehicle 20 so that the vehicle 20 starts moving or enters the intersection. When step S223 is executed, the series of processes ends.

[0079] 6-2. Second Example 11 is a flowchart showing another example of the process performed by the runway management device 21 when entering an intersection. The program 21c stored in the memory 21b of the runway management device 21 causes the processor 21a to execute a series of processes shown in the flowchart of FIG.

[0080] The processes from step S231 to step S239, step S242, and step S243 are similar to the processes from step S211 to step S219, step S222, and step S223 in Fig. 10. However, in step S239, if the traffic conditions do not match (step S239; No), the process returns to step S231.

[0081] In this way, if it is determined that the traffic conditions do not match between the time the remote assistance is received and the time the remote assistance request is sent, remote assistance may be requested again as shown in Figure 10, or, as shown in Figure 11, the driving management device 21 may again determine whether or not to depart or enter, and remote assistance may be requested again if the driving management device 21 cannot make a decision autonomously.

[0082] 7. Comparison In the above description, the driving management device 21 determines whether the traffic conditions match or do not match the traffic conditions at the time when the remote support request was sent, using the time when the remote support was received as the determination time. However, the time when the driving management device 21 determines whether the traffic conditions match or do not match does not have to be the time when the remote support is received. For example, the driving management device 21 may determine whether the traffic conditions match or do not match the traffic conditions at the time when the remote support request was sent when a predetermined time has elapsed since the driving management device 21 sent the remote support request. In this case, for example, the predetermined time is assumed to be the average time or maximum time of delay occurring between the vehicle 20 and the remote support terminal 30.

[0083] When the time when the remote assistance was received is set as the judgment time, the driving management device 21 judges whether the traffic conditions match or not by comparing the actual position of the target at the time when the remote assistance was received with the predicted position of the target at the time when the remote assistance was received. Similarly, by comparing the actual position of the target at the time when the judgment is made with the predicted position of the target at the time when the judgment is made, it is possible to judge whether the traffic conditions match or not with an arbitrary time as the judgment time.

[0084] In addition, by using the prediction information for obtaining the predicted position as prediction information generated at any point in the past, the driving management device 21 can determine whether the traffic conditions match or do not match not only at the time the remote assistance request is sent, but also at any point in the past.

[0085] However, when judging whether the traffic conditions at a certain judgment time point match those at a certain past time point, if the time from that past time point to the judgment time point is longer than the time period for which the prediction information is generated, the prediction information to be compared has not been generated. For example, if prediction information including predictions up to 50 seconds ahead is generated as prediction information, and an attempt is made to judge whether the traffic conditions match those at a past time point more than 50 seconds before the judgment time point, no prediction information that predicted the judgment time point has been generated at that past time point. In such a case, the situation at the judgment time point is deemed to be outside the prediction range, and the traffic conditions are determined to not match.

[0086] In the following, even when the driving management device 21 is described as determining whether the traffic conditions at the time when it receives remote assistance and the time when it sends a remote assistance request, this can be replaced with a determination of whether the traffic conditions at any determination time match those at any past time.

[0087] 8. Modifications of the embodiment 8-1. The first variant Fig. 12 shows an example of the configuration of the driving management device 21 in the first modified example. In the first modified example, the driving management device 21 includes an image processing unit 210 in addition to the functional units described in Fig. 9. The image processing unit 210 is a functional unit that processes the camera image included in the sensor information so that information about the target is projected and superimposed on the camera image. The processed information is transmitted to the remote support terminal 30 and displayed on the display device 32 together with the information transmitted by the remote support request unit 205 or in place of the information transmitted by the remote support request unit 205.

[0088] The image processing unit 210 may process the image so that, for example, three-dimensional information about the target detected by the target detection unit 201 using a laser or LiDAR is displayed superimposed on the camera image. Alternatively, the image may be processed so that the position coordinates and moving speed of the target are displayed superimposed on the camera image. The remote supporter 40 can provide more accurate remote support by making a decision on remote support with reference to the image processed in this way.

[0089] Alternatively, the image processing unit 210 may process the image so that information about the target whose actual position and predicted position are determined by the traffic condition agreement determination unit 207 to be inconsistent is projected onto the camera image. At this time, for example, the image may be processed so that the target whose position is determined to be inconsistent is highlighted, or the image may be processed so that three-dimensional information about the target whose position is determined to be inconsistent is displayed in the camera image. The processed image is transmitted to the remote assistance terminal 30 when remote assistance is requested again.

[0090] Here, the target whose position does not match is a target whose actual position does not exist within the range of the predicted position, a target whose predicted presence probability at the actual position is less than a predetermined value, or a target whose actual position is not within the range of the predicted position. i In this way, by making the information about the target that exists in a position different from the prediction stand out and making a remote support request again, the time required for the remote supporter 40 to make a decision on remote support can be shortened.

[0091] 8-2. The second variant The second modification is a modification that can be taken when remote assistance is performed when the vehicle 20 enters an intersection. Even if remote assistance is determined to be effective at the start of entering the intersection, if it is determined that the traffic conditions have changed after entering the intersection, there is a possibility that a new situation that the remote supporter 40 did not intend at the time of remote assistance has occurred. Therefore, the driving management device 21 judges whether the traffic conditions match or do not match not only when receiving remote assistance, but also continuously after the vehicle 20 enters the intersection. In other words, the judgment of whether the traffic conditions match or do not match is repeated at a predetermined period even after entering the intersection. At this time, the predicted position to be compared with the actual position of the target may be obtained from predicted information generated at a time a predetermined time before the judgment time, or may be obtained from predicted information generated at the time of transmitting the remote assistance request or the time of receiving remote assistance.

[0092] If it is determined that the traffic conditions have changed after entering the intersection, the driving management device 21 does not allow the vehicle 20 to continue entering the intersection and requests remote assistance again. At this time, the driving management device 21 stops or slows down the vehicle 20 to ensure safety. This determination of whether the traffic conditions match or do not match continues until the vehicle 20 has completely entered the intersection. This makes it possible to increase the safety of the vehicle 20 when passing through the intersection. After the vehicle 20 has completely entered the intersection, it is considered safer to allow the vehicle 20 to pass through the intersection rather than stopping or slowing down, so this determination of whether the traffic conditions match ends and the driving management device 21 controls the vehicle 20 to pass through the intersection by autonomous driving.

[0093] 8-3. The third variant In the third modified example, when determining whether the traffic conditions match, the actual position of the target is not compared with the predicted position, but the predicted positions are compared with each other. The two predicted positions that predict the same future time are compared. One is a predicted position obtained from prediction information generated when a remote assistance request is transmitted, and the other is a predicted position obtained from prediction information generated when remote assistance is received. The same future time may be, for example, a time that is a predetermined time ahead of the determination time, or a time when the vehicle 20 is expected to reach a certain point on the road. For example, when remote assistance is performed when passing through an intersection, the same future time may be a time when the vehicle 20 is expected to reach the center of the intersection or a time when the vehicle 20 is expected to finish passing through the intersection. At this time, the traffic condition match determination unit 207 can determine which time the same future time is to be by acquiring a predicted time until the vehicle 20 reaches a certain point on the road from the driving plan generation unit 208.

[0094] In the third modified example, the predicted position of the target is also expressed by a probability density or a range. If the predicted position of the target is expressed by a range, the coincidence of the predicted positions means that the predicted position generated at the time of receiving the remote support request is included in the range of the predicted position generated at the time of transmitting the remote support request. Alternatively, the predicted positions may be determined to coincide when a certain range or more of the predicted positions generated at the time of receiving the remote support is included in the range of the predicted positions generated at the time of transmitting the remote support request. If the predicted position of the target is expressed by a probability density, the two predicted positions are determined to coincide when the magnitude of overlap between the probability density distribution generated at the time of receiving the remote support request and the probability density distribution generated at the time of transmitting the remote support request is greater than a predetermined amount.

[0095] FIG. 13 shows an example of predicted positions to be compared in the third modified example. Here, the predicted positions of the targets are shown as ranges. The dotted ellipses show the predicted positions of the targets generated when receiving remote assistance, and the shaded ellipses show the predicted positions of the targets generated when transmitting a remote assistance request. In addition, these two predicted positions are both predicted positions of the targets at the time when the vehicle 20 is expected to finish passing through the intersection. In the example of FIG. 13, the dotted ellipses are included in the range of the shaded ellipses. In other words, since the two predicted positions match, it is expected that the traffic conditions will continue to match until the vehicle 20 finishes passing through the intersection.

[0096] For example, in a scene where the vehicle 20 passes through an intersection, the action of the vehicle 20 triggered by the remote support instructing the vehicle to enter the intersection is a continuous action from when the vehicle 20 starts entering the intersection to when the vehicle 20 completes passing through the intersection. In the third modified example, by comparing the predicted positions in this manner, it is possible to confirm that the remote support is effective not only at the time when the action according to the remote support is started, but also until the action is completed, for example, until the vehicle finishes passing through the intersection. This allows the effectiveness of the remote support to be determined more accurately.

[0097] 9. Summary As described above, when the driving management device in the embodiment of the present disclosure receives remote assistance, it confirms that the traffic conditions at the time of receiving the remote assistance match the traffic conditions at the time of sending the remote assistance request, and then executes vehicle control according to the remote assistance. This prevents the vehicle from driving according to remote assistance that is no longer valid due to changes in conditions caused by delays, thereby improving the safety of vehicle driving.

[0098] Furthermore, in the embodiment of the present disclosure, the effectiveness of remote assistance is determined not simply based on the length of the communication delay, but based on whether or not the traffic conditions match. This improves the accuracy of the determination, and makes it possible to determine that remote assistance is ineffective when the situation has changed significantly even if the communication delay is small. Conversely, it is possible to employ remote assistance when the situation has changed little even if the communication delay is large, thereby improving the efficiency of vehicle travel. In this way, it is possible to increase robustness against delays and appropriately respond to delays. [Explanation of symbols]

[0099] 10 Server 20 Self-driving vehicles 21 Driving management device 21a Processor 21b Memory 21c Program 22 External Sensor 23 Internal Sensor 24 Actuator 25 Communication Equipment 30 Remote support terminal 31 Computer 31a Processor 31b Memory 31c Program 32 Display device 33 Input Devices 35 Communication Equipment 40 Remote Supporter 50 vehicles 60 vehicles 70 Pedestrians 80 Pedestrians 100 Remote Support System 201 Target detection unit 202 Prediction Department 203 Prediction result memory unit 204 Judgment Department 205 Remote Support Request Department 206 Remote Support Receiving Unit 207 Traffic Condition Consistency Judgment Unit 208 Driving plan generation unit 209 Vehicle control unit 210 Image processing section

Claims

1. A driving management device that manages driving of a vehicle having an autonomous driving function, Sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; Sequentially generating prediction information representing future positions of targets present around the vehicle based on the sequentially acquired sensor information; Sending a remote support request to a remote supporter; receiving remote support from the remote supporter in response to the remote support request; and upon receiving confirmation that a position of a target obtained from sensor information acquired at a determination time point after the time point of receiving the remote assistance coincides with a position of a target obtained from prediction information generated at the time point of transmitting the remote assistance request and with the determination time point being the future time point, the vehicle is caused to autonomously travel in accordance with the remote assistance. A driving management device characterized by the above.

2. The travel management device according to claim 1, The determination time is the time when the remote support is received. A driving management device characterized by the above.

3. A driving management method for managing driving of a vehicle having an autonomous driving function, comprising: Sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; Sequentially generating prediction information representing future positions of targets present around the vehicle based on the sequentially acquired sensor information; Sending a remote support request to a remote supporter; receiving remote support from the remote supporter in response to the remote support request; and upon confirmation that a position of a target obtained from sensor information acquired at a determination time point after the time point of receiving the remote assistance coincides with a position of a target obtained from prediction information generated at the time point of transmitting the remote assistance request and with the determination time point being the future time point, causing the vehicle to autonomously travel in accordance with the remote assistance. A driving management method comprising:

4. A program for managing driving of a vehicle having an autonomous driving function, Sequentially acquiring sensor information from a recognition sensor mounted on the vehicle; Sequentially generating prediction information representing future positions of targets present around the vehicle based on the sequentially acquired sensor information; Sending a remote support request to a remote supporter; receiving remote support from the remote supporter in response to the remote support request; and upon confirmation that a target position obtained from sensor information acquired at a determination time point after the time point at which the remote assistance is received matches a target position obtained from prediction information generated at the time point at which the remote assistance request is transmitted and which sets the determination time point as the future time point, the vehicle is caused to autonomously travel in accordance with the remote assistance. A program characterized by:

Images