Information processing method and information processing system

Abstract

The information processing method includes the following processes: determining whether an emergency stop of a mobile body performing autonomous driving has occurred (S11); if it is determined that an emergency stop has occurred (Yes in S11), simulating safety control based on the driving condition history of the mobile body; based on the autonomous driving control shown in a log of the automatic driving system and the simulated safety control, determining the process that is the cause of the emergency stop from among the processes for autonomous driving control shown in the log (S12); obtaining a condition candidate corresponding to the determined process from a database related to conditions that may cause an emergency stop (S13); prompting an interface for an operator to input a condition candidate corresponding to the condition shown in the log (S14); and determining whether the autonomous driving control performed by the automatic driving system can be restarted based on the input condition candidate (S15).

Description

Technical Field

[0001] The present disclosure relates to an information processing method and the like that is executed when an autonomously traveling moving object makes an emergency stop. Background Art

[0002] With the development of autonomous driving technology in recent years, it is expected that transportation services can be provided by autonomous mobile bodies in an unmanned manner. In the actual operation of such services, it is required that operators can respond remotely in emergencies. In particular, the risk of accidents is high when recovering from an emergency stop, so the following architecture is required: the corresponding operator can understand the situation remotely and decide whether to stop the service or restart the driving according to whether there is a problem. For example, in the past, there was a method of remotely grasping the external condition of a mobile body by confirming the image of a vehicle-mounted camera (see patent document 1).

[0003] In addition, as an invention focusing on the processing of an autonomous driving system, there is the following display method for a moving body: obstacles detected by the autonomous driving system and the control of the moving body performed are visualized, so that the occupants can understand the causes of sudden movements based on automatic control and reassure the occupants (refer to patent document 2).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: (Japan) Patent No. 4861158

[0007] Patent Document 2: (Japanese) Patent Publication No. 2017-187839 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, in the method described in Patent Document 1, the reason why the automatic driving system stops suddenly is unknown only by the surrounding situation shown in the provided video. In addition, Patent Document 1 does not disclose or hint about the confirmation of the reason.

[0010] In addition, the display method for a mobile object described in Patent Document 2 is used to help a remote operator understand the situation handled by the automatic driving system that has suddenly moved. However, the judgment of whether there is actually a problem in resuming the driving of the mobile object depends on the operator. Therefore, it is difficult to effectively improve the safety of resuming driving.

[0011] The present disclosure provides an information processing method and an information processing system that can more reliably improve the safety of resuming driving when a moving object that is autonomously driving stops suddenly to avoid a collision with an obstacle.

[0012] Means for solving problems

[0013] An information processing method involved in one aspect of the present disclosure is an information processing method executed by a processor, comprising the following processing: determining whether an autonomously traveling mobile body has generated an emergency stop for avoiding a conflict with an obstacle; if it is determined that the emergency stop has occurred, simulating autonomous driving control, i.e., safety control, for autonomous driving so that the mobile body avoids a conflict with the obstacle without performing the emergency stop based on the driving condition history of the mobile body; obtaining a log of an automatic driving system that performs the autonomous driving control of the mobile body; determining a process that is a cause of the emergency stop from processes for autonomous driving control shown in the log based on the autonomous driving control shown in the log and the simulated safety control; obtaining a condition candidate corresponding to the determined process from a database related to conditions that may cause an emergency stop; prompting an interface for an operator to input the condition candidate corresponding to the condition shown in the log; and determining whether the autonomous driving control performed by the automatic driving system can be restarted based on the input condition candidate.

[0014] In addition, an information processing system involved in one embodiment of the present invention is an information processing system including one or more computers having processors, wherein the processor determines whether an autonomously traveling mobile body has generated an emergency stop for avoiding a conflict with an obstacle, and when it is determined that an emergency stop has occurred, based on the driving condition history of the mobile body, simulates autonomous driving control, i.e., safety control, for autonomous driving so that the mobile body avoids a conflict with the obstacle without performing the emergency stop, obtains a log of an automatic driving system that executes the autonomous driving control of the mobile body, determines a process that is a cause of the emergency stop from among processes for autonomous driving control shown in the log based on the autonomous driving control shown in the log and the simulated safety control, obtains a condition candidate corresponding to the determined process from a database related to conditions that may cause an emergency stop, prompts an interface for an operator to input the condition candidate corresponding to the condition shown in the log, and determines whether the autonomous driving control performed by the automatic driving system can be restarted based on the input condition candidate.

[0015] In addition, these inclusive or specific methods may also be implemented by devices, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of devices, systems, integrated circuits, methods, computer programs, and recording media.

[0016] Effects of the Invention

[0017] The information processing method and the information processing system according to the present disclosure can more reliably improve the safety of resuming driving when an autonomously traveling moving object makes an emergency stop to avoid a collision with an obstacle. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an overall structural diagram of the architecture including a mobile body having an autonomous driving system and a remote server involved in the implementation method.

[0019] Figure 2A This is a schematic diagram for explaining an image generated by the remote server when a process performed by the detection unit is identified as the cause of the emergency stop.

[0020] Figure 2B This is a schematic diagram for explaining an image generated by the remote server when the process performed by the detection unit is identified as the cause of the emergency stop of the moving body.

[0021] Figure 3A This is an image generated by the remote server when the processing performed by the detection unit is identified as the cause of the emergency stop of the moving body.

[0022] Figure 3B This is a schematic diagram for explaining an image generated by the remote server when the process performed by the detection unit is identified as the cause of the emergency stop of the moving body.

[0023] Figure 4A This is an example of an image generated by the remote server when the process performed by the prediction unit is determined as the cause of the emergency stop of the moving body.

[0024] Figure 4B This is a schematic diagram for explaining an image generated by the remote server when the process performed by the prediction unit is determined as the cause of the emergency stop of the moving body.

[0025] Figure 5A This is a schematic diagram for explaining an image generated by the remote server when the process performed by the prediction unit is determined as the cause of the emergency stop of the moving body.

[0026] Figure 5B This is a schematic diagram for explaining an image generated by the above-mentioned image generating unit when the processing performed by the predicting unit is determined as the cause of the emergency stop of the above-mentioned moving body.

[0027] Fig. 6AThis is a schematic diagram for explaining an image generated by the remote server when the process performed by the determination unit is determined as the cause of the emergency stop of the moving body.

[0028] Figure 6B This is a schematic diagram for explaining an image generated by the remote server when the process performed by the determination unit is determined as the cause of the emergency stop of the moving body.

[0029] Fig. 7A This is a schematic diagram for explaining an image generated by the remote server when the process performed by the determination unit is determined as the cause of the emergency stop of the moving body.

[0030] Figure 7B This is a schematic diagram for explaining an image generated by the remote server when the process performed by the determination unit is determined as the cause of the emergency stop of the moving body.

[0031] Figure 8 This is an example of a UI that is controlled by a user interface (hereinafter referred to as UI) control unit via the above-mentioned remote server and presented to an operator.

[0032] Fig. 9 This is an example of a UI presented based on an operator's input.

[0033] Fig.10 This is an example of a UI presented based on an operator's input.

[0034] Fig.11 This is a flowchart for explaining a series of processes of the information processing method involved in the embodiment. DETAILED DESCRIPTION

[0035] In order to solve the above-mentioned problems, an information processing method involved in one aspect of the present invention is an information processing method executed by a processor, which determines whether an autonomously traveling mobile body has generated an emergency stop for avoiding a conflict with an obstacle. When it is determined that the emergency stop has occurred, based on the driving condition history of the mobile body, an autonomous driving control, i.e., a safety control, is simulated for autonomous driving so that the mobile body avoids the conflict with the obstacle without performing the emergency stop, a log of an automatic driving system that performs the autonomous driving control of the mobile body is obtained, and based on the autonomous driving control shown in the log and the simulated safety control, a process that is the cause of the emergency stop is determined from the processes for autonomous driving control shown in the log, and a condition candidate corresponding to the determined process is obtained from a database related to conditions that may cause an emergency stop, an interface is prompted for an operator to input the condition candidate corresponding to the condition shown in the log, and based on the input condition candidate, it is determined whether the autonomous driving control performed by the automatic driving system can be restarted.

[0036] This makes it possible to more reliably improve the safety of resuming travel when the autonomously traveling moving object stops suddenly to avoid a collision with an obstacle.

[0037] In addition, for example, the safety control and the autonomous driving control shown in the log may be compared to obtain the start time of the safety control for executing the safety control in the autonomous driving control of the mobile body, that is, the safety control time, and determine the processing as the cause of the emergency stop based on the result of the determination of whether the result of the processing required for the execution of the safety control is output before the safety control time. More specifically, for example, the processing required for the execution of the safety control may also be at least one of a detection processing for obtaining a detection result indicating the surrounding conditions of the mobile body, a prediction processing for predicting the surrounding conditions of the mobile body using the detection result, and a judgment processing for determining a driving path corresponding to the predicted surrounding conditions.

[0038] Thus, whether autonomous driving control can be resumed is determined based on whether autonomous driving control is possible or the difference from ideal safe control, thereby making it possible to more reliably improve the safety of driving resumption.

[0039] In addition, for example, the interface may also include an image generated using the output of the determined processing and the observation data input to the autonomous driving system. In addition, the interface may also include an image generated further using the output of the safety control moment in the safety control. In addition, the interface may also be an interface for inputting the candidate of the condition corresponding to the condition shown in the log for a moment after the safety control moment.

[0040] This enables the operator to determine the appropriateness of the processing of the automatic driving system based on the surrounding conditions of the autonomously traveling moving object when it makes an emergency stop, or observation data indicating the surrounding conditions.

[0041] In addition, for example, whether the autonomous driving control can be restarted may be determined based on the value of a parameter that is set for the input condition candidate and indicates the safety or risk level of the execution of the autonomous driving control. In addition, the parameter may be based on the delay time length of the output of the process determined from the safety control moment. In addition, the parameter may be related to the time corresponding to the input condition candidate, and the condition candidate may be input for the time until the safety control moment.

[0042] This makes it easy to determine whether autonomous driving after an emergency stop can be resumed without relying on the operator, and the safety of resuming driving can be more reliably improved.

[0043] In addition, for example, the safety control may be a control for stopping the moving body at a parking position based on the emergency stop or at a position closer to the parking position. In addition, for example, the safety control may be a control for causing the moving body to travel while avoiding the obstacle.

[0044] In order to solve the above-mentioned problems, an information processing system involved in one embodiment of the present invention is an information processing system including one or more computers having processors, wherein the processor determines whether an autonomously traveling mobile body has generated an emergency stop for avoiding a conflict with an obstacle, and when it is determined that an emergency stop has occurred, based on the driving condition history of the mobile body, simulates autonomous driving control, i.e., safety control, for autonomous driving so that the mobile body avoids a conflict with the obstacle without performing the emergency stop, obtains a log of an automatic driving system that executes the autonomous driving control of the mobile body, determines a process that is the cause of the emergency stop from among processes for autonomous driving control shown in the log based on the autonomous driving control shown in the log and the simulated safety control, obtains a condition candidate corresponding to the determined process from a database related to conditions that may cause an emergency stop, prompts an interface for an operator to input the condition candidate corresponding to the condition shown in the log, and determines whether the autonomous driving control performed by the automatic driving system can be restarted based on the input condition candidate.

[0045] As a result, when an autonomously traveling moving object makes an emergency stop to avoid a collision with an obstacle, traveling can be resumed with more reliable improvement in safety.

[0046] In addition, these inclusive or specific methods may also be implemented by devices, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of devices, systems, integrated circuits, methods, computer programs, and recording media.

[0047] Hereinafter, embodiments of the information processing method and the information processing system according to the present disclosure will be described with reference to the accompanying drawings.

[0048] The numerical values, shapes, materials, structural elements, arrangement positions and connection methods of structural elements, steps, and the order of steps shown in the embodiments are examples and do not limit the spirit of the present invention.

[0049] [structure]

[0050] Figure 1 This is an overall structural diagram of the architecture including a mobile object having an autonomous driving system and a remote server involved in this embodiment.

[0051] The information processing method involved in the present disclosure includes Figure 1 The illustrated architecture of the autonomously traveling vehicle 100 and the remote server 200 wirelessly connected to the vehicle 100 is executed by a processor.

[0052] (Moving body)

[0053] The mobile object 100 includes an automatic driving system 110 , a receiving unit 120 , an observation unit 11 , and a mobile object control unit 12 .

[0054] The observation unit 11 observes by sensing the surroundings of the moving body 100. The observation unit 11 is implemented, for example, using a camera, Radar (Radio detecting and ranging) or LiDAR (Light detection and ranging or Laser imaging detection and ranging). Observation data representing the results of the observation are provided from the observation unit 11 to the automatic driving system 110.

[0055] The mobile body control unit 12 controls the travel of the mobile body 100 according to the instructions related to the driving control from the automatic driving system 110. For example, the mobile body control unit 12 is implemented using a circuit that controls the steering angle and acceleration and deceleration of the mobile body. In addition, information indicating the content of the control (hereinafter also referred to as steering / acceleration and deceleration information) is provided from the mobile body control unit 12 to the automatic driving system 110.

[0056] The automatic driving system 110 includes a detection unit 111 , a prediction unit 112 , and a determination unit 113 as functional components.

[0057] The detection unit 111 processes the observation data such as the image or point group acquired by the observation unit 111 to detect obstacles and output information on the type, size, relative position and relative speed of the obstacles to the moving object 100 .

[0058] The prediction unit 112 predicts the movement path of the detected obstacle by processing the information obtained from the detection unit 111 and the behavior prediction technology, and outputs the predicted movement path.

[0059] The judgment unit 113 dynamically plans the path of the mobile body 100 by processing the predicted movement path of the obstacle obtained from the prediction unit 112 and the steering / acceleration / deceleration information of the mobile body 100 obtained from the mobile body control unit 12. The judgment unit 113 determines the content of driving control including at least one of the next steering control and acceleration / deceleration control for making the mobile body 100 travel along the planned path (hereinafter also referred to as the planned path), and outputs an instruction including the determined content of driving control to the mobile body control unit 12. In addition, there is an upper limit predetermined as a limit of normal driving for the steering and acceleration / deceleration control amount instructed by the judgment unit 113. In the case where a control amount exceeding the upper limit, such as a control exceeding a prescribed deceleration, is required, the judgment unit 113 instructs braking for emergency stop. If such an instruction for emergency stop is issued, the driving control for driving the mobile body 100 based on the mobile body control unit 12 corresponding to the instruction from the judgment unit 113 is not executed until the prescribed procedure of the remote server 200 is completed.

[0060] The receiving unit 120 receives the driving restart permission signal by wireless communication with the transmitting unit 240 provided in the remote server 200. When the driving restart permission signal received by the receiving unit 120 indicates that driving restart is possible, the moving body 100 returns to a state where the moving body control unit 12 performs driving control according to the instruction from the determination unit 113.

[0061] The mobile body control unit 12, the automatic driving system 110 and the receiving unit 120 can be implemented using one or more ECUs (Electronic Control Units) included in various driving control systems included in the vehicle network system constructed on the vehicle network possessed by the mobile body 100. ECU is, for example, a device including a processor (microprocessor), digital circuits such as memory, analog circuits, communication circuits, etc. The memory is ROM (Read-Only Memory) and RAM (Random Access Memory), which can store control programs (computer programs) executed by the processor. The ECU provides the functions of the above-mentioned structural elements by the processor acting according to the control program. A computer program refers to a program that is a combination of multiple command codes for the processor in order to achieve a specified function.

[0062] (Remote Server)

[0063] The remote server 200 is an example of an information processing system including one or more computers having a processor. When the autonomously traveling mobile body 100 makes an emergency stop, the remote server 200 provides the operator with information that can be used to determine whether the emergency stop is caused by an abnormality in the automatic driving system 110 or is necessary due to the situation. In addition, the remote server 200 determines whether the autonomous driving can be resumed based on the input from the operator, and sends a signal indicating the determination result of whether the driving can be resumed to the mobile body 100.

[0064] The remote server 200 is, for example, a device including a processor, a memory, and a communication circuit. The memory is a ROM and a RAM, and can store a control program (computer program) executed by the processor. The remote server 200 includes a log recording unit 210, a first information processing unit 220, a second information processing unit 230, a transmission unit 240, an output device 21, and an input device 22. In addition, the various structural elements of the remote server 200 may be distributed in a plurality of servers to form an information processing system.

[0065] The output device 21 outputs information to the outside of the remote server 200. Examples of the output device 21 include a liquid crystal display and a speaker. The input device 22 receives input from the outside of the remote server 200. Examples of the input device 22 include a keyboard, a mouse, and a microphone. In addition, the output device 21 and the input device 22 may also be provided by a touch screen.

[0066] The log recording unit 210 obtains and records the sensor log 211, the driving log 212, and the processing log 213 through wireless communication with the automatic driving system 110. The sensor log 211 is a history of observation data such as an image or a point group input to the detection unit 111. The driving log 212 is a history of speed or position information of the mobile body 100 input to the judgment unit 113. The driving log 212 may also include an action history of actuators related to the driving of the mobile body 100 (such as brakes, accelerators, or steering actuators). The processing log 213 is a history of processing results of the detection unit 111, the prediction unit 112, and the judgment unit 113. In addition, for the judgment unit 113, the history of the planned path is also included in the processing log 213.

[0067] The first information processing unit 220 includes an operation detection unit 221, a time calculation unit 222, and a cause identification unit 223 as functional components. These components are implemented by a processor or the like.

[0068] The operation detection unit 221 refers to the processing log 213, monitors the issuance of the emergency stop instruction of the mobile body 100 by the judgment unit 113 of the automatic driving system 110, and detects the occurrence of the emergency stop. If the occurrence of the emergency stop is detected, the detection result is output as a trigger for subsequent processing performed by the time calculation unit 222, etc. The instruction of the emergency stop can also be indicated by a mark assigned to the instruction, or it can be judged from the content of the instruction of the braking control (the magnitude of the brake force, the distribution of the brake force, etc.). In addition, if the mobile body 100 is equipped with an emergency stop component such as a collision damage reduction brake that operates differently from the emergency stop instructed by the judgment unit 113, the operation detection unit 221 refers to the driving log 212, detects the operation of the emergency stop component, and outputs the detection result as the above-mentioned trigger.

[0069] The time calculation unit 222 calculates the safety control limit time from the driving log 212. The safety control in this embodiment means control of stopping the moving body 100 by normal autonomous driving control (that is, without emergency stop) at the position where the moving body 100 stops due to an emergency stop, and is a simulated control of the moving body 100 performed by the time calculation unit 222.

[0070] If the velocity v of the moving body at time t is t , set the position of the moving body to be l t , the time T when the moving body stops due to the emergency stop, and the maximum deceleration in the normal autonomous driving control is

[0071]

Number 1

[0072]

[0073] Then, the time T0 which becomes the solution of the minimization problem shown in the following formula 1 is calculated as the safety control limit time.

[0074]

Number 2

[0075]

[0076] In addition, Δt represents the processing delay time from when the automatic driving system 110 outputs the mobile body control of emergency stop to when the mobile body 100 actually starts the emergency stop. k ,l k+1 ) indicates that from position l k To position l k+1 For example, the Euclidean distance can be used.

[0077] The safety control limit time calculated as described above is an example of the safety control start time, i.e., the safety control time, which is the safety control start time for executing the safety control in the autonomous driving control of the mobile body 100. If the safety control limit time is expressed in another way, it can also be said that it is the latest time at which the control that can realize the safety control is started in the autonomous driving control of the mobile body 100. In the following, the safety control limit time is used for explanation, but the start time of the safety control may not necessarily be the latest time (i.e., the safety control limit time), and may be a time earlier than the latest time.

[0078] The cause determination unit 223 determines the cause of the emergency stop from among the processes for autonomous driving control executed in the automatic driving system 110, using the time T0 calculated by the time calculation unit 222 and the processing log 213. Specifically, the cause determination unit 223 determines the process as the cause of the emergency stop from the process performed by the detection unit 111 (hereinafter also referred to as the detection process), the process performed by the prediction unit 112 (hereinafter also referred to as the prediction process), and the process performed by the judgment unit 113 (hereinafter also referred to as the judgment process) based on the time from the time T0 to the time T when the moving body 100 stops due to the emergency stop. This process is described below.

[0079] As a premise, the emergency stop is performed to avoid a conflict with an obstacle, and at the time T' when the emergency stop has been initiated, at least the obstacle involved in the emergency stop is detected by the autonomous driving system 110. In addition, generally speaking, in the detection of the speed of the obstacle and the prediction of the moving path, an object tracking technology that is used to associate the obstacle with the timing is used. The autonomous driving system 110 involved in this embodiment also adopts the object tracking technology, and based on the information obtained by the prediction unit 112 or obtained from the outside using the same technology, the obstacle involved in the emergency stop detected at a certain time is regarded as the same obstacle as the obstacle detected at another time. Hereinafter, for the sake of explanation, the obstacle involved in the emergency stop will be referred to as a target.

[0080] Whether any of the detection process, prediction process, and judgment process is the cause of the emergency stop is determined based on the result of the process on the target.

[0081] The position of the target at time t output by the detection unit 111 is represented by detection(target,t). Here, if the detection unit 111 does not obtain the position of the target,

[0082] detection(target,t)=NULL

[0083] When detection(target, t) returns NULL after the safety control limit time T0, the cause identification unit 223 identifies the process performed by the detection unit 111 as the cause of the emergency stop.

[0084] When the moving body 100 does not determine the processing performed by the detection unit 111 as the cause of the emergency stop, the processing performed by the prediction unit 112 can be the cause of the emergency stop. The probability that the target at time t output by the prediction unit 112 moves to the position of the target at time T' where the emergency stop has started is represented by prediction (target, t).

[0085] When prediction(target, t) is lower than the threshold value after the safety control limit time T0, the cause determination unit 223 determines the processing performed by the prediction unit 112 as the cause of the emergency stop of the mobile body 100. As the position of the target at the time T' at which the emergency stop has been started, for example, the output detection(target, T') of the detection unit 111 as the previous stage processing can be used.

[0086] In the case where the processing performed by the detection unit 111 and the processing performed by the prediction unit 112 are not determined as the cause of the emergency stop of the mobile body 100, the processing performed by the judgment unit 113 can be the cause of the emergency stop. If the position of the target at the time T' when the emergency stop is started is predicted, the deceleration is ideally the maximum deceleration in the normal autonomous driving control.

[0087]

Number 3

[0088]

[0089] Therefore, it is determined whether the processing performed by the determination unit 113 is a cause of the emergency stop, based on whether the determination unit 113 has outputted an instruction to decelerate to satisfy the above-mentioned deceleration.

[0090] The deceleration of the moving object 100 at time t output by the determination unit 113 is represented by a t Here, when the moving object 100 is accelerated,

[0091] a t =NULL

[0092] In a t If NULL is returned after the safety control limit time T0, or if the following condition shown in equation 2 is satisfied

[0093]

Number 4

[0094]

[0095] In the case of a, the cause determination unit 223 determines the process performed by the determination unit 113 as the cause of the emergency stop. t The stopping distance of the moving body 100 is greater than the distance to the position where the moving body 100 stops due to the emergency stop.

[0096] Here, in a case where the processing performed by the judgment unit 113 is not determined as the cause of the emergency stop, the automatic driving system 110 and its judgment unit 113 output an instruction based on deceleration braking so that the vehicle can stop closer to the actual parking position without performing an emergency stop. However, even if an emergency stop is performed, it means that the cause of the emergency stop is outside the processing performed by the automatic driving system 110. In other words, it can be said that the moving body 100 is in a situation where it is impossible to continue driving under the driving control performed by the automatic driving system 110. In this embodiment, the moving body 100 in such a situation is treated as having a system error, and information indicating that "autonomous driving cannot be resumed" is output from the cause determination unit 223 to the sending unit 240.

[0097] As described above, the first information processing unit 220 detects the occurrence of an emergency stop and identifies the process that is the cause of the emergency stop from among the processes for autonomous driving control executed by the components of each function in the automatic driving system 110. The process identified as the cause is notified from the first information processing unit 220 to the second information processing unit 230.

[0098] The second information processing unit 230 includes an image generating unit 231, a status database (described later) Fig.11 232, UI control unit 233, and driving restart possibility determination unit 234 are also described as functional structural elements. In addition, these structural elements are realized by a processor or the like.

[0099] The image generation unit 231 generates an image based on the processing determined by the first information processing unit 220 as the cause of the emergency stop. For example, the image generation unit 231 uses the information extracted from the log recording unit 210 to generate an image indicating the situation in which the mobile body 100 has come to an emergency stop. The image generation unit 231, for example, generates an image corresponding to the time from the moment of the safety control limit to the moment when the mobile body stops due to the emergency stop. The generated image is presented to the operator who is concerned with the determination of whether the autonomous driving of the mobile body 100 can be resumed via the above-mentioned output device 21. With respect to this image, the operator is prompted to Figure 2A to Figure 7B This will be described later.

[0100] In the situation database 232 , candidates for situations that may cause an emergency stop due to processing by the automatic driving system 110 are registered.

[0101] Next, candidate examples of the above-mentioned situation related to the processing performed by the detection unit 111 are listed.

[0102] "The obstacle involved in the emergency stop is hidden in the blind spot when viewed from the moving object"

[0103] "The obstacle involved in the emergency stop is partially / temporarily hidden by other obstacles"

[0104] Furthermore, the following are examples of candidates of the above-mentioned situations related to the processing performed by the prediction unit 112 .

[0105] "The obstacle involved in the emergency stop has begun to move"

[0106] "The obstacle involved in the emergency stop was moving but stopped"

[0107] "The obstacle involved in the emergency stop is moving and has changed the direction of travel"

[0108] Furthermore, the following are examples of candidates of the above-mentioned situation related to the processing performed by the determination unit 113 .

[0109] "Other obstacles have entered the planned travel path of the moving object"

[0110] The candidate conditions are listed in advance by the developer or service provider based on the processing capability of the automatic driving system 110, and are registered in the condition database 232. If a large number of candidates are registered, it is difficult for the operator to select appropriately. On the other hand, if there are few candidates or there are omissions in the pre-listed conditions, there is a possibility that a condition that does not correspond to any candidate is often selected.

[0111] For example, when the number of candidates is equal to or greater than the first threshold, narrowing down of candidates or provisional selection of candidates as described later may be performed.

[0112] In addition, for example, when the number of candidates is less than the second threshold, when the number of selections that do not correspond to any candidate, i.e., the number of "no corresponding" selections, becomes greater than the third threshold, or when there is a candidate whose number of selections within a specified period is less than the fourth threshold, an alarm can also be notified to the operator.

[0113] Since it is believed that the above-mentioned problems will occur no matter how carefully the prior verification is performed, the update can be urged by the notification of the above-mentioned alarm. In addition, for the update of candidates, the image or system log of the situation where "no response" is selected can also be kept in a non-deleted manner. In addition, if possible, free descriptions made by the operator can also be kept. By using the information stored in this way, the selection of new candidates can be made more efficient. Furthermore, the autonomous driving system can also be improved based on the stored information. As a result, developers or service designers can better understand the scenarios for emergency stops, and the autonomous driving system is updated to avoid emergency stops, or the service route is changed to a route where emergency stops are less likely to occur, or the speed limit is reset to a speed where emergency stops are less likely to occur. As a result, improvements in the services provided by the autonomous driving system can be expected.

[0114] The above-described example of a situation in which an emergency stop may occur is realized using the output device 21 and the input device 22 and is used as a structural element of a UI described later.

[0115] The UI control unit 233 presents the processing in the automatic driving system 110 that has been identified as the cause of the emergency stop to the operator using the output device 21. The details of the UI will be described later using an example.

[0116] The driving restart possibility determination unit 234 determines whether the driving can be restarted based on the input result from the operator for the candidate of the situation that may cause an emergency stop. The driving restart possibility determination unit 234 is implemented by, for example, an AI (artificial intelligence) model that takes the emergency stop occurrence situation as input and outputs a determination result on whether the driving of the autonomously driving vehicle can be restarted based on autonomous driving. The method for determining whether the driving can be restarted is described later using an example.

[0117] The transmission unit 240 transmits the driving resumption permission signal through wireless communication with the reception unit 120 provided in the moving object 100 .

[0118] (Screen prompted by UI)

[0119] Next, with respect to the screen generated by the image generation unit 231 and presented to the operator as a UI, refer to Figure 2A to Figure 7B Provide explanation. Figure 2A to Figure 7B The image shown is a schematic diagram of an example of an image generated by the first information processing unit 220 according to the type of processing determined as the cause of the emergency stop.

[0120] 1. Detection processing is identified as a cause of emergency stop

[0121] Figure 2A as well as Figure 2B is used to explain the period from the safety control limit time T0 to the time when the detector 111 starts detecting the target. a The image generation unit 231 is a schematic diagram of an image generated by the time period up to , using the data generated by the observation unit 11 and the log of data generated by the automatic driving system 110. More specifically, Figure 2A This is a schematic diagram of an image generated by the image generation unit 231 based on the data output by the vehicle-mounted camera of the observation unit 11 of the moving object 100 (described later). Figure 3A , Figure 4A , Figure 5A , Fig. 6A , Fig. 7A Also, Figure 2B This is a schematic diagram of an image generated by the image generation unit 231 based on the data (after the target object (object) extraction process) output by the LiDAR of the observation unit 11 as the moving body 100 (described later). Figure 3B , Figure 4B , Figure 5B , Figure 6B , Figure 7B Same as above).

[0122] Figure 2A as well as Figure 2B Each image of is based on data obtained by the observation unit 11 at the safety control limit time T0. Figure 2B The image is drawn from a viewpoint located above the moving object 100 (described later). Figure 3B , Figure 4B , Figure 5B , Figure 6B , Figure 7B The same applies). Figure 2A as well as Figure 2B The transmission image of the human figure with a dotted outline in FIG. 1 is obtained by recording the sensor log 211 of the log recording unit 210 by the image generation unit 231 at the time T a The image of the detected target (in this case, a person) is overlapped.

[0123] If reference Figure 2A In the left region of the image, the image of the person on the trail as viewed from the moving body 100 (observation unit 11) is almost entirely hidden by a columnar object (“column” in the figure). Figure 2B, it can be seen that the person on the trail is not represented in the data output by the LiDAR. In order to understand the situation in which the moving body 100 has come to an emergency stop, the above-mentioned operator refers to these images along the time sequence including the safety control limit time T0. For example, the operator can understand by viewing these images that before the emergency stop of the moving body 100, the target is hidden on the back when viewed from the moving body 100, resulting in a situation in which the detection unit 111 cannot obtain the position of the target until the safety control limit time T0.

[0124] In addition, Figure 2A as well as Figure 2B The characters indicating the objects projected on the images are given for convenience of explanation, and such characters for explanation may be omitted in the drawings referenced below.

[0125] Figure 3A as well as Figure 3B is used to explain the time T when the detection unit 111 starts detecting the target. a This is a schematic diagram of an image generated by the image generation unit 231 using the data generated by the observation unit 11 and the log of data generated by the automatic driving system 110 during the time period up to the time T when the moving object 100 stops due to the emergency stop.

[0126] Figure 3A as well as Figure 3B The human figure in the figure is like a moment of becoming a The detection process is the image of the target person. Figure 2A as well as Figure 2B Its outline is shown only by dotted lines. Figure 3A as well as Figure 3B In FIG. 1 , as an image that matches the image that serves as a basis for the determination of execution of the emergency stop by the determination unit 113 , a dashed-dotted outline is superimposed on the image of the target by the image generation unit 231 .

[0127] For example as Figure 2A as well as Figure 2B Continue to watch the image Figure 3A as well as Figure 3B The operator of the image can understand that the determination in the automatic driving system 110 to execute the emergency stop of the moving body 100 is made for the target that the detection unit 111 started detecting at a time later than the safety control limit time T0.

[0128] 2. Cases where predictive processing is identified as a cause of emergency stop

[0129] Figure 4A as well as Figure 4Bis used to explain the time T when the detection unit 111 starts detecting the target. a The time period from the safety control limit time T0 to the time T1 at which the prediction unit 112 outputs the prediction result of the position of the target related to the emergency stop with a probability greater than a predetermined threshold value is T2. b This is a schematic diagram of an image generated by the image generating unit 231 using the data acquired by the observing unit 11 up to the time period .

[0130] exist Figure 4A as well as Figure 4B Of the images of the human figure reflected by the image of , only the image on the far left is originally reflected. For example, at time T a The image of the target represented by the observation data obtained by the observation unit 11. The images of the other two human figures are superimposed images based on the information obtained by the image generation unit 231 from the log recording unit 210. In each image, the image of the human figure having a dotted outline located at a point deviating to the right from the trail is at time T b The image output by the prediction unit 112 indicates the position of the target after a predetermined time. The image with a dotted outline at the tip of the arrow starting from the originally projected image indicates the time (time T) predicted by the prediction unit 112 after a predetermined time based on the image at the starting point of the arrow. a Later and than time T b That is, the arrow indicates the direction and size of the target movement predicted by the prediction unit 112 based on the image located at the starting point.

[0131] Watch Figure 4A as well as Figure 4B The operator of the image can determine whether there is a problem in the prediction processing performed by the prediction unit 112 by considering, for example, the conditions around the mobile body 100 and the target grasped from the image, or the time difference from the safety control limit time T0 and the predicted position of the target. If it is determined that there is a problem in the prediction processing performed by the prediction unit 112, the operator can also determine that the travel of the mobile body 100 by autonomous driving cannot be resumed.

[0132] Figure 5A as well as Figure 5B is used to explain the time T when the prediction result related to the emergency stop is output from the prediction unit 112 b This is a schematic diagram of an image generated by the image generation unit 231 using the data generated by the observation unit 11 and the log of data generated by the automatic driving system 110 during the time period up to the time T when the moving object 100 stops due to the emergency stop.

[0133] Figure 5A as well as Figure 5BThe image of the human figure shown by the solid line is at time T b The image of the target represented by the observation data obtained by the observation unit 11. The arrow indicates the direction and size of the target movement predicted by the prediction unit 112 based on the image at the starting point. The image of a human figure with a dotted outline located at a point deviating to the right from the trail is the position of the target involved in the prediction, and is also the image that serves as the basis for the judgment of the execution of the emergency stop by the judgment unit 113. For example, Figure 4A as well as Figure 4B In the continuation of the image, if the target image, the arrow and the dotted human figure move over time, and the dotted human figure matches the dotted human figure representing the outline of the image that serves as the basis for the judgment of executing the emergency stop, the dotted line becomes a dotted line.

[0134] For example as Figure 4A as well as Figure 4B Continue to watch the image Figure 5A as well as Figure 5B The operator of the image can easily grasp the time difference between the time corresponding to the image where the dotted line changes to the dotted line and the safety control limit time T0.

[0135] 3. The judgment unit determines the cause of the emergency stop

[0136] Fig. 6A as well as Figure 6B is used to explain the time T when the prediction result related to the emergency stop is output from the prediction unit 112 b Until the time T when the determination unit 113 outputs an instruction for control based on the determination result related to the emergency stop c This is a schematic diagram of an image generated by the image generation unit 231 using the data generated by the observation unit 11 and the log of data generated by the automatic driving system 110 up to the time. Fig. 6A as well as Figure 6B Each image contains the b The data obtained by the observation unit 11 is that the target moving object (“other moving object” in the figure) located in front of the moving object 100 at time T b In addition, Fig. 6A as well as Figure 6B The dashed arrow and dotted line segments visible in the lower right of the image of the other moving object in FIG. 1 are overlapped based on the information obtained by the image generation unit 231 from the log recording unit 210. The dashed arrow indicates that the determination unit 113 has not yet completed the time T. b The path of the moving body 100 planned up to time T. c The planned stop position related to the emergency stop instruction output by the determination unit 113.

[0137] Watch Fig. 6A as well as Figure 6B The operator of the image can confirm, for example, the planned stop position, the moving body 100, and the surrounding conditions of the target grasped from the image.

[0138] Fig. 7A as well as Figure 7B is used to explain the time T at which the instruction for control based on the determination result related to the emergency stop is output from the determination unit 113. c This is a schematic diagram of an image generated by the image generation unit 231 using the data generated by the observation unit 11 and the log of data generated by the automatic driving system 110 during the time period until the time T when the moving object 100 stops due to the emergency stop. Fig. 7A as well as Figure 7B Each image contains the c The data acquired by the observation unit 11 is the time T of two moving objects (“other moving object A” and “other moving object B” in the figure) located in front of the moving object 100 as targets. c In addition, Fig. 7A as well as Figure 7B The dotted line band and line segment visible at the lower right of the image of the other moving body A, and the arrow visible below the other moving body B are overlapped based on the information obtained by the image generation unit 231 from the log recording unit 210. The dotted line band and line segment represent the planned path and stop position of the moving body 100 determined by the judgment unit 113 in the judgment related to the emergency stop, as well as the path traveled by the moving body 100 up to time T and the stop position at time T. In other words, the plan performed by the judgment unit 113 and the control result of the moving body 100 performed by the moving body control unit 12 are consistent. In addition, the arrows represent the planned path and stop position of the moving body 100 determined by the judgment unit 113 up to time T. c The predicted path of the other moving object B output by the prediction unit 112 related to the instruction for control outputted so far.

[0139] observe Fig. 7A as well as Figure 7B An operator of the image can, for example, confirm the matching of the path and stop position planned for an emergency stop with the travel path and stop position of the moving body 100 as a control result, and the condition around the target when the instruction for the emergency stop control is output.

[0140] exist Figure 2A to Figure 7B The image illustrated in FIG. 1 is matched with the candidate of the situation that may cause an emergency stop registered in the above-mentioned situation database 232, and is used as a structural element of the UI described next.

[0141] (UI Structure)

[0142] The configuration of the UI that the UI control unit 233 provides to the operator via the output device 21 and the input device 22 and a usage example are described below.

[0143] The UI includes a component for presenting to the operator the processing in the automatic driving system 110 that has been identified by the cause identification unit 223 as the cause of the mobile body 100 coming to an emergency stop.

[0144] The UI further includes a component for presenting to the operator candidates of situations that may cause an emergency stop, which are obtained from the situation database 232 and correspond to the above-mentioned process determined as a cause of the emergency stop.

[0145] The UI further includes a component for presenting an image generated by the image generation unit 231 for allowing the operator to confirm the actual situation in which the emergency stop has occurred in order to select an appropriate candidate from the above-mentioned candidates.

[0146] The UI further includes a configuration for presenting to the operator whether or not the autonomous driving of the mobile body 100 can be resumed, as determined by the driving resumption possibility determination unit 234 .

[0147] For example, when the processing performed by the detector 111 is determined as the cause of the emergency stop of the mobile body 100, the UI presented to the operator is as follows: Figure 8 to Figure 10 Like that.

[0148] These UIs are a series of screens including examples of the above-listed structural elements. The operator can operate the search bar in the middle of the screen to view an image capturing the situation around the moving body 100 before and after the moving body 100 comes to an emergency stop. The image is an image generated by the image generation unit 231.

[0149] Figure 8 This is a screen that presents the image and the candidate of the situation that will cause the emergency stop as an option. In this screen, the input of the selected situation by the operator has not yet been performed.

[0150] exist Fig. 9 as well as Fig.10 In the UI screen shown, the result of the determination of the driving restart possibility determination unit 234 on the driving restart possibility determination is presented to the operator based on the result of the input by the operator (see the "AI automatic determination" column in the lower right of the screen).

[0151] The following describes the structural elements in each figure in more detail.

[0152] exist Figure 8 to Figure 10The text “It is possible that the obstacle detection failed and the vehicle 100 stopped suddenly.” visible below the search bar is a structural element that indicates to the operator that the cause determination unit 223 has determined that the process in the automatic driving system 110 that caused the vehicle 100 to stop suddenly was “detection”.

[0153] Furthermore, "the moving body is hidden in a blind spot" and "partially / temporarily hidden" listed below the text are examples of candidates for situations that may cause an emergency stop, and are examples of candidates that the UI control unit 233 obtains from the situation database 232 based on the "detection" process determined as the cause of the emergency stop. In addition, "no response" further existing below these is a candidate selected when the operator considers that the above two candidates obtained from the situation database 232 are not suitable for the actual situation.

[0154] Furthermore, the two images arranged side by side above the text indicating the determined cause are examples of images generated by the image generation unit 231 based on the logs stored in the log recording unit 210 and presented to the operator for checking the occurrence of the emergency stop. Figure 8 as well as Fig. 9 In the example, Figure 2A as well as Figure 2B The image illustrated in FIG. 1 is included in the image presented for confirming the occurrence situation of the emergency stop.

[0155] In the column of AI automatic determination, the result of the determination of the driving resumption possibility by the driving resumption possibility determination unit 234 is input. This determination is based on the input result of the operator with respect to the situation in which the emergency stop has occurred.

[0156] Fig. 9 This is an example of a UI that is prompted when an operator who has viewed the image generated by the image generation unit 231 and has grasped the situation in which a person as an obstacle is hidden by a columnar work object enters the selection of "hidden in a blind spot" from the displayed situation candidates. In this example, in the column of AI automatic determination, a message "driving restart OK" is displayed indicating the result of the determination of driving restart possibility by the driving restart possibility determination unit 234. In this example, a message "Emergency stop is working normally" is also displayed indicating that the automatic driving system 110 is normal.

[0157] Fig.10This is an example of a UI that is prompted when an operator who has seen the image generated by the image generation unit 231 and has realized that a person who is an obstacle is not sufficiently hidden from the front selects "No response" from the displayed status candidates and inputs it. In this example, in the column of AI automatic determination, the words "Driving restart NG" are displayed, indicating the result of the determination of whether driving can be restarted by the driving restart determination unit 234. In this example, the words "Emergency stop is abnormal. Driving cannot be restarted." are also displayed, indicating that there is a possibility that the automatic driving system 110 is abnormal.

[0158] These displays such as "driving restart OK" or "driving restart NG" serve as buttons for accepting input from the operator. If the operator who has confirmed the determination result presses these buttons, a driving restart permission signal indicating whether driving can be restarted according to the determination result is sent from the transmission unit 240 to the mobile body 100. In addition, in the examples shown in the figures, the operator only confirms the determination result, but the present invention is not limited to this. For example, a structural element that allows the operator to select whether to approve or deny the determination result (and then reselect the status) may be included in the UI.

[0159] (Method for determining whether driving can be resumed)

[0160] The driving restart possibility determination unit 234 determines whether the driving of the mobile body 100 can be restarted by autonomous driving in response to the input of the situation selected by the operator from the candidate situations that may cause an emergency stop. The following describes an example of each process of the automatic driving system 110 determined to have an abnormality with respect to the method of determining whether the driving can be restarted by the driving restart possibility determination unit 234.

[0161] In addition, in this description, as a preparation for the determination, a parameter indicating the safety or danger of the execution of the autonomous driving control is used, and this parameter is a parameter for the situation selected and input by the operator as an applicable situation. For example, a parameter having a value of 1 when indicating that the resumption of driving is safe and a value of 0 when indicating that it is not safe or dangerous is used. Alternatively, the value of this parameter may be pre-set in the situation database for each candidate of the situation presented to the operator.

[0162] 1. Detection processing is identified as a cause of emergency stop

[0163] Assume that the condition input as the operator's selection is that the obstacle involved in the emergency stop is "hidden in a blind spot" or "partially / temporarily hidden" (reference Fig. 9). In these situations, it is difficult to represent obstacles in the observation data from the observation unit 11 such as a camera or LiDAR, so it is difficult to detect. Therefore, it is assumed that an emergency stop must be made during autonomous driving by the automatic driving system 110, that is, it is not an abnormality of the automatic driving system 110, and it is safe to resume driving, and 1 is given as the value of the parameter.

[0164] In addition, suppose that an input is not applicable to either of the above two conditions, that is, "no applicable" is input (refer to Fig.10 ). In this case, if the automatic driving system 110 is normal, the obstacle is detected before it is sufficiently close, and an emergency stop should not occur. In other words, there is a possibility that there is an abnormality in the automatic driving system 110, and it is unsafe to resume driving. Therefore, 0 is given as the value of the parameter. In fact, even if it is not "hidden in a blind spot" or "partially / temporarily hidden", there are situations that are difficult to detect. For example, in the case of detecting an obstacle from a camera image, if the obstacle is assimilated with the background, it is difficult to detect. If the obstacle is a pedestrian, it is difficult to detect if it is squatting. In response to such a problem, for example, the following countermeasures are considered: a candidate such as "assimilated with the background" is newly registered in the database 232, and if the detection processing can detect the type of object, each candidate is registered to match the type.

[0165] In terms of language, how to distinguish between "hidden in a blind spot" and "partially / temporarily hidden" depends on the operator. In response to such ambiguity, the positional relationship between the obstacle and other obstacles detected outside it, or the positional relationship between the obstacle and the blind spot area that can be obtained from the map information can be expressed in numerical values. Therefore, the server can also determine whether the overlapping area between the obstacle and other obstacles or the blind spot area exceeds a threshold. If the threshold is exceeded, the information indicating the overlap between the obstacle and other obstacles or the blind spot area or the degree of the overlap is superimposed on the image at that time, and the overlapped image is prompted to the operator. In this way, the ambiguity can be effectively eliminated by allowing the operator to confirm the image with the information useful for judgment superimposed.

[0166] In addition, in the selection of the candidates, the candidates may be narrowed down in advance, and the narrowed down candidates may be presented. In addition, the result of the pseudo selection by the server may be presented, and the operator may confirm it. For example, based on the overlapping state between the obstacles in the past and other obstacles or blind spots, and the situation selected by the operator at this time, the candidates may be narrowed down to two or more situations, and the narrowed down candidates may be presented to the operator. In addition, instead of narrowing down, the candidate of one situation may be pseudo-selected, and the pseudo-selected candidate of the situation may be presented to the operator.

[0167] 2. Cases where predictive processing is identified as a cause of emergency stop

[0168] Assume that the operator selects and inputs one or more of the following conditions for the obstacle involved in the emergency stop: "started moving", "was moving but stopped", "changed the direction of travel during movement". In this case, the obstacle started or stopped moving suddenly, or the direction information that is difficult to predict has changed, so it is difficult to predict. Therefore, it is assumed that the situation in which an emergency stop must be made during autonomous driving by the automatic driving system 110, that is, it is not an abnormality of the automatic driving system 110, and it is safe to resume driving, and 1 is given as the value of the parameter.

[0169] In addition, it is assumed that "No response" is selected and input. In this case, since the obstacle has not moved or has moved in one direction, if the automatic driving system 110 is normal, the moving path of the obstacle is predicted sufficiently in advance, and an emergency stop should not occur. In other words, there may be an abnormality in the automatic driving system 110, and it is not safe to resume driving. Therefore, 0 is given as the value of the parameter.

[0170] In the real world, there is basically no movement with no change in direction. How to distinguish the start, stop, and sharp acceleration and deceleration of movement depends on the operator. For such ambiguity, if the detection results of the position of the obstacle are bundled in time sequence, it can be expressed as a numerical value. Therefore, the server can also detect the acceleration and deceleration of movement exceeding the threshold, and the change of direction, so that the detection results are overlapped with the image at that time, and the overlapped image is prompted to the operator. In this way, the ambiguity can be effectively eliminated by allowing the operator to confirm the image with information useful for judgment.

[0171] In addition, in the selection of candidates, the candidates may be narrowed down in advance, and the narrowed down candidates may be presented. In addition, the server may perform a separate process, make a selection in advance, and then confirm the selection result with the operator. For example, based on the acceleration and deceleration of the movement of the obstacle in the past or the change of the direction and the situation selected by the operator at this time, the candidates may be narrowed down to two or more situations, and the narrowed down candidates may be presented to the operator. In addition, instead of narrowing down, the candidate of one situation may be pseudo-selected, and the pseudo-selected candidate of the situation may be presented to the operator.

[0172] The above is because it is believed that in a case where there is full trust in the processing results on the server side of the above candidate selections, for example, in a case where an emergency stop is caused by the sudden appearance of a corresponding obstacle at a location where the existence of a blind spot has been confirmed through the service operation so far, the server can estimate the situation based on the content confirmed by the operator so far.

[0173] 3. Determine the situation where the judgment process becomes the cause of the emergency stop

[0174] Assume that the operator selects and inputs a situation such as "another obstacle suddenly enters the planned driving path of the mobile body". In this case, it is difficult to determine whether to perform speed control under deceleration so as not to get too close to the obstacle. Therefore, it is assumed that an emergency stop must be made during autonomous driving by the automatic driving system 110, that is, it is not an abnormality of the automatic driving system 110, and it is safe to resume driving, and 1 is given as the value of the parameter.

[0175] In addition, if it does not correspond to the necessary conditions as described above, it should be judged that the vehicle decelerates sufficiently in advance without getting too close to the obstacle and performs deceleration control without causing an emergency stop. In other words, there may be an abnormality in the automatic driving system 110, and it is unsafe to resume driving. Therefore, 0 is given as the value of the parameter.

[0176] Whether the movement of the moving body into another obstacle on the planned travel path is sudden or not depends on the operator. In view of such ambiguity, if the detection results of the positions of the other obstacles are bundled in time series, they can be expressed in numerical values.

[0177] Therefore, the server can also detect the acceleration and deceleration of movement and the change of direction that exceeds the threshold, overlap the detection result with the image at that time, and prompt the operator with the overlapped image. In this way, the ambiguity can be effectively eliminated by allowing the operator to confirm the image with information useful for judgment.

[0178] Alternatively, the server may perform a separate process to select in advance and then allow the operator to confirm the selection result. For example, based on the acceleration and deceleration of the movement of the obstacle or the change in direction in the past and the situation selected by the operator at this time, a candidate situation may be provisionally selected, and the candidate situation provisionally selected may be presented to the operator.

[0179] In addition, in determining whether the situation is sudden or not, the possibility of restarting may change depending on what kind of object the obstacle that enters the planned driving path is (for example, a car, a person, or an animal). Therefore, the candidates for the situation may be registered for each type of obstacle, and based on the acceleration and deceleration of the movement or the change in the direction of each type of obstacle in the past and the situation selected by the operator at this time, the candidates for the situation may be narrowed down to two or more, and the narrowed candidates may be presented to the operator. In addition, instead of narrowing down, a candidate for a situation may be pseudo-selected, and the pseudo-selected candidate for the situation may be presented to the operator.

[0180] In addition, for "other obstacles", when the processing performed by the detection unit 111 or the prediction unit 112 is also determined as a cause by the cause determination unit 223, the operator can also re-input the selection from the candidate selections of the above-mentioned conditions for each case, thereby giving the value of the parameter.

[0181] The driving restart possibility determination unit 234 outputs a determination result that driving can be restarted to the UI control unit 233 when the value given to the parameter in response to the operator's input is 1, and outputs a determination result that driving cannot be restarted to the UI control unit 233 when the parameter is 0. The UI control unit 233 controls the display in the column of "AI automatic determination" based on the determination result received from the driving restart possibility determination unit 234.

[0182] In addition, as a further development, the possibility of resuming driving may be determined more strictly.

[0183] For example, in the case where the detection process is the cause, if the obstacle involved in the emergency stop is hidden in a blind spot, an emergency stop may be necessary. However, if the situation changes from the situation where the obstacle involved in the emergency stop is hidden in a blind spot to the situation where the obstacle involved in the emergency stop can be observed, it is considered that the probability of the occurrence of control related to the accident such as the emergency stop will change greatly depending on how quickly the detection unit 111 detects the obstacle.

[0184] In this case, the image generation unit 231 obtains an output of how much time difference is expected between the identified process and the safety control limit time T0 from the identification result of the cause identification unit 223, and generates an image as described above.

[0185] Then, the operator follows the images generated by the image generating unit 231 in time series via the input device 22 , and inputs a candidate of which state the image at each time point corresponds to.

[0186] The driving restart possibility determination unit 234 receiving the input can determine whether the driving can be restarted according to whether the time from the situation change to the obstacle detection exceeds the allowable value of the processing delay. In addition, the time from the situation change (or from the obstacle detection after the situation change) to the prediction of the position of the obstacle, or the time from the situation change (or from the prediction of the position of the obstacle after the situation change) to the judgment of emergency stop can also be compared with the allowable value.

[0187] [action]

[0188] Next, a description will be given of the procedure of the information processing method according to the present embodiment, which is executed by the remote server 200 that communicates with the mobile object 100 . Fig.11This is a flowchart showing an example of the procedure of the information processing method involved in this embodiment.

[0189] When the remote server 200 acquires one of the sensor log 211 , the driving log 212 , and the processing log 213 transmitted from the automatic driving system 110 of the mobile body 100 , these logs are recorded in the log recording unit 210 (step S10 ).

[0190] When the operation detection unit 221 of the first information processing unit 220 detects an emergency stop of the mobile body 100, the processing log 213 or the driving log 212 is referred to (step S11). If no emergency stop occurs (No in step S11), the processing of the log obtained in step S10 ends.

[0191] When the occurrence of an emergency stop is detected (Yes in step S11), the time calculation unit 222 and the cause determination unit 223 of the first information processing unit 220 determine whether the cause of the emergency stop is determined to be a process performed by the detection unit 111, the prediction unit 112, or the judgment unit 113 of the automatic driving system 110, or a system error (step S12). When the cause of the emergency stop is determined to be a system error (No in step S12), a driving restart permission signal indicating that the resumption of autonomous driving is impossible is transmitted from the cause determination unit 223 to the mobile body 100 via the transmission unit 240 (step S18).

[0192] When the cause of the emergency stop is processing in the automatic driving system 110 (Yes in step S12), the UI control unit 233 of the second information processing unit 230 obtains candidates for conditions that may cause an emergency stop from the condition database 232 based on the processing determined as the cause of the emergency stop (step S13).

[0193] The UI control unit 233 presents the operator with a UI including an image of the actual situation in which the moving body 100 has come to an emergency stop, which is generated by the image generation unit 231 using the information extracted from the log recording unit 210, and a candidate of the situation obtained in step S13, via the output device 21 (step S14). The UI presented here is, for example, Figure 8 The screen shown here allows the operator to select and input appropriate content as a description of the situation causing an emergency stop.

[0194] The driving restart possibility determination unit 234 determines whether the driving of the mobile body 100 can be restarted by autonomous driving based on the input result of the situation corresponding to the actual situation selected by the operator from the candidates of the situation via the input device 22 (step S15). In addition, there are cases where the determination here is a false determination, or the determination is made after further input related to the situation is obtained from the operator. In the case where there is a UI for requesting further input related to the situation from the operator (Yes in step S16), the process from step S14 is repeated by the UI control unit 233.

[0195] When all the UIs that should be presented to the operator have been presented and inputs have been received (No in step S16), the UI control unit 233 presents the UI that reflects the result of the determination of whether the driving can be restarted by the driving restart determination unit 234 to the operator via the output device 21 to seek approval (or confirmation) (step S17). Fig. 9 or Fig.10 The screen shown is prompted.

[0196] If the UI control unit 233 detects that the operator has approved the result of the determination of whether or not to resume driving via the input device 22 (Yes in step S17), a signal indicating whether or not to resume driving corresponding to the approved result of the determination is transmitted from the transmission unit 240 to the mobile body 100 (step S19). In addition, if the operator refuses to approve the result of the determination of whether or not to resume driving (No in step S17), the process is set to return to step S14 in this example. After step S14 in this case, for example, the operator may be asked to reconfirm and reselect the corresponding situation, and the UI that was previously presented may be presented again.

[0197] (Variations and Supplementary Matters)

[0198] The information processing method involved in one or more embodiments of the present disclosure is not limited to the description of the above-mentioned embodiments. As long as it does not deviate from the purpose of the present disclosure, the various modifications that a person skilled in the art may conceive of are also included in the embodiments of the present disclosure. Examples of the following modifications and other supplementary matters to the description of the embodiments can be cited.

[0199] (1) In the above embodiment, the safety control is described as a control to stop the vehicle at the position where the vehicle stops due to an emergency stop by a normal autonomous driving control, but the method of the safety control is not limited to this. The safety control may also be a control to continue driving while avoiding obstacles involved in the emergency stop during actual driving by steering and accelerating or decelerating as needed.

[0200] (2) In the above-mentioned embodiment, the value of the parameter representing the degree of safety or the degree of danger is determined according to the condition selected and inputted by the operator from the candidates as the corresponding condition, but is not limited to this. For example, the value of the above-mentioned parameter may also be a value related to the time corresponding to the candidate of the condition inputted, and the candidate of the condition is inputted for the time until the safety control limit moment. More specifically, the closer the time period of the sudden intrusion route of other mobile bodies is to the safety control limit moment, the lower the possibility that the emergency stop as the action of the automatic driving system is abnormal, and the higher the degree of safety. As another example of the value of the parameter, a value representing the degree of safety or the degree of danger based on the length of the delay time from the safety control limit moment may also be added to the output of the processing determined as the cause of the emergency stop.

[0201] In addition, the frequent occurrence of emergency stops in a specific candidate may be considered as dangerous driving, so the value indicating the safety or danger level is updated every time a certain number of inputs to the same candidate is reached. In addition, the values ​​of the parameters listed here are not limited to binary values ​​such as 0 and 1, and more levels of values ​​can be used.

[0202] Furthermore, it is also considered to set the parameters with continuous values ​​from 0 to 1. It is considered that when an emergency stop occurs, the greater the speed of the moving body, the closer the distance to the obstacle, and the more emergency stops occur in the same candidate, the greater the danger. It is also considered to make correct answer data for safety or danger based on these data, and use the above parameters as weights for learning. Of course, the operator can also manually give the correct answer data.

[0203] (3) Some or all of the functional structural elements of each of the above-mentioned information processing systems may also be constituted by a system LSI (Large Scale Integration). System LSI is a multifunctional LSI manufactured by integrating multiple structural parts on a single chip. Specifically, it is a computer system including a microprocessor, ROM, RAM, etc. The ROM stores a computer program. The microprocessor operates according to the computer program, and the system LSI achieves the functions of each structural element.

[0204] In addition, here, it is set as system LSI, and sometimes it is called IC, LSI, super LSI, and ultra LSI due to differences in integration. In addition, the method of integrated circuit is not limited to LSI, and it can also be realized by a dedicated circuit or a general-purpose processor. After LSI is manufactured, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connection and setting of the circuit cells inside the LSI can also be used.

[0205] Furthermore, if the technology of integrated circuitization replacing LSI emerges due to the progress of semiconductor technology or other derived technologies, the technology can also be used to integrate functional blocks. It is also possible to apply biotechnology, etc.

[0206] (4) One aspect of the present disclosure is not limited to use Fig.11 The above-mentioned information processing method may also be a program executed by a computer and an information processing system including a computer. In addition, one aspect of the present disclosure may also be a non-temporary recording medium that records such a computer program and is readable by a computer.

[0207] Industrial Availability

[0208] The technology involved in the present disclosure can be utilized as an information processing method executed when an emergency stop occurs for an autonomously traveling mobile body, an information processing device that executes such an information processing method, or a program that causes an information processing device to execute such an information processing method.

[0209] Description of symbols

[0210] 11 Observation Department

[0211] 12 Mobile body control unit

[0212] 21 Output device

[0213] 22 Input Devices

[0214] 100 Mobile

[0215] 110 Autonomous Driving System

[0216] 111 Detection Department

[0217] 112 Forecasting Department

[0218] 113 Judgment Department

[0219] 120 Receiving Department

[0220] 200 Remote Server

[0221] 210 Logging Department

[0222] 211 Sensor Log

[0223] 212 Driving Log

[0224] 213 Processing Logs

[0225] 220 First Information Processing Department

[0226] 221 Work Inspection Department

[0227] 222 Time Calculation Department

[0228] 223 Cause Determination Department

[0229] 230 Second Information Processing Department

[0230] 231 Image Generation Unit

[0231] 232 Status Database

[0232] 233 UI Control Unit

[0233] 234 Driving restart possibility determination unit

[0234] 240 Sending Department

Claims

1. An information processing method is an information processing method executed by a processor, In the information processing method, determining whether the autonomously traveling moving body has made an emergency stop to avoid a collision with an obstacle, When it is determined that the emergency stop has occurred, based on the driving condition history of the mobile body, autonomous driving control, i.e., safety control, is simulated for autonomous driving so that the mobile body avoids the collision with the obstacle without performing the emergency stop. obtaining a log of an automatic driving system that executes the autonomous driving control of the mobile body, and determining, based on the autonomous driving control indicated in the log and the simulated safety control, a process that is a cause of the emergency stop from among processes for the autonomous driving control indicated in the log, acquiring a candidate condition corresponding to the determined process from a database related to conditions that may cause an emergency stop, and presenting an interface for an operator to input the candidate condition corresponding to the condition shown in the log; determining whether the autonomous driving control performed by the automatic driving system can be resumed based on the inputted condition candidate, The safety control is compared with the autonomous driving control shown in the log, and a safety control time which is a start time of the safety control for executing the safety control in the autonomous driving control of the mobile body is obtained.

2. The information processing method according to claim 1, The determination of the process as the cause of the emergency stop is performed based on a result of determination as to whether a result of the process required for execution of the safety control is output before the safety control timing.

3. The information processing method according to claim 2, The processing required for executing the safety control is at least one of a detection processing for obtaining a detection result indicating the surrounding condition of the moving body, a prediction processing for predicting the surrounding condition of the moving body using the detection result, and a judgment processing for determining a driving path corresponding to the predicted surrounding condition.

4. The information processing method according to claim 2, The interface includes an image generated using the determined output of the processing and observation data input to the automated driving system.

5. The information processing method according to claim 4, The interface includes an image generated by further using the output at the safety control time in the safety control.

6. The information processing method according to claim 5, The interface is an interface for inputting the status candidate corresponding to the status indicated by the log at a time after the security control time.

7. The information processing method according to claim 2, Whether the autonomous driving control can be resumed is determined based on a value of a parameter that is set for the input situation candidate and indicates a safety level or a risk level of execution of the autonomous driving control.

8. The information processing method according to claim 7, The parameter is based on a delay time length of an output of the process determined from the safety control timing.

9. The information processing method according to claim 7, The parameter is related to the time corresponding to the input state candidate, The state candidates are input for the time until the safety control timing.

10. The information processing method according to any one of claims 1 to 9, The safety control is control for stopping the moving body at a parking position by the emergency stop or at a position before the parking position.

11. The information processing method according to any one of claims 1 to 9, The safety control is a control for causing the moving object to travel while avoiding the obstacle.

12. An information processing system comprising one or more computers having a processor, The processor performs: determining whether the autonomously traveling moving body has made an emergency stop to avoid a collision with an obstacle, When it is determined that an emergency stop has occurred, based on the driving condition history of the mobile body, autonomous driving control, i.e., safety control, is simulated for autonomous driving so that the mobile body avoids a collision with the obstacle without performing the emergency stop. obtaining a log of an automatic driving system that executes the autonomous driving control of the mobile body, and determining, based on the autonomous driving control indicated in the log and the simulated safety control, a process that is a cause of the emergency stop from among processes for the autonomous driving control indicated in the log, acquiring a candidate condition corresponding to the determined process from a database related to conditions that may cause an emergency stop, and presenting an interface for an operator to input the candidate condition corresponding to the condition shown in the log; determining whether the autonomous driving control performed by the automatic driving system can be resumed based on the inputted condition candidate, The safety control is compared with the autonomous driving control shown in the log, and a safety control time which is a start time of the safety control for executing the safety control in the autonomous driving control of the mobile body is obtained.

Images