Remote support system

Abstract

To enhance safety in remote support for an automatic driving vehicle.SOLUTION: A remote support system for remotely supporting an automatic driving vehicle includes one or a plurality of processors. The one or the plurality of processors performs: acquiring support request information including contents of a remote support request from the automatic driving vehicle; acquiring an anxiety message to transmit an anxiety to be assumed when a remote supporter performs an erroneous remote support for the automatic driving vehicle based on the support request information; and notifying the remote supporter of the anxiety message through a remote support terminal to be operated by the remote supporter when the remote support is performed.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to technology for remotely assisting autonomous vehicles. [Background technology]

[0002] Patent Document 1 discloses a driving assistance control device, which presents driving assistance supplementary information including a warning (for example, "Watch out for pedestrians jumping out into the road") to an operator based on a database of past accident patterns.

[0003] Furthermore, Patent Document 2 discloses a vehicle remote control device. When the remote operator is not gazing at a target area where the remote operator should be gazing, the vehicle remote control device determines that the remote operator is not in an appropriate state for operation and highlights the target area on a monitor. Furthermore, Patent Document 3 discloses a method for determining responsibility for an accident involving an autonomous vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-189683 [Patent Document 2] Japanese Patent Publication No. 2022-188453 [Patent Document 3] Japanese Patent Publication No. 2020-166541 Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to further improve the safety of remote assistance for autonomous vehicles. [Means for solving the problem]

[0006] A remote assistance system according to the present disclosure is a system for remote assistance for an autonomously driven vehicle, and includes one or more processors. The one or more processors are configured to acquire assistance request information including the content of a remote assistance request from the autonomously driven vehicle, acquire a concern message based on the assistance request information to convey a possible concern if a remote assistance person erroneously provides remote assistance to the autonomously driven vehicle, and notify the remote assistance person of the concern message via a remote assistance terminal operated by the remote assistance person during remote assistance. [Effects of the Invention]

[0007] According to the present disclosure, a remote assistant is notified of specific concerns that may arise if incorrect remote assistance is provided. This encourages the remote assistant to provide remote assistance more carefully and meticulously. As a result, the safety of remote assistance for autonomous vehicles is improved. Furthermore, the remote assistant can more clearly recognize what to pay attention to in order to prevent the anticipated concerns from actually occurring. This also contributes to improving the safety of remote assistance. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram for explaining a configuration of a remote support system according to an embodiment. [Figure 2] FIG. 10 is a block diagram illustrating an example of a functional configuration related to notification of a concern message by the remote support system. [Figure 3] FIG. 10 is a diagram illustrating a first example of obtaining a message of concern. [Figure 4] FIG. 10 is a diagram illustrating a second example of obtaining a message of concern. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0010] 1. Remote support system FIG. 1 is a conceptual diagram illustrating the configuration of a remote assistance system 100 according to this embodiment. The remote assistance system 100 is a system for remotely assisting an autonomous vehicle 1. The remote assistance system 100 includes a remote assistance terminal 110 and a management device 120. More specifically, the autonomous vehicle 1 is equipped with an autonomous driving function of level 4 or higher that does not require a driver, for example. An example of the autonomous vehicle 1 is an autonomous bus that operates within a limited area.

[0011] The autonomously driven vehicle 1 and the remote assistance system 100 can communicate with each other via a wireless communication network. The autonomously driven vehicle 1 communicates with the remote assistance system 100 and requests remote assistance as necessary. In response to this remote assistance request, the remote assistance system 100 remotely assists the autonomously driven vehicle 1. More specifically, a human remote assistant (operator) remotely assists the autonomously driven vehicle 1 via the remote assistance system 100. The remote assistance system 100 can also be said to be a system that assists the remote assistant in remotely assisting the autonomously driven vehicle 1.

[0012] 1-1.Remote support terminal The remote support terminal 110 is a terminal device that a remote supporter operates when remotely supporting the autonomously driven vehicle 1. In other words, the remote support terminal 110 is configured to be used by the remote supporter to remotely support the autonomously driven vehicle 1.

[0013] The remote support terminal 110 includes a communication device 111, an output device 112, an input device 113, one or more processors 114 (hereinafter simply referred to as processors 114), and one or more memory devices 115 (hereinafter simply referred to as memory devices 115).

[0014] The communication device 111 communicates with the autonomously driven vehicle 1 and the management device 120. The output device 112 outputs various types of information. For example, the output device 112 includes a display device. The display device displays various types of information to the remote supporter. In other words, the display device presents various types of information to the remote supporter by displaying them. As another example, the output device 112 may include a speaker. The input device 113 accepts input from the remote supporter. For example, the input device 113 is an interface for accepting input from the remote supporter. Examples of the input device 113 include a touch panel, a keyboard, a mouse, etc.

[0015] The processor 114 executes various processes related to the control of the remote support terminal 110. Examples of the processor 114 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array). The processor 114 can also be called circuitry or processing circuitry. The storage device 115 stores various types of information. Examples of the storage device 115 include volatile memory, non-volatile memory, HDD, and SSD. The functions of the remote support terminal 110 may be realized by cooperation between the processor 114, which executes a control program, and the storage device 115. The control program is stored in the storage device 115. Alternatively, the control program may be recorded on a computer-readable recording medium.

[0016] 1-2.Management device The management device 120 manages the remote support system 100. The management device 120 may be configured with multiple servers that perform distributed processing. For example, the management device 120 manages multiple remote supporters and multiple remote support terminals 110. In addition, in response to a remote support request from the autonomously driven vehicle 1, the management device 120 assigns the remote supporter and the remote support terminal 110 to remotely support the autonomously driven vehicle 1.

[0017] The management device 120 includes a communication device 121, one or more processors 122 (hereinafter simply referred to as processors 122), and one or more storage devices 123 (hereinafter simply referred to as storage devices 123). The communication device 121 communicates with the autonomously driven vehicle 1 and the remote assistance terminal 110.

[0018] The processor 122 executes various processes. Examples of the processor 122 include a CPU, a GPU, an ASIC, and an FPGA. The processor 122 can also be called a circuitry or a processing circuitry. The storage device 123 stores various information. Examples of the storage device 123 include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the management device 120 may be realized by cooperation between the processor 122, which executes a management program, and the storage device 123. The management program is stored in the storage device 123. Alternatively, the management program may be recorded on a computer-readable recording medium. The management program may be provided via a network.

[0019] 1-3. Overview of remote support A display device included in output device 112 of remote support terminal 110 displays information such as the status of the area in which autonomously driven vehicle 1 is traveling, the position and status of autonomously driven vehicle 1, etc. The remote supporter monitors the information displayed on the display device. Then, the remote supporter remotely supports autonomously driven vehicle 1 as necessary.

[0020] The remote assistance includes remote assistance related to the driving of the autonomously driven vehicle 1 (driving assistance) and remote assistance related to the services provided by the autonomously driven vehicle 1 (service assistance).

[0021] A situation in which driving assistance from a remote supporter is necessary is, for example, a situation in which autonomous driving is difficult. The autonomously driven vehicle 1 performs various vehicle processes while autonomous driving is being performed. Examples of vehicle processes include a "recognition process," an "action determination process," and a "timing determination process." Specifically, in the recognition process, the autonomously driven vehicle 1 uses a recognition sensor to recognize the situation around the autonomously driven vehicle 1. For example, the autonomously driven vehicle 1 uses a camera to recognize the signal display of a traffic light (e.g., green light, yellow light, red light, right turn signal, etc.). In the action determination process, the autonomously driven vehicle 1 determines whether or not to perform an action based on the result of the recognition process. Examples of actions of the autonomously driven vehicle 1 include starting, stopping, turning right, turning left, changing lanes, etc. In the timing determination process, the autonomously driven vehicle 1 determines the execution timing for performing the above action.

[0022] Driving assistance is assistance related to each of the vehicle processes described above. For example, when sunlight hits a traffic light installed at an intersection, the accuracy of recognizing the traffic light display may decrease. If the recognition process is unable to accurately distinguish the traffic light display, the autonomous vehicle 1 requires remote assistance from a remote assistant for signal recognition. In this way, "assistance in recognizing the traffic light display" is an example of driving assistance. Furthermore, if the traffic light display cannot be distinguished, it is difficult to determine what action should be taken and when. Therefore, the autonomous vehicle 1 requires remote assistance from a remote assistant for action determination processing and timing determination processing as well. In this way, "assistance in starting decision (Go / Not Go) and assistance in deciding the starting timing" are also examples of driving assistance.

[0023] In an example where autonomous vehicle 1, which has doors for passengers to get on and off, provides passengers with a service to automatically open and close the doors, "assistance in determining whether to open and close the doors and the timing of opening and closing the doors" corresponds to an example of service support by a remote supporter. Also, in an example where autonomous vehicle 1 provides passengers with a service to check the safety inside the vehicle, "assistance in checking the safety inside the vehicle" corresponds to an example of service support.

[0024] The outline of the processing flow during remote assistance of the autonomously driven vehicle 1 is as follows.

[0025] That is, when remote assistance from a remote assistant is required, the autonomously driven vehicle 1 transmits "assistance request information REQ" indicating a request for remote assistance to the remote assistance system 100. The remote assistance system 100 receives the assistance request information REQ from the autonomously driven vehicle 1. The assistance request information REQ includes the content of the remote assistance request from the autonomously driven vehicle 1.

[0026] The management device 120 of the remote assistance system 100 notifies the remote assistance person of the received assistance request information REQ via the remote assistance terminal 110, and requests instructions from the remote assistance person for the autonomously driven vehicle 1. The remote assistance person determines instructions for the autonomously driven vehicle 1 based on the content of the remote assistance request included in the assistance request information REQ. The remote assistance person inputs the instructions into the remote assistance terminal 110. The remote assistance system 100 communicates with the autonomously driven vehicle 1, and notifies the autonomously driven vehicle 1 of the instructions from the remote assistance person.

[0027] The autonomously driven vehicle 1 receives instructions from the remote support person from the remote support system 100. Then, the autonomously driven vehicle 1 drives (autonomously drives) according to the instructions, or performs services such as opening and closing doors.

[0028] 2. Notification of Message of Concern To further enhance the safety of remote assistance for the autonomously driven vehicle 1, in this embodiment, when assistance request information REQ is received from the autonomously driven vehicle 1, the remote assistance system 100 acquires a "concern message MSG" based on the assistance request information REQ. The concern message MSG is a message for conveying concerns that may arise if the remote assistance person provides incorrect remote assistance to the autonomously driven vehicle 1. The remote assistance system 100 then notifies the remote assistance person of the concern message MSG via the remote assistance terminal 110.

[0029] 2 is a block diagram showing an example of a functional configuration related to the notification of a concern message MSG by the remote support system 100. The processing described with reference to FIG. 2 is executed, for example, by cooperation between the management device 120 and the remote support terminal 110. Specifically, as functional blocks related to the notification of a concern message MSG, the management device 120 includes a support request acquisition unit 124, an allocation unit 125, and a concern message acquisition unit 126. The remote support terminal 110 includes a concern message notification unit 116. Note that these functional blocks are realized in software when the above-mentioned management program and control program are executed by the processors 122 and 114, respectively.

[0030] The assistance request acquisition unit 124 acquires assistance request information REQ transmitted from the autonomously driven vehicle 1. As already described, the assistance request information REQ includes the "contents of the remote assistance request" from the autonomously driven vehicle 1. An example of the content of the remote assistance request for driving assistance is, "Please assist me in starting the vehicle while paying attention to pedestrians on the left." In other words, the content of the remote assistance request may include a target that the remote assistance person should recognize or focus on. Furthermore, an example of the content of the remote assistance request for service assistance is, "Please close the door." In other words, the content of the remote assistance request may include the content of the service assistance requested of the remote assistance person. The assistance request acquisition unit 124 transmits the assistance request information REQ to each of the allocation unit 125 and the concern message acquisition unit 126.

[0031] The allocation unit 125 allocates the remote supporter and the remote support terminal 110 to remotely support the automatically driven vehicle 1 based on the received support request information REQ.

[0032] The concern message acquisition unit 126 acquires a concern message MSG based on the received support request information REQ. The concern message MSG does not indicate "the content of the remote support request itself (i.e., what the remote supporter will do)," but rather "the anticipated concern if the remote supporter performs the requested remote support incorrectly (i.e., what will happen if the remote supporter makes a mistake)." Examples of combinations of the content of the remote support request (in other words, the content of the requested remote support) and the concern message MSG will be described later with reference to FIG. 3.

[0033] (Example of getting a concern message) 3 is a diagram for explaining a first example of acquiring a concern message MSG. In the first example, the concern message acquiring unit 126 uses a "concern message database" to acquire a concern message MSG. The concern message database indicates a correspondence (combination) between the contents of a plurality of predetermined types of remote support requests and a plurality of types of concern messages MSG.

[0034] Among the examples of concern messages MSG shown in FIG. 3, MSG_EX1 to MSG_EX5 relate to driving assistance, and MSG_EX6 to MSG_EX8 relate to service assistance.

[0035] Specifically, the concern message MSG_EX1 is in response to a remote assistance request to check for pedestrians beside a crosswalk when autonomous vehicle 1 passes through a crosswalk without a traffic light. The concern message MSG_EX1 indicates the possibility of "a collision between vehicle 1 and a pedestrian" as a possible concern if the remote assistance related to the remote assistance request is inappropriate.

[0036] The concern message MSG_EX2 is in response to a remote assistance request for assistance in having autonomous vehicle 1 start off while paying attention to pedestrians on the left, and indicates the above concern that "vehicle 1 may collide with a pedestrian."

[0037] The concern message MSG_EX3 is in response to a remote assistance request for assistance in recognizing traffic light displays ahead, and indicates the above concern that "vehicle 1 may collide with a pedestrian or another vehicle."

[0038] The concern message MSG_EX4 is in response to a remote assistance request for assistance in recognizing a falling object, and indicates the above concern, for example, that "if the object is a small animal rather than a paper bag, the object may be run over," or "if the object is a stone rather than a paper bag, the bottom of vehicle 1 may be damaged."

[0039] The concern message MSG_EX5 is in response to a remote assistance request for assistance in recognizing tree branches, and indicates the above concern that "the vehicle may hit a hard tree branch and damage the body of the vehicle."

[0040] Concern message MSG_EX6 is in response to a remote assistance request for assistance in closing the door of autonomous vehicle 1, and indicates the above concern that "a person may be caught in the door and injured, or vehicle 1 may move off while the person is still trapped."

[0041] The concern message MSG_EX7 is in response to a remote assistance request for assistance in opening the door of autonomous vehicle 1, and indicates the above concern that "a person leaning on the door may fall out of the vehicle."

[0042] The concern message MSG_EX8 is in response to a remote support request for assistance in checking the safety of the interior of the vehicle before departure, and indicates the above-mentioned concern that "people not holding on to the straps may fall."

[0043] The above-mentioned concern message database is stored, for example, in the storage device 123 of the management device 120. The concern message acquisition unit 126 (processor 122) acquires the concern message MSG associated with the content of the remote support request included in the support request information REQ by referring to the concern message database.

[0044] 4 is a diagram for explaining a second example of acquiring the message of concern MSG. In the second example, the message of concern acquisition unit 126 uses the "machine learning model 127" to acquire the message of concern MSG.

[0045] 4, the machine learning model 127 is trained to output a concern message MSG in response to the content of an input remote support request. The machine learning model 127 is stored, for example, in the storage device 123 of the management device 120. The concern message acquisition unit 126 (processor 122) acquires the concern message MSG by inputting the content of the remote support request included in the support request information REQ into the machine learning model 127.

[0046] The concern message acquisition unit 126 transmits the concern message MSG acquired as described above to the remote support terminal 110 assigned by the assignment unit 125, together with the contents of the remote support request.

[0047] The remote support terminal 110 receives the content of the remote support request and the concern message MSG from the management device 120. The concern message notification unit 116 notifies the remote supporter of the content of the remote support request and the concern message MSG.

[0048] Specifically, the concern message notification unit 116 displays, for example, a visual message as the concern message MSG on the output device 112 (display device). The visual message is, for example, at least one of text and an image. The concern message notification unit 116 may also output an audio message as the concern message MSG from the output device 112 (speaker). Alternatively, both a visual message and an audio message may be output as the concern message MSG.

[0049] 3.Effects As described above, according to the remote assistance system 100 of this embodiment, a concern message MSG based on assistance request information REQ received from the automatically driven vehicle 1 is notified to the remote assistance person. This allows the remote assistance person to be notified of specific concerns that may arise if incorrect remote assistance is provided. This encourages the remote assistance person to provide remote assistance more carefully and meticulously. As a result, the safety of remote assistance for the automatically driven vehicle 1 is improved.

[0050] Furthermore, by receiving the concern message MSG, the remote supporter can more clearly recognize what to pay attention to in order to prevent the anticipated concern from actually occurring. This also contributes to improving the safety of remote support. In relation to this effect, the following explanation is provided. That is, in the above-mentioned example of a remote support request requesting "check for pedestrians beside a crosswalk without a traffic light," the remote supporter is only requested to check for pedestrians, and the remote supporter is not requested to assist in checking for oncoming vehicles. Therefore, in this example, the remote support system 100 according to the present embodiment only notifies the remote supporter of the above concern, as in the above-mentioned concern message MSG_EX1, that "vehicle 1 may collide with a pedestrian." On the other hand, in another example of a remote support request requesting not only check for pedestrians but also check for oncoming vehicles, the remote support system 100 according to the present embodiment notifies the remote supporter of the above concern, not only that "vehicle 1 may collide with a pedestrian" but also that "vehicle 1 may collide with an oncoming vehicle." [Explanation of symbols]

[0051] 1 Autonomous driving vehicle, 100 Remote assistance system, 110 Remote assistance terminal, 114, 122 Processor, 115, 123 Storage device, 116 Concern message notification unit, 120 Management device, 124 Assistance request acquisition unit, 126 Concern message acquisition unit, 127 Machine learning model

Claims

1. A remote assistance system for remote assistance to an autonomous driving vehicle, one or more processors; the one or more processors: acquire assistance request information including the content of a remote assistance request from the autonomous driving vehicle; acquire, based on the assistance request information, a concern message for conveying a concern that is anticipated when a remote assistance person erroneously provides remote assistance to the autonomously driven vehicle; The remote supporter is notified of the concern message through a remote support terminal operated by the remote supporter during the remote support. It was configured as Remote support system.

2. The remote assistance system according to claim 1, The system further includes one or more storage devices that store a concern message database that indicates a correspondence between the contents of a plurality of types of remote support requests and a plurality of types of concern messages; The one or more processors are configured to acquire the concern message associated with the content of the remote assistance request included in the assistance request information by referring to the concern message database. Remote support system.

3. The remote assistance system according to claim 1, The remote assistance system further includes one or more storage devices that store a machine learning model trained to output a concern message regarding the content of the remote assistance request input. The one or more processors are configured to obtain the concern message by inputting the content of the remote assistance request included in the assistance request information into the machine learning model. Remote support system.

4. 4. A remote assistance system according to claim 1, The content of the remote support request includes a target that the remote supporter should recognize or focus on. Remote support system.

5. 4. A remote assistance system according to claim 1, The content of the remote support request includes the content of the service support requested from the remote supporter. Remote support system.

Images