Remote support system

Abstract

To reduce a load on a remote supporter who performs a remote support.SOLUTION: A remote support system includes one or a plurality of processors. The one or the plurality of processors acquire a plurality of tasks which are obtained by dividing the tasks according to the contents of a remote support to be requested from an automatic operation vehicle and also whose execution sequence is fixed. The one or the plurality of processors execute display processing for a remote support terminal to be operated by a remote supporter to display one or a plurality of videos required for executing the tasks, so as to allow the remote supporter to execute the plurality of tasks in the execution sequence.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to technology for remotely assisting autonomous vehicles. [Background technology]

[0002] Patent Document 1 discloses a remote assistance system. The remote assistance system determines recommended options for combinations of assistance processing and delegated processing based on a performance database that shows the performance of assistance request processing by autonomous vehicles for each scene. The remote assistance system then presents the determined recommended options to an operator. The assistance processing is vehicle processing performed by an operator on behalf of a target autonomous vehicle. On the other hand, the delegated processing is processing that the operator leaves to the target autonomous vehicle.

[0003] Patent Document 2 also discloses a remote support server. When the remote support server receives requests for remote support from at least two vehicles in the same time period, it organizes these vehicles into at least one group, determines the support order for the vehicles in the at least one group for each group, and transmits surrounding image information of the vehicles to at least one support device that provides remote support for the vehicles organized in the at least one group in accordance with the support order. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-148824 [Patent Document 2] Japanese Patent Publication No. 2023-054400 Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to further reduce the burden on remote supporters who provide remote support. [Means for solving the problem]

[0006] The remote assistance system according to the present disclosure includes one or more processors. The one or more processors are configured to acquire a plurality of tasks, each of which is divided and obtained according to the content of remote assistance requested by an autonomously driving vehicle, and which have a predetermined execution order. The one or more processors are also configured to execute a display process that displays one or more images required for executing the plurality of tasks on a remote assistance terminal operated by a remote assistance person, so that the remote assistance person can perform the plurality of tasks according to the execution order. [Effects of the Invention]

[0007] According to the present disclosure, the remote supporter does not need to determine the order in which remote support should be provided, thereby reducing the burden on the remote supporter. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram for explaining an overview of a remote support system according to an embodiment. [Figure 2] 10 is a flowchart illustrating an example of a flow of a process related to remote support according to an embodiment. [Figure 3] FIG. 4 is a diagram for explaining a first display example of the display process according to the embodiment. [Figure 4] FIG. 10 is a diagram for explaining a second display example of the display process according to the embodiment. [Figure 5] FIG. 10 is a diagram for explaining a third display example of the display process according to the embodiment. [Figure 6] FIG. 10 is a diagram for explaining a fourth display example of the display process according to the embodiment. [Figure 7] 10 is a flowchart showing an example of a processing flow regarding a first example of use of similarity according to the embodiment. [Figure 8] 10 is a flowchart showing an example of a processing flow regarding a second example of use of similarity according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0010] 1. Overview of the remote support system 1 is a conceptual diagram illustrating an overview of a remote assistance system 100 according to an embodiment. The remote assistance system 100 is a system for remotely assisting an autonomously driven vehicle 10. The remote assistance system 100 includes the autonomously driven vehicle 10, a remote assistance terminal 20, and a management device 30.

[0011] The autonomous vehicle 10 (hereinafter simply referred to as the vehicle 10) is equipped with, for example, a level 4 or higher autonomous driving function that does not require a driver. An example of the vehicle 10 is an autonomous bus that operates within a limited area. The vehicle 10 communicates with the management device 30 and requests remote assistance as needed. In response to this remote assistance request, the remote assistance system 100 remotely assists the vehicle 10. More specifically, a human remote assistant (operator) remotely assists the vehicle 10 via the remote assistance terminal 20. The remote assistance system 100 can also be said to be a system that assists the remote assistant in remotely assisting the autonomous vehicle 10.

[0012] The remote support terminal 20 is a terminal device operated by a remote supporter when remotely supporting the vehicle 10. In other words, the remote support terminal 20 is configured to be used by the remote supporter to remotely support the autonomously driving vehicle 10. The remote support terminal 20 includes a display device 27. The display device 27 displays information such as the situation of the area in which the vehicle 10 is traveling, the position and status of the vehicle 10, etc. The remote supporter monitors the information displayed on the display device 27. Then, the remote supporter remotely supports the vehicle 10 as necessary.

[0013] The remote assistance includes remote assistance related to the driving of the vehicle 10 (driving assistance) and remote assistance related to services provided by the vehicle 10 (service assistance).

[0014] A situation in which driving assistance from a remote supporter is necessary is, for example, a situation in which autonomous driving is difficult. The vehicle 10 performs various vehicle processes while performing autonomous driving. Examples of vehicle processes include a "recognition process," an "action determination process," and a "timing determination process." Specifically, in the recognition process, the vehicle 10 uses a recognition sensor to recognize the situation around the vehicle 10. For example, the vehicle 10 uses a camera to recognize the signal display of a traffic light (e.g., green light, yellow light, red light, right turn signal, etc.). In the action determination process, the vehicle 10 determines whether to perform an action based on the result of the recognition process. Examples of actions of the vehicle 10 include starting, stopping, turning right, turning left, changing lanes, etc. In the timing determination process, the vehicle 10 determines the execution timing for performing the above action.

[0015] Driving assistance is assistance related to each of the vehicle processes described above. For example, when sunlight hits a traffic light installed at an intersection, the accuracy of recognizing the traffic light display may decrease. If the recognition process is unable to accurately distinguish the traffic light display, the vehicle 10 requires remote assistance from a remote assistant for signal recognition. In this way, "assistance in recognizing the traffic light display" corresponds to an example of driving assistance. Furthermore, if the traffic light display cannot be distinguished, it is difficult to determine what action should be taken and when. Therefore, the vehicle 10 requires remote assistance from a remote assistant for action determination processing and timing determination processing. In this way, "assistance in starting judgment (Go / Not Go) and assistance in determining the starting timing" also corresponds to an example of driving assistance.

[0016] In an example where vehicle 10, which has doors for passengers to get on and off, provides passengers with a service to automatically open and close the doors, "assistance in determining whether to open and close the doors and the timing of opening and closing the doors" corresponds to an example of service support by a remote supporter. Also, in an example where autonomous vehicle 10 provides passengers with a service to check the safety inside the vehicle, "assistance in checking the safety inside the vehicle" corresponds to an example of service support.

[0017] The management device 30 manages the remote support system 100. The management device 30 may be configured with multiple servers that perform distributed processing. The management device 30 manages multiple remote supporters and multiple remote support terminals 20. For example, in response to a remote support request from a vehicle 10, the management device 30 assigns one remote supporter and one remote support terminal 20 to remotely support the vehicle 10. Alternatively, the management device 30 may assign multiple remote supporters and multiple remote support terminals 20 to remotely support one vehicle 10. Furthermore, in order to provide remote support to multiple vehicles 10 collectively, the management device 30 may assign multiple remote supporters and multiple remote support terminals 20 to remotely support the multiple vehicles 10.

[0018] When remote support from a remote supporter is required, the vehicle 10 transmits "support request information REQ" indicating a request for remote support to the management device 30. The management device 30 receives the support request information REQ from the vehicle 10. The support request information REQ includes the content of the remote support requested by the vehicle 10.

[0019] The management device 30 notifies the remote supporter of the received support request information REQ via the remote support terminal 20, and requests instructions for the vehicle 10 from the remote supporter. The remote supporter determines instruction INS for the vehicle 10 according to the content of the remote support included in the support request information REQ. More specifically, when remote support is provided, the remote support terminal 20 displays, on the display device 27, an image I captured by the camera 17 mounted on the vehicle 10 that is the target of remote support. The remote supporter views the image I displayed on the display device 27 and determines instruction INS for the vehicle 10. The remote supporter inputs the instruction INS into the remote support terminal 20. The remote support terminal 20 (or the management device 30) communicates with the autonomously driven vehicle 10 and notifies the vehicle 10 of the instruction INS from the remote supporter.

[0020] The vehicle 10 receives instructions from the remote support person from the remote support terminal 20 (or the management device 30). Then, the vehicle 10 drives (automatically drives) according to the instructions, or performs services such as opening and closing doors.

[0021] In addition, the above-mentioned video I used during remote assistance is transmitted in real time from the vehicle 10 to, for example, the management device 30 via communication. The transmitted image information is transferred from the management device 30 to the remote assistance terminal 20. Alternatively, the video I may be transmitted directly from the vehicle 10 to the remote assistance terminal 20.

[0022] 2.Configuration example 2-1.Autonomous vehicles The vehicle 10 includes a communication device 11, a sensor group 12, a traveling device 13, and a control device 14. The communication device 11 performs wireless communication with the outside of the vehicle 10. For example, the communication device 11 performs wireless communication with the remote support terminal 20 and the management device 30.

[0023] The sensor group 12 includes, for example, a recognition sensor, a vehicle state sensor, and a position sensor. The recognition sensor recognizes (detects) the situation around the vehicle 10. The recognition sensor includes one or more cameras 17 (hereinafter simply referred to as cameras 17) that capture multiple images of the situation around the vehicle 10. The vehicle state sensor detects the state of the vehicle 10 (e.g., speed, acceleration, yaw rate, steering angle). The position sensor detects the position and orientation of the vehicle 10.

[0024] The traveling device 13 includes a steering device, a drive device, and a braking device. The steering device steers the wheels of the vehicle 10. The drive device is a power source that generates driving force for the vehicle 10, and includes, for example, at least one of an electric motor and an internal combustion engine. The braking device generates braking force.

[0025] The control device 14 controls the vehicle 10. The control device 14 includes one or more processors 15 (hereinafter simply referred to as processors 15) and one or more storage devices 16 (hereinafter simply referred to as storage devices 16). The processor 15 executes various processes, including processes related to the control of the vehicle 10. Examples of the processor 15 include a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). The storage device 16 stores various information. Examples of the storage device 16 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD). The functions of the control device 14 may be realized by cooperation between the processor 15 (processing circuit) that executes a computer program and the storage device 16. The computer program is stored in the storage device 16. Alternatively, the computer program may be recorded on a computer-readable recording medium or provided via a network.

[0026] 2-2.Remote support terminal The remote support terminal 20 includes a communication device 21, an output device 22, an input device 23, and a control device 24.

[0027] The communication device 21 communicates with the vehicle 10 and the management device 30. The output device 22 outputs various information. The output device 22 includes a display device 27. The display device 27 includes one or more monitors and displays various information to the remote supporter. The output device 22 may include a speaker. The input device 23 accepts input from the remote supporter. Examples of the input device 23 include a touch panel, a keyboard, a mouse, etc.

[0028] The control device 24 controls the remote support terminal 20. The control device 24 includes one or more processors 25 (hereinafter simply referred to as processor 25) and one or more storage devices 26 (hereinafter simply referred to as storage device 26). The processor 25 executes various processes. Examples of the processor 25 include a CPU, a GPU, an ASIC, and an FPGA. The storage device 26 stores various information. Examples of the storage device 26 include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the control device 24 may be realized by cooperation between the processor 25 (processing circuit) that executes a computer program and the storage device 26. The computer program is stored in the storage device 26. Alternatively, the computer program may be recorded on a computer-readable recording medium or provided via a network.

[0029] 2-3.Management device The management device 30 includes a communication device 31, one or more processors 32 (hereinafter simply referred to as processors 32), and one or more storage devices 33 (hereinafter simply referred to as storage devices 33). The communication device 31 communicates with the vehicle 10 and the remote assistance terminal 20.

[0030] The processor 32 executes various processes. Examples of the processor 32 include a CPU, a GPU, an ASIC, and an FPGA. The storage device 33 stores various information. Examples of the storage device 33 include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the management device 30 may be realized by cooperation between the processor 32 (processing circuit) that executes a computer program and the storage device 33. The computer program is stored in the storage device 33. Alternatively, the computer program may be recorded on a computer-readable recording medium, or may be provided via a network.

[0031] 3. Remote support details In remote assistance, the remote assistant may be required to make a decision taking into consideration multiple factors. In other words, remote assistance may involve multiple check items. For example, in remote assistance with the operation of closing a door, the remote assistant must check the video I showing the interior of the vehicle 10 to confirm that no passengers are near the door, and must check the video I showing the exterior of the vehicle to confirm that no pedestrians are trying to get into the vehicle 10. Furthermore, the more check items the remote assistant is required to check, the greater the burden on the remote assistant. Therefore, it is desirable to be able to reduce the burden on the remote assistant required to check multiple check items.

[0032] Therefore, in this embodiment, the remote support system 100 acquires a plurality of tasks T (in other words, a plurality of confirmation items) that are obtained by dividing them according to the content of the remote support requested from the vehicle 10 and that have a predetermined execution order O. Then, the remote support system 100 executes a "display process." The display process is a process of displaying one or a plurality of images I required for executing the plurality of tasks T on the remote support terminal 20 so that the remote supporter can perform the plurality of tasks T according to the execution order O.

[0033] 2 is a flowchart showing an example of a process flow for remote support according to an embodiment. The process of this flowchart is executed, for example, by the management device 30 (processor 32) and the remote support terminal 20 (processor 25) in cooperation with each other.

[0034] In step S100, the management device 30 determines whether or not there is a remote assistance request from the vehicle 10 (more specifically, whether or not assistance request information REQ has been received). As a result, if assistance request information REQ has not been received (step S100; No), the process proceeds to end. On the other hand, if assistance request information REQ has been received (step S100; Yes), the process proceeds to step S102.

[0035] In step S102, the management device 30 executes a process of acquiring a plurality of tasks T according to the content of the remote assistance included in the received assistance request information REQ. For example, in the case of remote assistance for the operation of closing a door, a "task T1 for confirming that no passengers are near the door" and a "task T2 for confirming that no pedestrians are trying to get into the vehicle 10" are acquired as the plurality of tasks T.

[0036] First and second examples of a method for dividing the content of remote support into multiple tasks T will be described in order. In the first example, the management device 30 uses a task database to acquire the multiple tasks T. The task database indicates the correspondence (combination) between multiple types of predetermined remote support content and multiple types of tasks T, and is stored in, for example, the storage device 33. The management device 30 references the task database to acquire multiple tasks T associated with the content of remote support (scene) included in the current support request information REQ. This division method is suitable when the content of remote support requested from the remote supporter is determined to some extent in advance.

[0037] In a second example, the management device 30 uses a machine learning model to acquire multiple tasks T. This machine learning model is trained to output multiple tasks T corresponding to the content of remote support that is input. The machine learning model is stored in, for example, the storage device 33. The management device 30 inputs the content of the remote support included in the current support request information REQ into the machine learning model, thereby acquiring multiple tasks T corresponding to the content.

[0038] 2, the control device 14 (processor 15) of the vehicle 10 may execute the process of acquiring multiple tasks T instead of the management device 30. More specifically, the control device 14 may acquire multiple tasks using the division method described above and transmit support request information REQ including information on the acquired multiple tasks T to the management device 30. Furthermore, when the control device 14 acquires multiple tasks T, the control device 14 may acquire multiple tasks T so as not to include items that can be determined (confirmed) on the vehicle 10 side in the tasks T to be performed by the remote supporter. This allows for more efficient remote support.

[0039] In step S104, the management device 30 executes a process of acquiring the execution order O of the tasks T obtained in step S102. Hereinafter, first and second examples of a method for determining the execution order O will be described in order.

[0040] In a first example, the management device 30 uses an execution order database to determine the execution order O. The execution order database indicates the correspondence (combination) between a plurality of predetermined types of tasks T and a plurality of types of execution orders O, and is stored in, for example, the storage device 33. The management device 30 refers to the execution order database to obtain the execution order O associated with the plurality of tasks T obtained in step S102.

[0041] More specifically, the execution order database may be configured, for example, so that the execution order O of a task T in which the range that the remote supporter needs to view is narrower is set earlier. For example, the range that the remote supporter needs to view when determining the color of a traffic light is narrower than when determining the presence or absence of pedestrians, and the range that the remote supporter needs to view when determining the presence or absence of pedestrians is narrower than when determining the presence or absence of oncoming vehicles. Therefore, the execution order O may be determined so that the task T of determining the color of the traffic light is set earlier, followed by the task T of determining the presence or absence of pedestrians and the task T of determining the presence or absence of oncoming vehicles. Furthermore, in an example of remote support that checks the presence or absence of vehicles in front of and behind the vehicle 10 that is the remote support target, the range that the remote supporter needs to view is narrower in the front than in the rear of the vehicle 10. Therefore, the execution order O of the task T of checking the front may be set earlier than the execution order O of the task T of checking the rear.

[0042] Furthermore, the execution order database may be configured so that the execution order O of a task T whose surrounding circumstances ST, which are the subject of judgment for the task T, are unlikely to change over time, is earlier than the execution order O of a task T whose surrounding circumstances ST are likely to change over time. This makes it possible to appropriately suppress re-execution of a task, as will be described later in Section 4-1.

[0043] In a second example, the management device 30 uses a machine learning model to determine the execution order O. This machine learning model is trained to output the execution order O for multiple input tasks T. The machine learning model is stored in, for example, the storage device 33. The management device 30 inputs the multiple tasks T obtained in step S102 into the machine learning model to obtain the execution order O for the multiple tasks T.

[0044] 2, the control device 14 (processor 15) may execute the process of determining the execution order O instead of the management device 30. More specifically, the control device 14 may determine the execution order O using the above-described determination method and transmit support request information REQ including information on the multiple tasks T for which the execution order O has been determined to the management device 30.

[0045] In step S106 following step S104, the remote support terminal 20 (processor 25) executes the above-mentioned "display process." Specifically, as already explained, during remote support, one or more images I captured by the camera 17 of the vehicle 10 that is the remote support target are displayed on the display device 27 of the remote support terminal 20. The remote support terminal 20 receives information on multiple tasks T, each with a predetermined execution order O, from the management device 30. Then, in the display process, the remote support terminal 20 displays one or more images I required for executing the multiple tasks T on the display device 27 so that the remote supporter performs the multiple tasks T in accordance with the execution order O. Hereinafter, first to fourth display examples of the display process will be explained in order.

[0046] (First display example) 3 is a diagram illustrating a first display example of the display process according to the embodiment. In the first display example, the remote support terminal 20 displays a plurality of images I one by one at a predetermined position P on the screen SC1 of the display device 27 according to an execution order O. In the first display example, the entire screen SC1 corresponds to the "predetermined position P" according to the present disclosure.

[0047] Specifically, Fig. 3 illustrates two images I1 and I2 required for executing two tasks T1 and T2 (execution order O: T1 → T2). As an example, the content of the remote assistance is "assistance in starting the vehicle 10 from a stopped state," task T1 is "confirmation that there are no stopped vehicles ahead of the vehicle," and task T2 is "confirmation that there are no overtaking vehicles behind the vehicle." Image I1 is an image taken by a camera 17 that captures a range of 180° ahead of the vehicle, and image I2 is an image taken by another camera 17 that captures a range of 180° behind the vehicle.

[0048] As shown in FIG. 3 , the remote support terminal 20 first displays the image I1 on the screen SC1 in accordance with the execution order O. That is, the remote support terminal 20 requests the remote supporter to respond (answer) to task T1. When the remote supporter confirms that there are no stopped vehicles ahead of the vehicle after viewing image I1, the remote supporter operates the input device 23 to give an affirmative response indicating that task T1 has been completed. Upon receiving the response, the remote support terminal 20 switches the image I displayed on the screen SC1 from image I1 to image I2. In other words, the shooting direction of image I is switched. When the remote supporter confirms that there are no overtaking vehicles behind the vehicle after viewing image I2, the remote supporter operates the input device 23 to give an affirmative response indicating that task T2 has been completed. This corresponds to the completion of all tasks T1 and T2, i.e., the remote supporter has issued a command to start the vehicle 10. Therefore, the remote support terminal 20 (or the management device 30) that has received the response notifies the vehicle 10 of the start instruction (corresponding to the instruction INS) from the remote supporter, thereby completing the remote support.

[0049] 3 also shows an example in which one remote supporter provides remote support, and multiple images I (e.g., I1 and I2) are displayed on the same screen SC1 of one remote support terminal 20 operated by the remote supporter. However, the first display example may also be applied to remote support in which multiple remote supporters operating multiple remote support terminals 20 share multiple tasks T. More specifically, for example, each time one task T is completed, an image I for the next task T may be displayed on the screen of the remote support terminal 20 operated by the next remote supporter who will be in charge of the next task T according to the execution order O.

[0050] (Second display example) 4 is a diagram illustrating a second display example of the display process according to the embodiment. In the second display example, the remote support terminal 20 also displays a plurality of images I one by one at predetermined positions P on the screen SC2 of the display device 27 according to the execution order O. However, the second display example differs from the first display example in the following points.

[0051] That is, in the second display example, the display device 27 has a screen SC2 that displays three images I1 to I3. As shown in FIG. 4, the screen SC2 can display the images I1 to I3 arranged vertically (up and down on the paper). The screen SC2 may be configured with one monitor or multiple monitors. In addition, in the second display example, the position of the image at the bottom of the screen SC2 corresponds to the "predetermined position P" according to the present disclosure.

[0052] As shown in FIG. 4, the remote support terminal 20 first displays, in accordance with the execution order O, image I1 (e.g., image of the front of the vehicle) for task T1 (e.g., checking the road ahead) at a predetermined position P on the screen SC2. When task T1 is completed, the remote support terminal 20 displays image I2 (e.g., image of the right side of the vehicle) for task T2 (e.g., checking the right side) at a predetermined position P. More specifically, image I1 is hidden, and images I2 and I3 are moved downward. When task T2 is completed, the remote support terminal 20 displays image I3 (e.g., image of the left side of the vehicle) for task T3 (e.g., checking the left side) at a predetermined position P. More specifically, image I2 is hidden, and image I3 is moved downward. In this way, in the second display example, images I1 to I3 from three cameras 17 with different shooting directions are displayed one by one at predetermined positions P in accordance with the execution order O.

[0053] (Third display example) 5 is a diagram illustrating a third display example of the display process according to the embodiment. In the third display example, the remote support terminal 20 displays an auxiliary display indicating the execution order O on the screen SC3 of the display device 27.

[0054] Specifically, the display device 27 has a screen SC3 that displays three images I1 to I3 arranged in the horizontal direction (left-right direction on the paper) as shown in Fig. 5. The images I1 to I3 are the same as those shown in Fig. 4. The screen SC3 may be configured with one monitor or multiple monitors. In addition, in the third display example, a highlighted frame 40 surrounding each of the images I1 to I3 is used as an auxiliary display indicating the execution order O.

[0055] As shown in FIG. 5, the remote support terminal 20, in accordance with the execution order O, first highlights the frame 40 surrounding the image I1 (e.g., image of the front of the vehicle) for task T1 (e.g., checking the road ahead). When task T1 is completed, the remote support terminal 20 highlights the frame 40 surrounding the image I2 (e.g., image of the right side of the vehicle) for task T2 (e.g., checking the right side). When task T2 is completed, the remote support terminal 20 highlights the frame 40 surrounding the image I3 (e.g., image of the left side of the vehicle) for task T3 (e.g., checking the left side). In this way, in the third display example, on a screen SC3 on which a plurality of images I (e.g., I1 to I3) are displayed side by side while a plurality of tasks T are being executed, the frames 40 surrounding the images I corresponding to the tasks T to be executed are highlighted in order.

[0056] Additionally, the highlighting of the frame 40 as an auxiliary display may be performed using an attention-grabbing color such as red, or by flashing the frame 40 a predetermined number of times.

[0057] (Fourth display example) FIG. 6 is a diagram illustrating a fourth display example of the display process according to the embodiment. In the fourth display example, the remote support terminal 20 also displays an auxiliary display indicating the execution order O on the screen SC3 of the display device 27. However, the auxiliary display in the fourth display example is a display of a short message (at least one of letters and numbers) indicating the execution order O. The short messages are, for example, order 1, order 2, and order 3, and are displayed directly on the screen SC3 together with the images I1 to I3. Furthermore, as illustrated in FIG. 6, the short messages may be displayed in combination with messages indicating the contents of tasks T1 to T3 (e.g., check ahead, check right ahead, check left ahead). In addition, for example, the short message such as order 1 may be displayed larger than the messages indicating the contents of tasks T1 to T3.

[0058] Furthermore, the display of a short message as an auxiliary display may be applied to an example of remote support in which only one image I is displayed on the screen of the display device 27 as an image required for the execution of multiple tasks T. More specifically, a combination of a short message indicating the execution order O for each task T (e.g., Order 1, Order 2, ...) and a message indicating the content of task T may be displayed on the screen together with one image I.

[0059] According to the remote support system 100 described above, the remote supporter does not need to determine the order in which remote support should be performed. This reduces the burden on the remote supporter. More specifically, by using multiple tasks T that are obtained by dividing them according to the content of the remote support and whose execution order O is determined as described above, remote support that requires complex situational judgment can be reduced to multiple simple judgment elements (items to be checked). The remote supporter simply needs to execute such multiple tasks T in order according to the display on the remote support terminal 20 (display device 27). This reduces the burden on the remote supporter appropriately.

[0060] 4. Use of similarity of surrounding circumstances 4-1. Determine whether the task needs to be executed again In the above-described remote assistance in which multiple tasks T are executed one by one according to a predetermined execution order O, it is desirable to appropriately consider the temporal validity (validity period) of a positive answer (judgment result) regarding a task T that is completed early during the execution of the remote assistance. The reason for this is that the surrounding situation ST of the vehicle 10, which is the judgment target for the task T that is completed early, may change before all of the multiple tasks T that make up the remote assistance are completed.

[0061] Therefore, when providing remote assistance, the remote assistance system 100 may additionally execute a process that uses the similarity DS, as in the following first and second use examples. Specifically, the multiple images I are images of multiple surrounding conditions ST of the vehicle 10 captured by the camera 17. Each of the multiple tasks T requires the remote assistance person to make a judgment about one surrounding condition ST included in the multiple surrounding conditions ST. The similarity DS here is the similarity of the image I of the surrounding conditions ST (in other words, the degree of change in the surrounding conditions ST).

[0062] (First use case) 7 is a flowchart showing an example of a process flow relating to a first example of use of the similarity DS according to the embodiment. The process of this flowchart is executed by, for example, the management device 30 (processor 32). The process is started, for example, when remote assistance is started.

[0063] In step S200, the management device 30 determines whether or not there is a completed task T (completed task Tc) among the multiple tasks T included in this remote support. This determination is made, for example, based on whether or not the management device 30 has received a positive response indicating that a certain task T has been completed. If there is a completed task Tc (step S200; Yes), the process proceeds to step S202. On the other hand, if there is no completed task Tc (step S200; No), the process proceeds to S208.

[0064] In step S202, the management device 30 calculates the similarity DS of the completed task Tc. As an example, the calculation of the similarity DS can be performed using a known video (image) recognition technique that uses video features extracted from the video I of the surrounding situation ST that is the target of judgment in the completed task Tc. The calculated similarity DS is the similarity based on the video I of the surrounding situation ST at the time of completion of the completed task Tc, i.e., the similarity of the video I of the current surrounding situation ST to the reference video I.

[0065] In step S204 following step S202, the management device 30 determines whether the calculated similarity DS is lower than a predetermined threshold. As a result, if the similarity DS has decreased (step S204; Yes), the process proceeds to step S206. On the other hand, if the similarity DS is not lower than the threshold (step S204; No), the process proceeds to step S208.

[0066] In step S206, the management device 30 requests the remote supporter to re-execute the completed task Tc whose similarity DS has decreased via the remote support terminal 20. More specifically, the management device 30 corrects the execution order O so that the completed task Tc is executed again after the multiple tasks T have been completed according to the original execution order O. In addition, if the similarity DS of the completed task Tc is not lower than the threshold, the management device 30 does not request the remote supporter to re-execute the completed task Tc.

[0067] In step S208, the management device 30 determines whether the current remote support has been completed. This determination is made, for example, based on whether all tasks T have been completed without the similarity DS falling below a threshold. As a result, if the remote support has not been completed (step S208; No), the processes of steps S200 to S206 are repeatedly executed each time an individual task T is completed. On the other hand, if the remote support has been completed (step S208; Yes), the process proceeds to END.

[0068] According to the first use example, by using the similarity DS, it is possible to appropriately suppress re-execution of the task. Also, it is possible to reduce the load on the remote supporter while ensuring the reliability (time validity) of the judgment result of the remote supporter for each task T.

[0069] (Second use case) 8 is a flowchart showing an example of a process flow relating to a second example of use of the similarity DS according to the embodiment. The process of this flowchart differs from the process of the flowchart shown in FIG. 7 in the following points.

[0070] In FIG. 8, if a completed task Tc exists (step S200; Yes), the process proceeds to step S300. In step S300, the management device 30 determines whether a predetermined time has elapsed since the completion of the completed task Tc. The likelihood of the similarity DS decreasing may differ depending on the content of the task T. For this reason, the predetermined time may be determined for each individual task T. If the predetermined time has elapsed (step S300; Yes), the process proceeds to step S302. On the other hand, if the predetermined time has not elapsed (step S300; No), the process proceeds to step S208. In other words, if the predetermined time has not elapsed, the similarity DS will not be calculated even if a completed task Tc exists.

[0071] In step S302, the management device 30 calculates the similarity DS of the completed task Tc for which the above-mentioned predetermined time has elapsed. In step S302, the calculated similarity DS is also the similarity between the image I of the current surrounding situation ST and the image I of the surrounding situation ST at the time of completion of the completed task Tc. Thereafter, the process proceeds to step S204.

[0072] According to the second use example, the time when the similarity DS needs to be determined is specified and the similarity DS is calculated (see steps S300 and S302). This reduces the computation load of the similarity DS while achieving the same effect as the first use example.

[0073] 4-2.Determining the execution order of tasks The similarity DS of the surrounding circumstances ST may be used, for example, as follows to determine the execution order O of multiple tasks T. That is, an execution order database (see step S104) indicating the correspondence between multiple tasks T and their execution orders O may be configured so that the execution order O of a task T for which the similarity DS is easy to calculate (determine) is earlier than the execution order O of a task T for which the similarity DS is difficult to calculate (determine).

[0074] Compared with a task T whose execution order O is later, a task T whose execution order O is earlier takes longer from the completion of the task T until the completion of remote assistance, so there is a greater chance of an opportunity to calculate the similarity DS to determine whether or not the task needs to be executed again (see Section 4-1). According to the example of determining the execution order O described here, it becomes easier to easily determine the similarity DS for a task T whose calculation opportunity is more likely to occur, so it becomes possible to appropriately determine whether or not the task needs to be executed again.

[0075] An example of remote assistance to which this determination example is applied is as follows. That is, in the example of remote assistance for a right turn of a vehicle 10 having multiple tasks T, including "task T1 for confirming that a right-turn arrow signal is displayed" and "task T2 for confirming that there are no oncoming vehicles that would obstruct the right turn," it can be said that determining a change in the signal display of a traffic light using the similarity DS (signal recognition) is easier than recognizing an oncoming vehicle while traveling using the similarity DS. Therefore, in this example, the execution order O is determined so that task T1 is performed before task T2. [Explanation of symbols]

[0076] 10 Autonomous driving vehicle, 15, 25, 32 Processor, 17 Camera, 20 Remote support terminal, 27 Display device, 30 Management device, 40 Frame, 100 Remote support system

Claims

1. Acquire a plurality of tasks that are divided and obtained in accordance with the content of remote assistance requested by the autonomous driving vehicle, and whose execution order is determined; A display process is executed to display one or more images required for executing the plurality of tasks on a remote support terminal operated by the remote supporter so that the remote supporter can perform the plurality of tasks in the execution order. one or more processors configured to Remote support system.

2. The remote assistance system according to claim 1, the one or more images include a plurality of images; In the display process, the one or more processors display the plurality of images one by one at predetermined positions on the screen of the remote support terminal in accordance with the execution order. Remote support system.

3. The remote assistance system according to claim 1, In the display process, the one or more processors display an auxiliary display indicating the execution order on a screen of the remote support terminal. Remote support system.

4. 4. A remote assistance system according to claim 1, the one or more images include a plurality of images; The plurality of images are images of a plurality of surrounding situations of the autonomously driven vehicle captured by one or more cameras, Each of the plurality of tasks requires the remote supporter to make a judgment about a peripheral situation included in the plurality of peripheral situations, the one or more processors: When a task included in the plurality of tasks is completed, a similarity of an image of the surrounding situation that is the subject of the judgment in the completed task, which is the completed task, is calculated based on the time of completion of the completed task; If the degree of similarity is lower than a threshold, request the remote supporter to re-execute the completed task via the remote support terminal; If the similarity is not lower than the threshold, do not request re-execution of the completed task. Remote support system.

5. 4. A remote assistance system according to claim 1, the one or more images include a plurality of images; The plurality of images are images of a plurality of surrounding situations of the autonomously driven vehicle captured by one or more cameras, Each of the plurality of tasks requires the remote supporter to make a judgment about a peripheral situation included in the plurality of peripheral situations, the one or more processors: After a task included in the plurality of tasks is completed, when a predetermined time has elapsed since the completion of the completed task, the similarity of the video of the surrounding situation that is the subject of the judgment in the completed task is calculated based on the completion time of the completed task; If the degree of similarity is lower than a threshold, request the remote supporter to re-execute the completed task via the remote support terminal; If the similarity is not lower than the threshold, do not request re-execution of the completed task. Remote support system.

Images