Information processing method, information processing device, and program

Abstract

An information processing method according to the present disclosure is executed by an information processing device that assists in remote control, by operators, of a plurality of movable bodies each configured to be able to autonomously move to execute a prescribed task, the remote control including remote monitoring and remote operation, by the operators, of the plurality of movable bodies, and the remote monitoring and the remote operation being based on video data transmitted from each of the plurality of movable bodies. The information processing method involves: when a first remote request having a first priority and a second remote request having a second priority higher than the first priority are generated within a prescribed period, performing an emergency treatment for one remote request from among the first remote request and the second remote request; lowering the priority of said one remote request as a result of the emergency treatment; and then performing remote control for the other remote request.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] In recent years, various types of services using autonomous mobile objects have been put into practical use, and remote control systems that can remotely monitor and operate these mobile objects are being developed. In these remote control systems, an operator is assigned in response to a request for assistance (remote request) from a mobile object that requires remote control, and the assigned operator then intervenes in the control of the mobile object, such as remote operation.

[0003] For example, Patent Document 1 discloses a technology in a remote control system for vehicle driving that performs remote control according to priorities that change depending on the vehicle's driving environment in order to reduce the possibility of the vehicle's driving being unable to be properly controlled.

[0004] Japanese Patent Application Laid-Open No. 2023-046333

[0005] In such a situation, in remote control that takes into account the priorities of multiple autonomously traveling mobile objects, it is conceivable that multiple remote requests may be issued from multiple mobile objects within a predetermined period of time. However, with regard to remote control that takes into account the priorities of multiple autonomously traveling mobile objects, sufficient consideration has not been given to how to deal with the case where multiple remote requests with different priorities are issued within a predetermined period of time, and there is room for improvement.

[0006] One of the objectives of the present disclosure is to provide remote control that takes into account the priority of multiple autonomously moving vehicles, and to be able to respond appropriately even when multiple remote requests with different priorities occur within a specified period of time.

[0007] An information processing method according to the present disclosure is executed by an information processing device that supports an operator in remote control of a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task, by the operator, including remote monitoring and remote operation of the plurality of mobile objects based on video data transmitted from each of the plurality of mobile objects. The information processing method includes, when a first remote request having a first priority and a second remote request having a second priority higher than the first priority are generated within a predetermined period, implementing emergency measures for either the first remote request or the second remote request, lowering the priority of the one remote request by the emergency measures, and then implementing remote control for the other remote request.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of a remote control system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of each device included in the remote control system according to an embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of an information processing device that realizes each device included in the remote control system according to an embodiment. FIG. 4 is a flowchart illustrating an example of the flow of information processing executed in a server device according to an embodiment. FIG. 5 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 6 is a diagram illustrating an example of a relationship between remote request content and priority according to an embodiment. FIG. 7 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 8 is a diagram illustrating an example of a relationship between an operator's response status and remote control according to an embodiment. FIG. 9 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 10 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 11 is a diagram illustrating an example of a relationship between remote request content and an estimated response time according to an embodiment. FIG. 12 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 13 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 14 is a diagram illustrating an example of a screen display on an operator terminal according to an embodiment. FIG. 15 is a diagram illustrating an example of a display on an operator terminal according to an embodiment. FIG. 16 is a diagram illustrating an example of a relationship between first aid for an approaching pedestrian and priority after the treatment according to an embodiment. Fig. 17 is a diagram illustrating an example of a screen display in an operator terminal according to the embodiment. Fig. 18 is a diagram illustrating an example of a relationship between first aid in response to an approaching pedestrian and an estimated response time according to the embodiment. Fig. 19 is a diagram illustrating an example of a screen display in an operator terminal according to the embodiment. Fig. 20 is a diagram illustrating an example of a screen display in an operator terminal according to the embodiment. Fig. 21 is a diagram illustrating an example of a screen display in an operator terminal according to the embodiment. Fig. 22 is a diagram illustrating an example of a screen display in an operator terminal according to the embodiment. Fig. 23 is a diagram illustrating a first application example. Fig. 24 is a diagram illustrating a second application example. Fig. 25 is a diagram illustrating a third application example.

[0009] Hereinafter, embodiments of an information processing method, an information processing system, an information processing device, a mobile object, a program, and a recording medium according to the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] In the description of the present disclosure, components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings may be given the same reference numerals, and descriptions thereof may be omitted as appropriate. Furthermore, even when the same or substantially the same parts are shown, the dimensions and proportions may be different depending on the drawing. Furthermore, for example, in order to ensure the visibility of the drawings, reference numerals may be given to only the main components in the description of each drawing, and reference numerals may not be given to components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings.

[0011] (Configuration example of remote control system) Fig. 1 is a diagram showing an example of the configuration of a remote control system 1 according to an embodiment. As shown in Fig. 1, the remote control system 1 includes a server device 10, a plurality of operator terminals 20a, 20b to 20m (m is any integer), and a plurality of vehicles 30a, 30b to 30n (n is any integer).

[0012] 1, the server device 10 and each of the plurality of vehicles 30a, 30b to 30n are connected to each other so as to be able to communicate with each other via a network N such as the Internet. Also, the server device 10 and each of the plurality of operator terminals 20a, 20b to 20m are connected to each other so as to be able to communicate with each other via any telecommunications line such as a LAN (Local Area Network). Note that communication between the server device 10 and each of the plurality of operator terminals 20a, 20b to 20m may be realized via the network N.

[0013] As an example, the remote control system 1 according to the embodiment can be constructed by applying edge computing. In this case, for example, a vehicle 30 is used as the network edge portion, but other devices may also be used as the edge portion.

[0014] In the description of the present disclosure, when the multiple operator terminals 20a, 20b to 20m are not distinguished from one another, they are simply referred to as "operator terminal 20." Similarly, when the multiple vehicles 30a, 30b to 30n are not distinguished from one another, they are simply referred to as "vehicle 30."

[0015] The number (m) of operator terminals 20 and the number (n) of vehicles 30 included in the remote control system 1 can be changed as desired depending on design conditions, etc. In addition, the server device 10 in the remote control system 1 may be realized by the cooperation of multiple server devices.

[0016] Here, the server device 10 according to the embodiment is an example of an information processing device (remote control support device) that performs remote control support such as remote monitoring and operation of the vehicle 30. The operator terminal 20 according to the embodiment is an example of an information processing device (remote control terminal) that an operator uses for remote control in a remote control room. The vehicle 30 according to the embodiment is an example of a mobile body capable of autonomous driving, and is used to provide various services.

[0017] As an example, the mobile object is configured to be capable of performing various tasks, including autonomous driving, related to various services provided by the remote control system 1, such as delivery, security, cleaning, childcare, nursing care, sales, farm work, manufacturing, loading and unloading, transportation, and construction. The mobile object is not limited to the vehicle 30; various mobile objects configured to be movable at least in response to remote operation by an operator can be used as appropriate. The mobile object may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Furthermore, the mobile object may be, for example, an automatic guided vehicle (AGV), or various robots such as construction machinery, agricultural machinery, aircraft, and drones. Furthermore, these mobile objects may transport people, transport objects other than people, or provide specific services other than transportation.

[0018] The vehicle 30 of the present disclosure is a mobile body configured to move autonomously and perform predetermined tasks. The vehicle 30 is configured to move and perform predetermined tasks according to remote control by an operator who monitors multiple mobile bodies. As an example, the vehicle 30 transmits images captured by a camera mounted on the vehicle to the server device 10. As an example, when the vehicle 30 falls into a state where it is unable to travel autonomously, such as when an obstacle such as a fallen object or a vehicle parked on the road is detected in its path, the vehicle 30 transmits an assistance request (remote request) to the server device 10 requesting remote control, i.e., assistance through remote operation.

[0019] As an example, the server device 10 transmits an image captured by a camera mounted on the vehicle 30 to the operator terminal 20 that performs remote monitoring. As an example, when the server device 10 receives a request for assistance (remote request) from the vehicle 30, the server device 10 transmits a remote operation request (remote request) to the operator terminal 20, requesting the operator to perform remote operation.

[0020] As an example, the operator terminal 20 displays a display screen including an image captured by a camera mounted on the vehicle 30. The display screen includes at least one image of at least one vehicle 30 for which the operator operating the operator terminal 20 is responsible for remote control. The operator monitors (monitors) the status of each of the at least one vehicle 30 under their control while viewing the display screen of the operator terminal 20. As an example, when the operator terminal 20 receives a remote operation request (remote request) from the server device 10, the operator terminal 20 notifies the operator operating the terminal that assistance via remote operation (remote control) has been requested. The operator operates the operator terminal 20 while viewing the display screen to remotely operate the vehicle 30 for which assistance has been requested (remote request), and provides assistance (remote control), such as moving the vehicle 30.

[0021] As such, the remote control system 1 according to this embodiment is an information processing system configured to be capable of executing an information processing method (remote control support method) that supports remote control, in which an operator remotely controls a vehicle 30, based on images for remote control captured by a camera mounted on the vehicle 30.

[0022] The configurations of the server device 10, the operator terminal 20, and the vehicle 30 will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of each device (server device 10, operator terminal 20, and vehicle 30) included in the remote control system 1 according to the embodiment. Fig. 2 illustrates the configuration of one operator terminal 20 and one vehicle 30 included in the remote control system 1, but the configurations of the other operator terminals 20 and vehicles 30 included in the remote control system 1 are also similar.

[0023] (Example of Vehicle Configuration) First, the configuration of the vehicle 30 will be described.

[0024] As shown in FIG. 2 , the vehicle 30 includes a sensor 301 , a remote control request determination unit 302 , a control information acquisition unit 303 , a control unit 304 , and a drive unit 305 .

[0025] The sensor 301 includes at least one sensor for acquiring information about the vehicle itself and its surroundings. The output (sensor information) of the sensor 301 is provided to the remote request determination unit 302 and the control unit 304. As an example, the sensor 301 includes one or more cameras (imaging devices) that capture images of the surroundings of the vehicle 30. As an example, the sensor 301 includes at least one sensor that detects at least one of the state, speed, acceleration, current position, presence or absence of surrounding objects, number of surrounding objects, type of surrounding objects, and distance to the surrounding objects of the vehicle 30.

[0026] The remote request determination unit 302 determines, based on the sensor information from the sensor 301, whether or not the vehicle is in a situation requiring remote control, i.e., whether or not to send a support request (remote request) to the server device 10.

[0027] As an example, the remote request determination unit 302 acquires information about surrounding objects based on sensor information from the sensor 301. Here, surrounding objects refer to objects detected around the vehicle 30, such as surrounding obstacles such as fallen objects or walls, or surrounding traffic participants (surrounding traffic participants) such as pedestrians, bicycles, motorcycles, and automobiles. Note that stopped traffic participants, such as parked vehicles, may be treated as obstacles. In other words, the remote request determination unit 302 acquires information about the situation around the vehicle 30 based on the sensor information from the sensor 301. For example, the remote request determination unit 302 acquires information indicating the presence or absence of surrounding objects. For example, the remote request determination unit 302 acquires information indicating the type of surrounding object, such as "pedestrian," "car," "bicycle," "motorcycle," or "obstacle." For example, the remote request determination unit 302 acquires information indicating the number of surrounding objects, such as "2 pedestrians," "2 cars," or "1 obstacle." For example, the remote request determination unit 302 acquires information indicating the positions of surrounding objects, such as "pedestrian: 6 m to the rear right, 5 m to the rear left," "two cars: 25 m to the front right," or "one obstacle: 4 m in front." For example, the remote request determination unit 302 acquires information indicating whether a surrounding object is present on the driving route of the host vehicle. For example, the remote request determination unit 302 acquires information indicating whether a surrounding object is approaching the host vehicle, or information indicating whether there is a possibility of a collision between the surrounding object and the host vehicle.

[0028] As an example, the remote request determination unit 302 determines whether remote control is necessary based on the surrounding conditions of the vehicle 30 indicated by the information about the surrounding objects. For example, if the "surrounding conditions of the vehicle" are "no surrounding objects," the remote request determination unit 302 determines that remote control is "not necessary." For example, if the "surrounding conditions of the vehicle" are "one stopped vehicle detected ahead and present on the driving route," the remote request determination unit 302 determines that remote control for "street parking avoidance" is "necessary." For example, if the "surrounding conditions of the vehicle" are "an obstacle detected ahead and present on the driving route," the remote request determination unit 302 determines that remote control for "obstacle avoidance" is "necessary." For example, if the "surrounding conditions of the vehicle" are "a pedestrian detected around and does not appear to be approaching," the remote request determination unit 302 determines that remote control is "not necessary." Here, the "necessity" of remote control refers to a case in which a support request (remote request) is transmitted from the vehicle 30 to the server device 10. Similarly, remote control is "not" necessary when a request for assistance (remote request) is not sent from the vehicle 30 to the server device 10.

[0029] The above-mentioned criteria are merely examples, and other criteria may be used, or a combination of multiple criteria may be used. The criteria used for each determination may be predetermined and stored in the memory of the control device of the vehicle 30, for example.

[0030] Note that the acquisition of information about surrounding objects based on sensor information from sensor 301 may be performed by control unit 304. In other words, remote control request determination unit 302 may be configured to acquire information about surrounding objects from control unit 304, and determine whether or not remote control is required based on the surrounding conditions of vehicle 30 indicated by the information about the surrounding objects.

[0031] Furthermore, if the remote request determination unit 302 determines that a situation requires remote control, it transmits a request for assistance (remote request) to the server device 10. This remote request includes, for example, information indicating the content and type of the remote request and information on surrounding objects, such as the number of surrounding traffic participants. For example, the remote request includes information indicating that a remote request has been generated. For example, the remote request includes information indicating the content of the generated remote request, such as passing a pedestrian, avoiding parking on the road, avoiding falling objects, or following a child. For example, the remote request includes information indicating the location where the remote request has been generated, such as (300, 200). For example, the remote request includes information indicating the time (time) when the remote request was generated, such as "11:25:30." For example, the remote request includes information on the surrounding conditions of the vehicle 30, such as information on traffic participants surrounding the vehicle 30.

[0032] Furthermore, the remote request determination unit 302 transmits information regarding the vehicle status of the host vehicle for remote control to the server device 10 based on sensor information from the sensor 301, etc. As an example, the information regarding the vehicle status of the host vehicle for remote control includes video (image) data obtained by capturing images of the surroundings of the vehicle 30, which is displayed to the operator during remote monitoring and control. As an example, the information regarding the vehicle status of the host vehicle for remote control includes various information indicating the current status of the vehicle 30, such as the remaining battery level, temperature, speed, acceleration, and current location, which is presented to the operator during remote monitoring and control. Note that if the remote request determination unit 302 determines after transmitting the remote request that remote control is no longer necessary, or if autonomous driving can be resumed without remote control by the operator, the remote request determination unit 302 may transmit information indicating that autonomous control will be resumed and autonomous driving will be resumed to the server device 10.

[0033] The control information acquisition unit 303 acquires information (control information) regarding the control contents such as steering, acceleration / deceleration, stopping, starting, and voice output of the vehicle 30, which are transmitted from the emergency response information transmission unit 113 and automatic stop control information transmission unit 116 of the server device 10, and the operator response information transmission unit 207 of the operator terminal 20.

[0034] The control unit 304 controls the operations related to various tasks of the vehicle 30, including steering and driving such as accelerating and decelerating the vehicle 30, based on sensor information from the sensor 301 or control information acquired by the control information acquisition unit 303.

[0035] The driving unit 305 is driven under the control of the control unit 304. The driving unit 305 is configured to include a mechanism capable of performing operations for executing various tasks of the vehicle 30, such as a mechanism that can drive the vehicle 30 in a predetermined direction for a predetermined distance at a predetermined speed.

[0036] (Configuration Example of Server Device) Next, the configuration of the server device 10 will be described.

[0037] As shown in FIG. 2 , the server device 10 includes a remote request information acquisition unit 101, a priority calculation unit 102, an operator status acquisition unit 103, an operator status determination unit 104, a priority determination unit 105, a remote request response time calculation unit 106, an interruption possibility determination unit 107, an interruption possibility time calculation unit 108, a remote request response determination unit 109, an emergency treatment determination unit 110, an emergency treatment time required calculation unit 111, an emergency treatment content determination unit 112, an emergency treatment information transmission unit 113, a remote request information transmission unit 114, an automatic stop control information generation unit 115, and an automatic stop control information transmission unit 116.

[0038] The remote request information acquisition unit 101 acquires information transmitted from the remote request determination unit 302 of the vehicle 30. As an example, the remote request information acquisition unit 101 receives a support request (remote request) transmitted from the remote request determination unit 302 of the vehicle 30 in a situation where remote control is required, and notifies the priority calculation unit 102 of a response to the received remote request. The remote request information acquisition unit 101 also receives information on the vehicle status for remote control transmitted from the remote request determination unit 302 of the vehicle 30, and transmits the received information on the vehicle status for remote control to a corresponding operator terminal 20 via, for example, the remote request information transmission unit 114.

[0039] The priority calculation unit 102 calculates the priority of the generated remote request based on the information of the assistance request (remote request) received by the remote request information acquisition unit 101 from the vehicle 30 .

[0040] The operator status acquisition unit 103 receives the operator response information transmitted from the operator response information transmission unit 207 of the operator terminal 20. This operator response information includes information on the operator status indicating whether or not the operator is currently responding to a remote request.

[0041] The operator status determination unit 104 determines whether the operator is performing remote control based on the information on the operator status received by the operator status acquisition unit 103 from the operator terminal 20. As an example, if the operator is performing remote control (first remote control) in response to a previous remote request (first remote request), the operator status determination unit 104 notifies the priority determination unit 105. As an example, if the operator is not performing remote control, the operator status determination unit 104 notifies the remote request information transmission unit 114 of a response to the remote request (second remote request).

[0042] Here, a case where a new second remote request occurs while the operator is responding to a previous remote request (first remote request), i.e., while performing remote control (first remote control) in response to the first remote request, is an example of a case where multiple remote requests occur from multiple mobile bodies within a specified period of time.

[0043] Furthermore, after the operator status determination unit 104 interrupts the first remote control in response to the first remote request in order to perform the second remote control or emergency treatment in response to the second remote request, the operator status determination unit 104 determines whether the second remote control or emergency treatment has been completed, for example, based on information about the operator status. When the second remote control or emergency treatment is completed, the operator status determination unit 104 notifies the remote request information transmission unit 114 of resumption of response to the first remote request. Alternatively, the operator status determination unit 104 continues the first remote control while suspending response to the second remote request or suspending response after performing emergency treatment, and then determines whether the first remote control has been completed, for example, based on information about the operator status. When the first remote control is completed, the operator status determination unit 104 notifies the remote request information transmission unit 114 of response to the second remote request.

[0044] When the priority determination unit 105 receives a notification from the operator status determination unit 104, i.e., when the operator is performing the first remote control, the priority determination unit 105 determines which remote request has a higher priority: the newly generated second remote request or the first remote request currently being handled. As an example, when the newly generated second remote request has a higher priority than the first remote request currently being handled, the priority determination unit 105 notifies the remote request response time calculation unit 106 to handle the second remote request. As an example, when the currently handled first remote request has a higher priority than the newly generated second remote request, the priority determination unit 105 notifies the remote request information transmission unit 114 to continue handling the first remote request and notifies the automatic stop control information generation unit 115 to stop the vehicle 30 that issued the second remote request.

[0045] The remote request response time calculation unit 106 calculates the response time (required time) required to respond to the newly generated second remote control request by the second remote control.

[0046] The interruption possibility determination unit 107 determines whether the first remote control for the currently handled first remote request can be interrupted. As an example, the interruption possibility determination unit 107 determines whether the first remote control can be interrupted based on the status of traffic participants around the vehicle 30 that issued the currently handled first remote request. As an example, if the currently handled first remote control can be interrupted, the interruption possibility determination unit 107 notifies the interruption possibility time calculation unit 108 of a response to the second remote request. As an example, if the currently handled first remote control cannot be interrupted, the interruption possibility determination unit 107 notifies the automatic stop control information generation unit 115 to stop the vehicle 30 that issued the second remote request. Note that the interruption possibility determination unit 107 may determine whether the first remote control can be interrupted based on the status of traffic participants around the vehicle 30 that issued the currently handled first remote request and the operator's judgment.

[0047] If the interruptibility determination unit 107 determines that interruption is possible, the interruptible time calculation unit 108 calculates an interruptible time during which the first remote control for the first remote request currently being handled can be interrupted.

[0048] The remote request response determination unit 109 compares the required time for the newly generated second remote request calculated by the remote request response time calculation unit 106 with the allowable interruption time for the currently being handled first remote request calculated by the allowable interruption time calculation unit 108. Furthermore, the remote request response determination unit 109 determines how to respond to each remote request based on the comparison result. For example, if the required time for the newly generated second remote request is shorter than the allowable interruption time for the currently being handled first remote control, the remote request response determination unit 109 determines to interrupt the currently being handled first remote control and to handle the newly generated second remote request (second remote control) which has a higher priority. In this case, the remote request response determination unit 109 notifies the remote request information transmission unit 114 of the response to the second remote request and notifies the automatic stop control information generation unit 115 of the stop of the currently being handled vehicle 30 which issued the first remote request. As an example, if the required time for a newly generated second remote request is longer than the allowable interruption time for the first remote request currently being handled, the remote request response determination unit 109 decides to consider emergency measures for the second remote request and notifies the emergency measures determination unit 110 of the response to the second remote request.

[0049] The emergency treatment determination unit 110 determines whether the priority of a newly generated second remote request with a higher priority will be lowered below the priority of the first remote request being handled by the emergency treatment. The emergency treatment determination unit 110 also identifies candidate emergency treatments that will lower the priority of the newly generated second remote request below the priority of the first remote request being handled by the emergency treatment. For example, if the priority of the newly generated second remote request will be lowered below the priority of the first remote request being handled by the emergency treatment, the emergency treatment determination unit 110 notifies the emergency treatment required time calculation unit 111 of candidate emergency treatments for the second remote request. For example, if the priority of the newly generated second remote request does not become lower than the priority of the first remote request being handled by the emergency treatment, the emergency treatment determination unit 110 determines to interrupt the currently handled first remote control and handle the newly generated second remote request (second remote control). In this case, the emergency treatment determination unit 110 notifies the remote request information transmission unit 114 of the response to the second remote request, and notifies the automatic stop control information generation unit 115 of the stop of the vehicle 30 that issued the first remote request and is currently being responded to.

[0050] The first aid time calculation unit 111 calculates the time required for each first aid candidate in response to the second remote request identified by the first aid determination unit 110 .

[0051] The emergency measure content determination unit 112 determines whether any emergency measure candidate for the second remote request calculated by the emergency measure required time calculation unit 111 has a duration shorter than the allowable interruption time of the first remote control. As an example, if there is an emergency measure for the second remote request that has a duration shorter than the allowable interruption time of the first remote control, the emergency measure content determination unit 112 determines the content of the emergency measure to be implemented for the second remote request. For example, the emergency measure content determination unit 112 determines that, among the emergency measures that have a duration shorter than the allowable interruption time of the first remote control, the emergency measure with the lowest priority or the emergency measure with the shortest duration is to be implemented for the second remote request. In this case, the emergency measure content determination unit 112 notifies the emergency measure information transmission unit 113 of the determined content of the emergency measure for the second remote request. For example, if there is no emergency measure for the second remote request that requires a shorter time than the allowable interruption time for the first remote control, the emergency measure content determination unit 112 determines to interrupt the first remote control currently being handled and to perform the second remote control for the newly generated second remote request in accordance with the priority. In this case, the emergency measure content determination unit 112 notifies the remote request information transmission unit 114 of the response to the second remote request and notifies the automatic stop control information generation unit 115 of the stop of the vehicle 30 that issued the first remote request and is currently being handled.

[0052] The emergency treatment information transmitting unit 113 transmits information about the emergency treatment in response to the second remote request determined by the emergency treatment content determining unit 112 to the operator terminal 20. In addition, the emergency treatment information transmitting unit 113 transmits the information about the emergency treatment in response to the second remote request to the vehicle 30 that issued the second remote request as information about the control content of the vehicle 30 (control information).

[0053] The remote request information transmission unit 114 transmits various remote control information related to remote control, including a remote request, to the operator terminal 20. This remote control information includes information for displaying camera images (video), notifications, and the like on the operator terminal 20.

[0054] The automatic stop control information generating unit 115 generates control information for automatically stopping the target vehicle 30 .

[0055] The automatic stop control information transmitting unit 116 transmits the control information generated by the automatic stop control information generating unit 115 to the target vehicle 30 .

[0056] (Configuration Example of Operator Terminal) Next, the configuration of the operator terminal 20 will be described.

[0057] The operator terminal 20 includes a remote request information receiving unit 201 , a remote request information notifying unit 202 , a display unit 203 , a first aid information receiving unit 204 , a first aid information notifying unit 205 , an input unit 206 , and an operator response information transmitting unit 207 .

[0058] The remote request information receiving unit 201 receives various remote control information transmitted from the remote request information transmitting unit 114 of the server device 10. As described above, this remote control information is information related to remote control, including a remote request and information related to the vehicle status for remote control from each vehicle 30.

[0059] The remote request information notification unit 202 notifies the display unit 203 of the information related to the remote control received by the remote request information receiving unit 201. Specifically, the remote request information notification unit 202 transmits information for displaying a camera image (video), a notification, etc. on the operator terminal 20 to the display unit 203 based on the information related to the remote control received by the remote request information receiving unit 201.

[0060] The display unit 203 displays the information notified from the remote request information notifying unit 202 and the first aid information notifying unit 205 on a display device, such as a liquid crystal display, mounted on or connected to the operator terminal 20. For example, the display unit 203 displays a display screen including at least one image for remote operation and monitoring of at least one vehicle 30 transmitted from the server device 10 on a display device mounted on or connected to the operator terminal 20. In addition, this display screen includes various notifications to be presented to the operator based on the information from the remote request information notifying unit 202 and the first aid information notifying unit 205.

[0061] The first aid information receiving unit 204 receives various remote control information transmitted from the first aid information transmitting unit 113 of the server device 10. As described above, this remote control information includes information related to first aid.

[0062] The first aid information notification unit 205 transmits information for displaying a notice or the like on the operator terminal 20 to the display unit 203 based on the information relating to the first aid received by the first aid information receiving unit 204 .

[0063] The input unit 206 acquires the results of the operator's operations on input devices such as a keyboard, mouse, touch panel, and microphone that are mounted on or connected to the operator terminal 20. For example, when there is a remote request or notification of emergency treatment, or when the operator determines to intervene in control during remote monitoring, the operator uses the input unit 206 to input information related to remote control of the vehicle 30. The input unit 206 notifies the operator response information transmission unit 207 of the acquired operation results.

[0064] The operator response information transmitting unit 207 transmits the operator's response in the remote control to the server device 10 based on the notification from the input unit 206. As an example, the operator response information transmitting unit 207 generates information (control information) regarding the details of remote control by the operator, such as steering, accelerating / decelerating, stopping, starting, and audio output of the vehicle 30, based on the notification from the input unit 206, and transmits the information to the vehicle 30 via, for example, the server device 10. As an example, the operator response information transmitting unit 207 generates information (operator response information) indicating the operator's status based on the notification from the input unit 206, and transmits the information to the server device 10.

[0065] 3 is a diagram showing an example of the hardware configuration of the information processing device 4 that realizes the functions of each device (server device 10, operator terminal 20, vehicle 30) included in the remote control system 1 according to the embodiment. The information processing device 4 is a computer that performs overall control of the operation of each device included in the remote control system 1.

[0066] The information processing device 4 that realizes each function of the vehicle 30 may be a computer such as an ECU (Electronic Control Unit) provided inside the vehicle 30, a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit).

[0067] As shown in FIG. 3, the information processing device 4 includes a processor 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 43, and a device I / F (interface) unit 44.

[0068] The processor 41 is at least one processor such as a CPU (Central Processing Unit), etc. As the at least one processor, various types of processors such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. can be used in addition to or instead of the CPU.

[0069] As an example, in an information processing device 4 that realizes each function of the server device 10, a processor 41 comprehensively controls the operation of the information processing device 4, and realizes the functions of a remote request information acquisition unit 101, a priority calculation unit 102, an operator status acquisition unit 103, an operator status judgment unit 104, a priority determination unit 105, a remote request response time calculation unit 106, an interruption possibility judgment unit 107, an interruption possibility time calculation unit 108, a remote request response judgment unit 109, an emergency treatment judgment unit 110, an emergency treatment time required calculation unit 111, an emergency treatment content determination unit 112, an emergency treatment information transmission unit 113, a remote request information transmission unit 114, an automatic stop control information generation unit 115, and an automatic stop control information transmission unit 116.

[0070] As an example, in an information processing device 4 that realizes each function of the operator terminal 20, a processor 41 comprehensively controls the operation of the information processing device 4 and realizes the functions of a remote request information receiving unit 201, a remote request information notification unit 202, a display unit 203, a first aid information receiving unit 204, a first aid information notification unit 205, an input unit 206, and an operator response information transmitting unit 207.

[0071] As an example, in an information processing device 4 that realizes each function of the vehicle 30, a processor 41 comprehensively controls the operation of the information processing device 4 and realizes the functions of a remote request judgment unit 302, a control information acquisition unit 303, and a control unit 304.

[0072] 2 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions possessed by each device included in the remote control system 1 are not limited to these. In this embodiment, the processor 41 loads a program stored in the ROM 42 into the RAM 43 and executes it, thereby realizing the functions possessed by each device, including the functions of each part described above. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0073] Each functional unit of each device included in the remote control system 1 may be divided into two or more functional units. Alternatively, all or part of two or more functional units may be integrated into one functional unit. Furthermore, each functional unit of each device included in the remote control system 1 may be realized by one processor, or may be realized by two or more processors working together. Furthermore, all or part of two or more functional units of each device included in the remote control system 1 may be realized by one processor.

[0074] The ROM 42 is a non-volatile memory that stores various information including programs executed by the processor 41. The memory of the information processing device 4 is not limited to the ROM 42, and various recording media and recording devices such as a hard disk drive (HDD), a solid state drive (SSD), and flash memory can be used as appropriate. The RAM 43 is a volatile memory that has a working area for the processor 41. The device I / F unit 44 is an interface for connecting each device included in the remote control system 1 to other devices of the information processing device 4, such as a communication device (not shown), a display device (not shown), and an input device (not shown).

[0075] Next, an example of the operation of the remote control system 1 according to the embodiment will be described with reference to the drawings. Note that the operation procedures and processing flows described below are merely examples, and the order of steps can be changed, some steps can be deleted, and other steps can be added as desired.

[0076] FIG. 4 is a flowchart showing an example of the flow of information processing executed in the server device 10 according to the embodiment.

[0077] As an example, assume that an operator is remotely controlling a plurality of vehicles 30 assigned to the operator while viewing a display screen displayed by the operator terminal 20. This display screen includes a plurality of remote control images of the plurality of vehicles 30 acquired by the cameras (sensors 301) of the plurality of vehicles 30 to be monitored.

[0078] Here, an example is shown in which four vehicles 30a to 30d are assigned to one operator who operates the operator terminal 20, and four remote control images 701a to 701d of the four vehicles 30a to 30d to be monitored are displayed on each display screen. In this use case, a vehicle 30 in which an event requiring remote control has occurred notifies the operator of a remote control request via the server device 10. The other vehicles 30 continue to perform tasks such as driving as usual.

[0079] In addition, notifications on each display screen may be superimposed on image 701, or may be displayed in an area set aside for displaying notification information, such as by providing an image area for displaying image 701 and a notification area for displaying notification information in the display area for each vehicle 30.

[0080] The display screen of the operator terminal 20 may be a screen or image generated by the operator terminal 20 based on display information from the server device 10, or may display an image (display information) generated in the server device 10.

[0081] The flow in Figure 4 is triggered by the occurrence of an event that requires the operator to remotely control the vehicle, such as crossing an intersection or avoiding an obstacle, i.e., the reception of a remote request issued from any vehicle 30, and is started before the remote request is sent to the operator terminal 20 to request remote control by the operator.

[0082] First, the remote request information acquisition unit 101 acquires remote request information from the vehicle 30 (S101).

[0083] The remote request information acquired from the vehicle 30 may be displayed on the operator terminal 20. FIG. 5 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. FIG. 5 illustrates a display screen 601 when a fallen object 801 (surrounding object) is detected ahead of the vehicle 30c's path and a remote request (event) to "avoid the falling object" is generated. In this case, the operator is "monitoring" or "watching" the other vehicles 30a, 30b, and 30d, and is in a state where he or she can immediately respond to the remote request from the vehicle 30c. The operator recognizes the remote request from the vehicle 30c on the display screen 601 and begins responding to it.

[0084] As shown in FIG. 5 , the display screen 601 may display a notification 711 indicating that a remote control request has been made. The notification 711 is an example display that notifies the operator of the remote control request by highlighting the image 701c of the vehicle 30c in which the remote control request event occurred with a bold outline. Also, as shown in FIG. 5 , the display screen 601 may display a notification 713 indicating the content of the remote control request. The notification 713 is an example display that notifies the operator of the content of the remote control request using a notification message such as "Avoid falling objects." Also, as shown in FIG. 5 , the display screen 601 may display a notification 715 indicating the location where the remote control request has been made. The notification 715 is an example display that notifies the operator of the location where the remote control request has been made using coordinates such as "(300, 200)." Note that, for the other vehicles 30a, 30b, and 30d, the notification 715 may display the current location of the vehicle 30, such as "(50, 700)," "(600, 400)," or "(200, 20)." 5, a notification 717 indicating the time when the remote request occurred may be displayed on the display screen 601. The notification 717 is an example of a display that notifies the operator of the time when the remote request occurred, such as "11:12:23."

[0085] The notification 715 of the location where the remote request occurred is not limited to displaying the detailed location using coordinates such as "(X coordinate, Y coordinate)" but may also display information about the area in addition to or instead of displaying the coordinates if the area to travel in has been determined. The information about the area to travel in may be, for example, the name of a country or region such as "Japan" or "America," the name of a city such as "Osaka" or "Tokyo," or the name of a predetermined region such as "Area A" or "Area B."

[0086] In addition, the notification 717 of the time when the remote request occurred may display the time for each vehicle 30, for example, when an operator is assigned to remotely control vehicles 30 at multiple locations and there is a time difference between the locations where each vehicle 30 is traveling.

[0087] In this way, by configuring the system to notify the operator that a remote control request has occurred, the operator can smoothly carry out remote control or emergency measures.

[0088] The priority calculation unit 102 calculates the priority of the newly generated second remote request (S102).

[0089] As an example, the priority calculation unit 102 calculates the priority of the newly generated second remote request based on the priority information 51 that is set in advance for each type of generated remote request and the information on the surrounding traffic participants of the vehicle acquired in the processing of S101. Specifically, the priority calculation unit 102 calculates the priority of the newly generated second remote request as a value obtained by subtracting the "number of traffic participants that caused the remote request: 1" from the sum of the "priority according to the type of generated remote request" and the "number of surrounding traffic participants approaching the vehicle 30."

[0090] FIG. 6 is a diagram illustrating an example of information 51 indicating the relationship between remote request content and priority according to the embodiment. This information 51 is, for example, preset and stored in the memory of the server device 10. For example, as shown in FIG. 6 , priorities such as "5," "5," "3," "3," "4," "6," etc. are preset for each type of remote request, such as "approaching pedestrian," "approaching bicycle," "avoiding roadside parking," "avoiding fallen objects," "following children," and "approaching vehicles behind." For example, since the first remote request of vehicle 30c currently being handled is "avoiding fallen objects," its priority is calculated as "3," which is the preset priority of "3," plus "1" for the number of surrounding traffic participants, and minus "1" for the number of traffic participants that caused the remote request. For example, if the second remote request of vehicle 30b, which has just occurred, is "approaching pedestrian," and the surrounding traffic participants approaching vehicle 30b are "one pedestrian" and "one vehicle," its priority is calculated as "6" using the priority calculation formula of "5 + 2 - 1."

[0091] Note that information on the calculated priority may be displayed on the operator terminal 20. Fig. 7 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. Fig. 7 illustrates an example of the display screen 602 when, in a state in which a first remote request for "avoiding a falling object" has been issued for vehicle 30c, "one" pedestrian 803 (surrounding object) is detected ahead in the path of vehicle 30b, and "one" vehicle 805 (surrounding object) approaching from behind is detected, resulting in a second remote request (event) for "pedestrian approach."

[0092] As shown in FIG. 6 , a "falling object avoidance" notification 713 and a "pedestrian approaching" notification 714 are display examples on the display screen 602 that notify the operator of the content of the remote request. The display screen 602 may also display a calculated priority notification 719. The notification 719 is a display example that notifies the operator that the vehicle 30b and the vehicle 30c have a "Priority: 6" and a "Priority: 3," respectively. The display screen 602 may display the type and number of surrounding traffic participants involved in the priority calculation using icons or notification text. The display screen 602 may also notify the operator that a newly generated second remote request has a higher priority than the first remote request currently being handled, if applicable.

[0093] In this way, by providing the operator with the priority of each remote control request, the operator can easily determine which remote control request should be prioritized. This allows the operator to smoothly respond to a remote control request with a higher priority, and to easily make decisions such as suspending a remote control operation currently being performed when a new remote control request with a higher priority is generated.

[0094] The operator status acquisition unit 103 acquires the status of the operator (S103).

[0095] As an example, the operator status acquisition unit 103 may acquire information regarding whether the operator is performing remote control. As an example, the operator status acquisition unit 103 may acquire the operator's response status, such as "monitoring," "watching," "stopping / resuming operation," or "remotely operating." As an example, the operator status acquisition unit 103 may acquire details of the operator's response status, such as "responding to remote operation, expected end time of response 13:47."

[0096] The operator status determination unit 104 determines whether the operator is currently performing a first remote control in response to a previously issued first remote request (S104).

[0097] FIG. 8 is a diagram illustrating an example of information 52 indicating a relationship between an operator's response status and remote control according to the embodiment. This information 52 is, for example, preset and stored in the memory of the server device 10. As an example, as illustrated in FIG. 8 , when the information acquired as the operator's response status (operator status) is "monitoring" or "watching," the operator status determination unit 104 determines that the operator is not responding to the first remote control in response to the first remote request. As an example, as illustrated in FIG. 8 , when the information acquired as the operator's response status is "performing a stop / restart operation" or "performing remote operation," the operator status determination unit 104 determines that the operator is responding to the first remote control in response to the first remote request.

[0098] Based on the information indicating the operator status, the operator's response status for each vehicle 30 may be displayed on the operator terminal 20. Fig. 9 is a diagram showing an example of a screen display on the operator terminal 20 according to the embodiment. Fig. 9 illustrates a display screen 603 when a second remote request for "approaching pedestrian" is issued from vehicle 30b during a first remote control of "avoiding falling objects" by the operator for vehicle 30c.

[0099] As shown in FIG. 9 , when there is a vehicle 30 currently being remotely controlled, notifications 711 and 712 indicating which vehicle 30 the operator is currently handling may be displayed on the display screen 603. The notifications 711 and 712 are display examples that provide notifications with different display modes, such as the color, line type, thickness, and blinking of the frame of the image 710. For example, in the present disclosure, the notification 711 illustrated with a bold frame is a display example, for example, of a yellow frame, related to the vehicle 30 for which a remote request has been issued. For example, the notification 712 illustrated with a bold dashed frame is a display example, for example, of a red frame, related to the vehicle 30 for which a remote request has been issued and for which the operator is currently remotely controlling. Note that the notifications 711 and 712 are not limited to being displayed in different modes, and may be provided using notification text indicating which vehicle 30 the operator is currently handling, such as "Remote request issued," "Operator currently handling," or "Handling vehicle 3."

[0100] In this way, with the configuration for displaying the response status of each operator to each vehicle 30, the screen display of an operator who is simultaneously receiving multiple remote requests will have the frame of the vehicle 30 that the operator is responsible for filled with yellow. Even in this state, it is easy to determine that there is room for other operators who are not simultaneously receiving remote requests, as the frame is not colored on the screen display of other operators. Therefore, it is easy to determine whether or not to support an operator who is simultaneously receiving multiple remote requests. Furthermore, not only the operators, but also the person (manager) who makes the decision to assign operators to support can easily determine which operators to assign to which operators.

[0101] If the operator is not currently performing the first remote control in response to the previously issued first remote request (S104: No), the operator status determination unit 104 performs the second remote control in response to the second remote request (S105). Specifically, the operator status determination unit 104 notifies the remote request information transmission unit 114 of the response to the remote request (second remote request) acquired in the processing of S101. Then, the flow of FIG. 4 ends.

[0102] On the other hand, if the operator is currently responding to a first remote control for a previously issued first remote request (S104: Yes), the priority determination unit 105 determines whether the priority of the newly issued second remote request is higher than the priority of the first remote request currently being responded to (S106).

[0103] If the currently-being-responded first remote request has a higher priority than the newly-issued second remote request (S106: No), the priority determination unit 105 continues the first remote control (S107). Specifically, the priority determination unit 105 notifies the remote request information transmission unit 114 of the continuation of the response to the first remote request, and notifies the automatic stop control information generation unit 115 of the stop of the vehicle 30 that issued the second remote request. Then, the flow in FIG. 4 ends.

[0104] Note that the remote control system 1 according to the present disclosure may notify the operator of the newly generated remote request and notify the operator that the first remote control will be continued if the currently-operated first remote request has a higher priority than the newly generated second remote request. Alternatively, the remote control system 1 according to the present disclosure may not notify the operator of the newly generated remote request if the currently-operated first remote request has a higher priority than the newly generated second remote request. This prevents the operator from becoming confused about which remote request to respond to due to the notification. Furthermore, if a notification is to be given, the operator may also be notified that the first remote control will be continued, thereby preventing confusion among the operator.

[0105] On the other hand, if the first remote control currently being handled has a priority of "avoiding roadside parking, priority: 3" and the newly generated second remote control has a priority of "approaching pedestrian, priority: 5," the priority determination unit 105 notifies the remote request response time calculation unit 106 of a response to the second remote request, because the priority "3" of the first remote control currently being handled is lower than the priority "5" of the newly generated second remote control.

[0106] Note that the priority comparison result and the request to the operator based on the comparison result may be displayed on the operator terminal 20. Fig. 10 is a diagram showing an example of a screen display on the operator terminal 20 according to the embodiment. Fig. 10 illustrates a display screen 604 in a case where, during a first remote control of "avoiding a falling object" by the operator for vehicle 30c, a second remote request of "approaching pedestrian" is issued from vehicle 30b, which has a higher priority than the first remote request.

[0107] 10 , the display screen 604 may display a notification 721 indicating a request for an operator to take action based on the comparison result, such as "Remote control of vehicle 3 suspended, action recommended for vehicle 2." The display screen 604 may also display a notification 723 indicating the priority comparison result, such as "Remote control request from vehicle 2, high priority," which indicates which of the two has a higher priority. The display screen 604 may also display a notification indicating the content of a newly generated second remote request with a high priority and a situation related to the second remote request, such as "Vehicle approaching from behind."

[0108] When a new remote control request occurs during remote control, the remote control system 1 according to the present disclosure may notify the request in a manner that distinguishes whether the remote request has a higher priority than the vehicle currently being handled or a lower priority. As an example, the remote control system 1 according to the present disclosure may use different notification modes for high-priority remote requests and low-priority remote requests. For example, the remote control system 1 according to the present disclosure may display high-priority remote request notifications in a red frame on the display screen 604 and low-priority remote request notifications in a blue frame. The color of the notification frame is merely an example and can be changed as appropriate. The notification mode is not limited to the frame color, and may also be different, such as by using different line types or blinking.

[0109] In this way, by displaying the priority comparison result and the request content to the operator based on the comparison result, the operator can be informed that the newly generated second remote request has a high priority and that a request for handling it is required, which can facilitate the operator's subsequent response. Also, it can help the operator decide whether to interrupt or continue the first remote request that is currently being handled.

[0110] If the newly generated second remote request has a higher priority than the first remote request currently being handled (S106: Yes), the remote request response time calculation unit 106 calculates the required time for the second remote control for the second remote request (S108).

[0111] As an example, the remote request response time calculation unit 106 calculates the required time for the second remote control based on an estimated response time that is set in advance for each type of remote request that has occurred. Specifically, the remote request response time calculation unit 106 may calculate the value of the "estimated response time for the occurred remote request" as the required time for the second remote control of the newly occurred second remote request. Alternatively, the remote request response time calculation unit 106 may calculate the value obtained by subtracting "the number of traffic participants that caused the remote request: 1" from the "estimated response time for the occurred remote request" as the required time for the second remote control of the newly occurred second remote request. Note that the remote request response time calculation unit 106 may calculate the required time for the second remote control based further on information about traffic participants around the vehicle 30. Specifically, the remote request response time calculation unit 106 may calculate the required time for the second remote control of the newly generated second remote request as the sum of the ``estimated response time for the generated remote request'' and the ``number of surrounding traffic participants,'' such as the number of traffic participants within x meters around the vehicle, minus the ``number of traffic participants that caused the remote request to be generated: 1.''

[0112] FIG. 11 is a diagram illustrating an example of information 53 indicating the relationship between remote request content and estimated response time according to the embodiment. This information 53 is, for example, preset and stored in the memory of the server device 10. For example, as shown in FIG. 11 , estimated response times such as "7 minutes," "6 minutes," "10 minutes," "5 minutes," "12 minutes," "4 minutes," etc. are preset for each type of remote request, such as "approaching pedestrian," "approaching bicycle," "avoiding roadside parking," "avoiding fallen objects," "following children," "approaching rear vehicle," etc. For example, if a newly generated second remote request is "approaching pedestrian" and the "surrounding traffic participants within 5 meters of the vehicle" are "one pedestrian" and "one vehicle," the required time is calculated as "8 minutes" using the response time calculation formula "7 + 2 - 1."

[0113] The calculation result of the required time for responding to the generated remote control request may be displayed on the operator terminal 20. FIG. 12 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 12 , the display screen 605 may display a notification 725 of an "estimated response time" for the second remote control of the generated second remote request, such as "estimated response time: 8 minutes," for the vehicle 30b for which a new second remote request has been generated. Furthermore, the display screen 605 may display a notification 726 of an "estimated response completion time" for the vehicle 30c, such as "estimated response completion time: 4 minutes," calculated by subtracting the "elapsed time since response" from the "estimated response time," when a first remote control is being handled in a situation in which a new second remote request has been generated.

[0114] In this way, by displaying the calculation results of the time required to respond to the second remote request that has occurred, the operator can easily determine whether or not the first remote control that is currently being responded to can be interrupted, whether to interrupt the first remote control that is currently being responded to and perform the second remote control in response to the second remote request, or whether to provide emergency measures in response to the second remote request.

[0115] After the required time for the second remote control in response to the second remote request is calculated, the interruption possibility determination unit 107 determines whether the ongoing first remote control can be interrupted (S109).

[0116] As an example, the interruption possibility determination unit 107 determines whether or not the operation can be interrupted based on information indicating the presence or absence of a temporary evacuation location around the vehicle 30 that issued the first remote request currently being handled. For example, assume that map information (such as road widths and potential evacuation locations) regarding potential evacuation locations is prepared in advance and stored in a location accessible to the interruption possibility determination unit 107, such as an internal memory. For example, the interruption possibility determination unit 107 acquires surrounding situation information for each of a group of potential evacuation locations within a predetermined range (within ○ meters of the current location) around the vehicle 30 that issued the first remote request currently being handled. The surrounding situation information includes, for example, information indicating the presence or absence of traffic participants, such as people and vehicles. For example, the interruption possibility determination unit 107 determines the presence or absence of a potential evacuation location based on the surrounding situation information for each of the group of potential evacuation locations. For example, a potential evacuation location is a potential evacuation location that does not contain any traffic participants. For example, the interruption possibility determination unit 107 determines that the operation can be interrupted if at least one potential evacuation location is present.

[0117] For example, when there is no place to take shelter, the interruption possibility determination unit 107 may determine whether it is possible to stop the vehicle on the spot and interrupt the operation, based on the vehicle information and the map information. For example, the interruption possibility determination unit 107 may determine whether it is possible to overtake or pass other vehicles, based on the vehicle width from the vehicle information and the road width from the map information.

[0118] As an example, if there is no place to retreat to, the interruption possibility determination unit 107 may determine whether or not the first remote request is interruptible based on the status of traffic participants around the vehicle 30 that has issued the currently-served first remote request. For example, the interruption possibility determination unit 107 determines that the first remote request is interruptible if the number of surrounding traffic participants is greater than a predetermined number. For example, the interruption possibility determination unit 107 determines that the first remote request is interruptible if the surrounding traffic participants are within a predetermined range (within ○ meters). For example, the interruption possibility determination unit 107 determines that the first remote request is interruptible if there is a surrounding traffic participant approaching the vehicle 30 within the predetermined range (within △ meters) or if there is a surrounding traffic participant with a possibility of collision.

[0119] As an example, in addition to the above conditions, the operator may determine whether or not the operation can be interrupted. This allows the operator to flexibly determine whether or not the operation can be interrupted, taking into consideration detailed circumstances attributable to the operator, such as their own level of proficiency and confidence in their ability to respond.

[0120] The remote control system 1 may present the operator with interruption support information to assist in determining whether or not interruption is possible. This interruption support information may, for example, display a location where the vehicle currently in the process of responding can be temporarily stopped when it is recommended to interrupt the process and respond to a remote request with a higher priority. For example, the interruption support information may further display how long the vehicle can be temporarily stopped. For example, the interruption support information may display the time during which the vehicle can be stopped when the process is temporarily interrupted to respond to a remote request with a higher priority. This allows the operator to understand how long the vehicle can be stopped to respond to the remote request with a higher priority.

[0121] The determination result of whether or not interruption is possible may be displayed on the operator terminal 20. Fig. 13 is a diagram showing an example of a screen display on the operator terminal 20 according to the embodiment. As shown in Fig. 13, on the display screen 606, a notification 727 regarding vehicle 30b for which a new second remote request has occurred may be displayed on an image 710 of vehicle 30c currently being handled. This notification 727 is an example of a display of factors for determining whether or not interruption is possible, a result of determining whether or not interruption is possible, such as "Interruption is possible because there are no surrounding traffic participants. Please interrupt the remote control of vehicle 3 and handle vehicle 2," and a request for response to the operator based on the result.

[0122] In this way, by displaying the determination result of whether or not the operation can be interrupted, the operator can easily understand whether or not the operation can be interrupted. Furthermore, the operator can easily understand the subsequent actions to be taken, so that the subsequent actions can be carried out smoothly. Furthermore, since the operator can understand the factors that determined whether or not the operation can be interrupted, when the operator makes the final decision, this information can be one of the factors for making the decision, that is, it can support the operator's decision.

[0123] If the ongoing first remote control cannot be interrupted (S109: No), the interruption possibility determination unit 107 automatically stops the vehicle for which the second remote request has been issued (S110). Specifically, the interruption possibility determination unit 107 notifies the automatic stop control information generation unit 115 of the stop of the vehicle 30 that issued the second remote request. Then, the flow in FIG. 4 ends.

[0124] In addition, the remote control system 1 according to the present disclosure may, after determining that the first remote control being handled cannot be interrupted, periodically determine whether or not the remote control can be interrupted while it is continuing, and if it becomes possible to interrupt the ongoing remote control, interrupt the ongoing remote control and handle the remote request with a higher priority.

[0125] On the other hand, if the currently active first remote control can be interrupted (S109: Yes), the interruptible time calculation unit 108 calculates the interruptible time of the currently active first remote control based on a predetermined rule (S111).

[0126] As an example, the allowable interruption time calculation unit 108 calculates the allowable interruption time based on the status of surrounding traffic participants. For example, when there is a surrounding traffic participant approaching the vehicle 30, the allowable interruption time calculation unit 108 calculates the allowable interruption time as a predicted time until the surrounding traffic participant approaches within a predetermined range (within △ m) around the vehicle 30.

[0127] As an example, the allowable interruption time calculation unit 108 determines a predetermined time as the allowable interruption time.

[0128] As an example, the allowable interruption time calculation unit 108 determines the allowable stopping time according to the stopping rule of the place where the vehicle is stopped as the allowable interruption time. For example, when the vehicle is stopped by evacuating it somewhere, the allowable interruption time calculation unit 108 determines the allowable stopping time according to the stopping rule of the place where the vehicle is stopped as the allowable interruption time.

[0129] As an example, the allowable interruption time calculation unit 108 calculates the allowable interruption time based on the scheduled service provision time. For example, the allowable interruption time calculation unit 108 calculates a predicted scheduled service provision time, which is a predicted scheduled service provision time, from the current time, the current position of the vehicle, and the vehicle's traveling speed, and determines the difference between the scheduled service provision time and the predicted scheduled service provision time as the allowable interruption time. Note that if the value of the difference is negative, i.e., if it is predicted that the scheduled service provision time will not be met at the current time, the allowable interruption time calculation unit 108 may set the allowable interruption time to zero (0). For example, if the value of the difference is zero (0) or negative, i.e., if the allowable interruption time is zero (0), the interruptibility determination unit 107 may determine that interruption is not possible.

[0130] As an example, the allowable interruption time calculation unit 108 calculates the allowable interruption time based on the maximum allowable interruption time and the status of surrounding traffic participants. For example, the maximum allowable interruption time is predetermined to be "10 minutes." For example, when there are no surrounding traffic participants approaching the vehicle 30, the allowable interruption time calculation unit 108 calculates the allowable interruption time by subtracting "the number of surrounding traffic participants" x "30 seconds" from the maximum allowable interruption time. For example, when there are surrounding traffic participants approaching the vehicle 30, the allowable interruption time calculation unit 108 calculates the allowable interruption time as the predicted time until the surrounding traffic participants approach within a predetermined range (within △ meters) around the vehicle 30.

[0131] For example, in the case where there are two surrounding traffic participants and no surrounding traffic participants approaching the vehicle, the interruption time calculation unit 108 calculates the interruption time as "9 minutes" ("10 minutes" - "30 seconds" x 2).

[0132] For example, in a case where "surrounding traffic participants: 1 person, 1 car," "there are surrounding traffic participants approaching the vehicle," and "the predicted time until the approaching car comes within 5 meters of the vehicle: 6 minutes," the interruption time calculation unit 108 calculates an interruption time of "6 minutes."

[0133] The calculated allowable interruption time may be displayed on the operator terminal 20. Fig. 14 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in Fig. 14 , on the display screen 607, a notification 729 regarding the vehicle 30b for which a new second remote request has been issued may be displayed on an image 710 of the vehicle 30c currently being served. This notification 729 is a display example of information used as a reference when calculating the allowable interruption time, such as "No approaching surrounding traffic participants, allowable interruption time: 10 minutes," and the presence or absence of surrounding traffic participants approaching the vehicle among the surrounding traffic participants.

[0134] In this way, by displaying the calculated allowable interruption time, the operator can easily grasp the allowable interruption time. Therefore, the operator can easily grasp how much time he or she can allocate to responding to the vehicle 30 with high priority, which makes it easier to respond. Furthermore, when the operator makes a final decision on whether to interrupt, the information can be used as one of the factors for that decision, i.e., it can support the operator's decision. For example, if the allowable interruption time is known, the operator can not only simply temporarily stop the vehicle, but also appropriately select emergency measures that can be implemented depending on the time, such as calling out to the vehicle.

[0135] The remote request response determination unit 109 determines whether the interruptible time of the first remote control calculated in the process of S111 is shorter than the required time of the second remote control calculated in the process of S108 (S112).

[0136] If the allowable interruption time of the first remote control is longer than the required time of the second remote control (S112: No), the flow of FIG. 4 proceeds to the process of S117.

[0137] On the other hand, for example, when the remote request response determination unit 109 determines that "the remote request that has occurred is a pedestrian approaching" and "surrounding traffic participants within 5 meters of the vehicle are one pedestrian and one vehicle," it calculates "the required time for the second remote control: 7 + 2 - 1 = 8 minutes." Furthermore, for example, when the remote request response determination unit 109 determines that "surrounding traffic participants are one person and one vehicle," "there are surrounding traffic participants approaching the vehicle," and "the predicted time until the approaching vehicle comes within 5 meters of the vehicle is 6 minutes," it calculates "the allowable interruption time: 6 minutes." In such a case, the remote request response determination unit 109 notifies the emergency procedure determination unit 110 of a response to the second remote request because the "allowable interruption time: 6 minutes" is shorter than the "required time for the second remote control: 8 minutes."

[0138] The comparison result between the allowable interruption time of the first remote control and the required time of the second remote control may be displayed on the operator terminal 20. FIG. 15 is a diagram illustrating an example of a display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 15 , on the display screen 608, a notification 731 regarding the vehicle 30b for which a new second remote control request has been generated may be displayed on an image 710 of the vehicle 30c currently being handled. This notification 731 is an example of a display indicating which time was shorter as a result of the comparison, such as "First aid for vehicle 2 will be requested because the allowable interruption time is short," or indicating that the operator should be requested to take action based on the result. For example, if the time required to handle the newly generated second remote request (second remote control) is shorter than the allowable interruption time of the first remote control, the display indicating that the operator should perform the second remote control in response to the newly generated second remote request may be displayed. For example, if the time required to respond to the newly generated second remote request (second remote control) is longer than the time during which the first remote control can be interrupted, the display indicating that the operator should take action based on the results will be, "We request that you implement emergency measures for the newly generated remote request."

[0139] In this way, by displaying the comparison result between the time that the first remote control can be interrupted and the time required for the second remote control, the operator can easily understand the details of the next response, and can respond smoothly. Furthermore, when the operator decides whether to perform remote control or emergency measures in response to the second remote request, this can be assisted in making that decision.

[0140] When interrupting the first remote control in progress, the allowable interruption time may vary depending on how the interruption is performed. Therefore, the remote control system 1 according to the present disclosure may determine whether emergency measures are required, taking into consideration how the interruption is performed. For example, if the vehicle is stopped on the spot, such as in the middle of the road, to respond to the emergency, the allowable interruption time is short. For example, if the vehicle is evacuated to the shoulder of the road, such as in a location where only stopping is possible, the allowable interruption time is medium. For example, if the vehicle is evacuated to the shoulder of the road, such as in a location where parking is possible, the allowable interruption time is long.

[0141] If the time during which the first remote control can be interrupted is shorter than the time required for the second remote control (S112: Yes), the emergency treatment determination unit 110 determines whether the priority of the second remote request will be lower than the priority of the first remote request due to the emergency treatment (S113).

[0142] As an example, the emergency treatment determination unit 110 determines whether the priority of the second remote request will be lower than the priority of the first remote request as a result of the emergency treatment, based on the relationship between the emergency treatment in response to the approach of a pedestrian and the priority after the treatment. FIG. 16 is a diagram illustrating an example of information 54 indicating the relationship between the emergency treatment in response to the approach of a pedestrian and the priority after the treatment according to the embodiment. This information 54 is temporarily stored, for example, in the memory of the server device 10. For example, as shown in FIG. 16 , priorities such as "3," "4," "2," "2," "1," "2," "1," etc. are preset for emergency treatments such as "stop," "call out," "pull over to the shoulder," "stop + call out," "stop + pull over to the shoulder," "call out + pull over to the shoulder," "stop + call out + pull over to the shoulder," etc. For example, in the case where the remote request currently being handled is "Avoid roadside parking, priority: 3" and the remote request newly generated is "Pedestrian approach, priority: 5", there is a candidate emergency measure 541 with a lower priority, so the remote request response determination unit 109 selects the candidate 541 and notifies the emergency measure time calculation unit 111.

[0143] The determination result of whether the priority will be lowered by emergency measures may be displayed on the operator terminal 20. FIG. 17 is a diagram illustrating an example of a screen display on the operator terminal 20 according to an embodiment. As illustrated in FIG. 17 , on the display screen 609, a notification 733 regarding vehicle 30b, for which a new second remote control request has been issued, may be displayed on an image 710 of vehicle 30c currently being handled. This notification 733 is an example of a display of the determination result of whether the priority will be lowered by emergency measures, i.e., information indicating whether emergency measures are meaningful or not, such as "Please perform emergency measures on vehicle 2 because the priority has been lowered." This response request is, for example, a request to the operator to perform emergency measures if the priority will be lowered by emergency measures. For example, the response request is a request to the operator to perform remote control if the priority will not be lowered by emergency measures. The display screen 609 is not limited to this. Alternatively, the notification 733 may display information indicating how much the priority will be lowered.

[0144] In this way, by displaying the result of the determination as to whether emergency measures will lower the priority, the operator can easily understand the details of the next response, allowing for a smooth response. Furthermore, when the operator needs to determine whether to perform remote control or emergency measures in response to the second remote request, this can be assisted in making that determination.

[0145] If the priority of the second remote control request does not become lower than the priority of the first remote control request as a result of the emergency measures (S113: No), the flow of FIG. 4 proceeds to the process of S117.

[0146] On the other hand, if the priority of the second remote request becomes lower than the priority of the first remote request due to emergency treatment (S113: Yes), the emergency treatment time calculation unit 111 calculates the time required for emergency treatment for the second remote request (S114).

[0147] As an example, the first aid time calculation unit 111 calculates the time required for first aid in response to the second remote request based on a predetermined time required for first aid. The first aid time calculation unit 111 may calculate the time required for first aid in response to the second remote request based further on the status of traffic participants around the vehicle 30 performing the first aid. FIG. 18 is a diagram illustrating an example of information 55 indicating the relationship between first aid measures for approaching a pedestrian and estimated response times according to the embodiment. This information 55 is temporarily stored, for example, in the memory of the server device 10. FIG. 18 illustrates candidate first aid measures 541 with lower priorities. For example, as shown in FIG. 18 , estimated response times such as "5 minutes," "2 minutes," "6 minutes," "6 minutes," "7 minutes," etc. are preset for each of first aid measures such as "pulling to the shoulder," "stopping and calling out," "stopping and pulling to the shoulder," "calling out and pulling to the shoulder," "stopping and calling out and pulling to the shoulder," etc.

[0148] For example, the emergency response time calculation unit 111 calculates the response time (required time) for each emergency response by subtracting the number of traffic participants that caused the remote request (1) from the sum of the estimated response time for the emergency response and the number of surrounding traffic participants within a predetermined range (within 0 meters) of the vehicle 30. As an example, if the number of surrounding traffic participants approaching the vehicle is one pedestrian and one vehicle, the response time (required time) for each emergency response is calculated using the response time calculation formula: (estimated response time) + 2 - 1. In the example of FIG. 18 , the response times (required times) calculated for each emergency response, such as "pulling to the shoulder," "stopping and calling out," "stopping and pulling to the shoulder," "calling out and pulling to the shoulder," "stopping and calling out and pulling to the shoulder," and so on, are "6 minutes," "3 minutes," "7 minutes," "7 minutes," "8 minutes," and so on.

[0149] The calculated time required for emergency measures may be displayed on the operator terminal 20. FIG. 19 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 19 , a notification 735 regarding a vehicle 30b for which a new second remote request has been issued may be displayed on the display screen 610 on its image 710b. This notification 735 is an example of a display of the time required for emergency measures, the minimum and / or maximum time required for emergency measures, and the type of emergency measures to be implemented, such as "Estimated response time: Stop + Call: Minimum 3 minutes Stop + Call + Pull over to the shoulder: Maximum 8 minutes." The display screen 610 may also display a notification 735 notifying information about surrounding traffic participants involved in the calculation.

[0150] In this way, by displaying the calculated time required for emergency treatment, it is possible to provide support for the operator in deciding on the content of emergency treatment to be performed in response to the second remote request.

[0151] After the time required for the emergency treatment is calculated, the emergency treatment content determination unit 112 determines whether emergency treatment in response to the second remote control request is possible during the time during which the first remote control can be interrupted (S115).

[0152] As an example, if there is no emergency measure that is shorter than the interruptible time of the first remote control among the response times (required times) of the emergency measures calculated in the processing of S114, the emergency measure content determination unit 112 determines that emergency measure for the second remote request is impossible within the interruptible time of the first remote control. As an example, if there is any emergency measure that is shorter than the interruptible time of the first remote control among the response times (required times) of the emergency measures calculated in the processing of S114, the emergency measure content determination unit 112 determines that emergency measure for the second remote request is possible within the interruptible time of the first remote control. In this case, the emergency measure content determination unit 112 determines to perform, for example, the emergency measure with the lowest priority among the emergency measures that have a required time shorter than the interruptible time of the first remote control.

[0153] The emergency measure content determination unit 112 may select the emergency measure that can be completed in the shortest time, not limited to the emergency measure with the lowest priority. This makes it possible to minimize the impact on the vehicle 30 whose remote control has been interrupted, even if the interruption time is shortened.

[0154] For example, in the example of Figure 18, if the "peripheral traffic participants: 1 person, 1 car," "there are peripheral traffic participants approaching the vehicle," and "predicted time until the approaching vehicle comes within a predetermined range (within 5 m of the vehicle): 6 minutes" is calculated based on this, the emergency measure content determination unit 112 determines that "temporary stop + calling out" should be performed as emergency measure because the response time (required time) calculated for "temporary stop + calling out" is 3 minutes, which is the shortest.

[0155] The result of the determination as to whether emergency treatment is possible may be displayed on the operator terminal 20. FIG. 20 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 20, a notification 737 regarding the vehicle 30b for which a new second remote request has been issued may be displayed on the display screen 611 on its image 710b. This notification 737 is an example of a display of the result of the determination as to whether emergency treatment is possible, which emergency treatment will be performed, and the factors behind the decision to perform the emergency treatment, such as "Allowable interruption time: 6 minutes, Pause + call: 3 minutes, Please perform the above emergency treatment." The notification 737 may display factors for the decision to perform emergency treatment, such as priority, allowable interruption time, and response time.

[0156] In this way, by displaying the result of the determination of whether or not first aid is possible, the operator can easily understand what kind of first aid will be performed, allowing for a smooth response. Furthermore, it is possible to provide support for the operator in determining whether or not first aid can be performed in response to the second remote request and what kind of first aid will be performed.

[0157] If emergency measures can be taken in response to the second remote request during the interruption time of the first remote control (Yes in S115), the emergency measures content determination unit 112 interrupts the ongoing first remote control and implements emergency measures in response to the second remote request (S116). Specifically, the emergency measures content determination unit 112 notifies the emergency measures information transmission unit 113 of the contents of the emergency measures taken in response to the second remote request. The emergency measures information transmission unit 113 then transmits information about the emergency measures taken in response to the second remote request determined by the emergency measures content determination unit 112 to the operator terminal 20. The emergency measures information transmission unit 113 also transmits the information about the emergency measures taken in response to the second remote request to the vehicle 30 that issued the second remote request as information about the control contents of the vehicle 30 (control information).

[0158] As an example, in the vehicle 30, the control unit 304 performs autonomous control of emergency measures based on the control content from the emergency measure information transmission unit 113 acquired by the control information acquisition unit 303. For example, the operator performs remote control of emergency measures in accordance with the content of emergency measures displayed on the display unit 203 and notifications of countermeasures based on the content.

[0159] For example, if the determined emergency measure for the second remote request is "stop and call out," control content to slow down and stop the vehicle 30 is transmitted to the vehicle 30. For example, if the determined emergency measure for the second remote request is "stop and call out," the operator is notified or requested to press a speech button that causes the vehicle 30 to output a voice message saying "You go first" after the vehicle 30 has stopped. Alternatively, for example, if the determined emergency measure for the second remote request is "stop and call out," the operator may be notified or requested to press a speech button that causes the operator to say "You go first" after the vehicle 30 has stopped.

[0160] Note that the operator terminal 20 may display implementation details corresponding to the emergency measures. FIG. 21 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 21 , the display screen 612 displays that the vehicle 30c, which was being subjected to the first remote control, has stopped, and a notification 712 indicates that the operator is currently responding to the vehicle 30b that issued the second remote request. Furthermore, the display screen 612 may display a notification 739 regarding the vehicle 30b that is the target of emergency measures on its image 710b. The notification 739 is an example of a display of the emergency measures, the control to be performed on the vehicle 30, and a specific response request to the operator, such as "Stop + Call. Reduce vehicle speed and stop. Please call out to surrounding traffic participants."

[0161] In this way, with a configuration that displays the implementation details according to the content of the emergency treatment, the operator can easily understand the content of the emergency treatment to be performed, and can therefore respond smoothly.

[0162] If emergency measures cannot be taken in response to the second remote request during the allowable interruption time of the first remote control (S115: No), the emergency measure content determination unit 112 interrupts the first remote control and performs the second remote control (S117). Specifically, because performing emergency measures or responding to the remote request (second remote control) would have a significant impact on the vehicle 30 for which the first remote control was interrupted (exceeding the allowable interruption time), the emergency measure content determination unit 112 interrupts the first remote control currently being handled and performs the second remote control for the newly generated second remote request according to the priority. In this case, the emergency measure content determination unit 112 notifies the remote request information transmission unit 114 of the response to the second remote request and notifies the automatic stop control information generation unit 115 of the stop of the vehicle 30 currently being handled that issued the first remote request.

[0163] For example, the second remote control corresponding to the newly generated second remote request may be a remote intervention to avoid surrounding traffic participants by remote operation. For example, the second remote control corresponding to the newly generated second remote request may be a remote intervention to change the route to avoid being affected by surrounding traffic participants (avoiding collisions). For example, the second remote control corresponding to the newly generated second remote request may be a remote intervention to check for safety and perform a restart operation when restarting a stopped vehicle.

[0164] After the emergency repair or the second remote control in response to the second remote request is completed, the operator status determination unit 104 resumes the suspended first remote control (S118). Specifically, the operator status determination unit 104 determines whether the second remote control or the emergency repair is completed, for example, based on information about the operator status. Then, when the second remote control or the emergency repair is completed, the operator status determination unit 104 notifies the remote request information transmission unit 114 of the resumption of the response to the first remote request. Note that the operator status determination unit 104 may also notify the vehicle 30 of the resumption of the response to the first remote request. In response to this, the operator resumes the first remote control based on the notified content, and the suspended remote control based on the notified control content is resumed at the vehicle 30. Then, the flow of FIG. 4 ends.

[0165] For example, when the operator status judgment unit 104 resumes the first remote control for the vehicle 30 that has suspended roadside parking avoidance, the operator status judgment unit 104 does not notify the vehicle 30 of the control content, but requests the operator to "remotely operate the vehicle to prevent roadside parking that has been suspended."

[0166] A notification to the effect that the first remote control will be resumed may be displayed on the operator terminal 20. FIG. 22 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the embodiment. As illustrated in FIG. 22 , the display screen 613 displays a notification 712 indicating that the operator is currently responding to the first remote control to be resumed. Furthermore, the display screen 612 may display a notification 741 regarding the vehicle 30c for which the first remote control is to be resumed on the image 710b of the vehicle 30b on which the first remote control has been performed or the second remote control. This notification 741 is an example of a display requesting the resumption of the first remote control, such as "Emergency treatment completed. Please resume remote operation of vehicle 3." The display screen 613 may also display a notification 741 notifying the operator of the control to be performed on the vehicle 30 that was suspended or a specific response request to the operator.

[0167] As one example, the remote control (remote intervention) in response to the remote request of "approaching pedestrian" that is implemented (resumed) after the emergency action of "temporarily stopping" (see FIG. 16) has been implemented may be a remote control to avoid the pedestrian if there is still a pedestrian nearby, and then a remote control to resume the driving may be implemented.

[0168] The remote control system 1 according to the present disclosure may select and respond to emergency measures as a remote intervention measure that can be implemented in a shorter time as one of the remote intervention measures in response to a remote request.

[0169] In this way, by displaying a notification that the first remote control will be resumed, the operator can easily understand that the first remote control can be resumed, and therefore can respond smoothly. Furthermore, after emergency measures have been taken, it is possible to provide support for the operator in determining whether to perform the first remote control of the vehicle 30 that was suspended or the second remote control of the vehicle 30 on which emergency measures have been taken.

[0170] In the present disclosure, priority can be expressed as "degree of necessity for immediate response." In other words, priority according to the present disclosure may be calculated based on whether the "degree of necessity for immediate response" is high. This "degree of necessity for immediate response" may also be expressed as the necessity of immediate response.

[0171] One aspect of the "degree of necessity for immediate response" is whether or not the incident will cause inconvenience to surrounding traffic participants (especially people).

[0172] For example, if a nearby traffic participant (person, vehicle) is approaching the robot (approaching XX), continuing to drive the robot is likely to cause a nuisance to the nearby traffic participant (person, driver), and the timing of the nuisance is also early. For this reason, the remote control system 1 according to the present disclosure may set a high priority for a nearby traffic participant (person, vehicle) approaching the robot (approaching XX) from the perspective of "whether or not the nearby traffic participant (especially person) will be inconvenienced." Note that the traveling speed of the nearby traffic participant may also affect the timing of the nuisance. For this reason, the remote control system 1 according to the present disclosure may set a higher priority for an approaching vehicle that is moving at a high speed.

[0173] For example, if a nearby traffic participant follows the robot (a child clinging to the robot), there is a high possibility that the child will block the path ahead, preventing the robot from moving. In such cases, the nearby traffic participant (child) will likely cling to the robot of their own volition, and is unlikely to perceive the child as a nuisance. On the other hand, the inability to move is a nuisance for the person receiving the service. For this reason, the remote control system 1 according to the present disclosure may set a higher priority for a case in which a nearby traffic participant follows the robot (a child clinging to the robot) than for a case in which a nearby traffic participant (a person, a vehicle) is approaching the robot (approaching XX), from the perspective of "whether or not it will cause inconvenience to nearby traffic participants (especially people)."

[0174] For example, when avoiding roadside parking or fallen objects (avoidance of XX), if the vehicle continues traveling, it will stop before colliding with the roadside parking or fallen objects. In such a case, if there are no people around (there are no people in the roadside parking), there are no people in the surrounding traffic participants (people) and the vehicle will not be in the way. For this reason, the remote control system 1 according to the present disclosure may set a low priority for avoiding roadside parking or fallen objects (avoidance of XX) from the perspective of "whether or not it will cause inconvenience to surrounding traffic participants (especially people)."

[0175] For example, in the example of FIG. 6 , a high priority of "5" or higher is set for proximity-related remote request contents such as "approaching pedestrian," "approaching bicycle," and "approaching vehicle behind." Furthermore, among proximity-related remote request contents, a priority of "5" is set for "approaching pedestrian" and "approaching bicycle," while a higher priority of "6" is set for "approaching vehicle behind," which relates to a speeding vehicle. For example, in the example of FIG. 6 , a remote request content such as "keep an eye on children" is set to a priority of "4," which is the second highest after proximity-related remote request contents. For example, in the example of FIG. 6 , a low priority of "3" is set for remote request contents such as "avoid roadside parking" and "avoid falling objects."

[0176] Furthermore, one aspect of the "degree of necessity for immediate response" is whether or not remote control must be implemented urgently.

[0177] For example, when "remote control must be performed urgently," there may be a case where there is no time margin for the vehicle's task, such as when there is no time margin before the arrival deadline. As an example, the remote control system 1 according to the present disclosure may determine whether there is time margin based on the vehicle's task information (destination, presence or absence of an arrival deadline, arrival deadline, etc.), the driving schedule, and the current location of the vehicle. As an example, the remote control system 1 according to the present disclosure may set a high priority for the remote request of the vehicle if there is no time margin. Furthermore, the remote control system 1 according to the present disclosure may set the priority level depending on the amount of time margin.

[0178] For example, a case where "remote control must be performed urgently" may occur in a service where a vehicle transports people, such as a taxi, without an explicit arrival deadline, in order to improve the quality of service to customers. As an example, the remote control system 1 according to the present disclosure may determine whether the quality of the service is related to the responsiveness of the remote control based on the task content and information about the person receiving the service (customer information). As an example, the remote control system 1 according to the present disclosure may set a high priority for a remote request for a vehicle if the quality of the service is related to responsiveness. Furthermore, the remote control system 1 according to the present disclosure may assign a higher or lower priority depending on the degree of contribution to responsiveness. Furthermore, the remote control system 1 according to the present disclosure may assign a higher or lower priority depending on whether the customer pays an additional fee for a service-related option, such as by purchasing the option.

[0179] For example, a case where "remote control must be performed urgently" may be when other vehicles are passing each other on a narrow road and overtaking is not possible, thereby causing an obstruction to other traffic participants. Another case where "remote control must be performed urgently" may be when heavy traffic volume makes it dangerous for a stopped vehicle. These cases also fall under the consideration of "whether or not a nuisance will be caused to surrounding traffic participants (especially people)." For example, the remote control system 1 according to the present disclosure may determine whether a road is narrow based on map information, vehicle information (vehicle width), surrounding traffic participant information, etc., and, if the road is narrow, may set a high priority for the remote control request for that vehicle. For example, the remote control system 1 according to the present disclosure may determine the number of surrounding traffic participants (traffic volume) based on map information, vehicle information (vehicle width), surrounding traffic participant information, etc., and, if traffic volume is high, may set a high priority. Furthermore, the remote control system 1 according to the present disclosure may assign a high or low priority to a vehicle based on road width and traffic volume.

[0180] In addition, for remote requests with a high "degree of need for immediate response," there may be some that can be reduced to a "degree of need for immediate response" through emergency measures, and some that cannot.

[0181] For example, an example of a situation in which the "degree of necessity for immediate response" can be reduced by emergency measures is when stopping a vehicle while waiting for a remote request would cause an obstruction to the traffic of other traffic participants. As an example, if stopping a vehicle while waiting for a remote request would cause an obstruction to the traffic of other traffic participants, the "degree of necessity for immediate response" can be reduced from the perspective of "whether or not it causes an inconvenience to surrounding traffic participants (especially people)" by emergency measures such as moving the vehicle to a location where it will not cause an obstruction.

[0182] For example, an example of a situation in which the "degree of necessity for immediate response" cannot be lowered by emergency measures is when the arrival deadline is approaching and waiting for a remote request will result in the deadline being missed. As an example, if the arrival deadline is approaching and waiting for a remote request will result in the deadline being missed, the "degree of necessity for immediate response" cannot be lowered because the arrival deadline will not be met even if emergency measures are taken.

[0183] The remote control system 1 according to the present disclosure may determine whether the remote request is one for which the "degree of necessity for immediate response" can be reduced by emergency measures, and may decide whether to perform emergency measures based on the determination result. For example, the remote control system 1 according to the present disclosure may determine to perform emergency measures when the remote request is one for which the "degree of necessity for immediate response" can be reduced by emergency measures.

[0184] Hereinafter, application examples of the remote control system 1 according to the present disclosure will be described with reference to the drawings.

[0185] (First Application Example) This application example illustrates a use case of avoiding roadside parking.

[0186] FIG. 23 is a diagram for explaining the first application example.

[0187] First, as illustrated in the upper part of FIG. 23 , assume that the operator of “Vehicle 1,” which issued the first remote request, is currently responding to a “falling object avoidance” operation. The “falling object avoidance” operation (first remote control) is a remote control operation with an “estimated response time” (required time) of “5 minutes,” and the current “estimated response completion time” is “3 minutes.” Furthermore, assume that during the first remote control operation in response to the first remote request from “Vehicle 1,” “Vehicle 2” discovers “Vehicle A” (parked on the street, surrounding object) parked on the street, issues a second remote request, and is currently waiting for a “street parking avoidance” operation (second remote control). The “estimated response time” (required time) of this second remote control operation for “street parking avoidance” is a remote control operation with a “estimated response time” (required time) of “5 minutes,” and the current “estimated response completion time” remains at “5 minutes” because the operation is currently waiting, i.e., before any action has been taken. Therefore, the total response completion time of the operators at the time illustrated in the upper part of FIG. 23 , i.e., the time until the responses to each remote request are completed, is “8 minutes.”

[0188] In this situation, as illustrated in the middle part of FIG. 23 , assume that “Vehicle B” (a surrounding object) approaches “Vehicle 2” from behind while “Vehicle 2” is waiting for the second remote control, and “Vehicle B” parks on the road behind “Vehicle 2,” resulting in “Parking B.” In this case, “Vehicle 2” is sandwiched between “Parking A” in front and “Parking B” behind, making the second remote control of “Vehicle 2” more difficult and requiring a long time to return. Furthermore, even if “Vehicle 2” is configured to be able to autonomously travel simply by reversing, in a situation where a vehicle (a surrounding traffic participant) is approaching from behind, it is difficult for the autonomous traveler to reversing to approach the approaching vehicle without colliding with it in order to secure space for the second remote control, and remote operation by an operator is required. In the example of FIG. 23 , the first remote control of “Vehicle 1” has an “Estimated Response Time: 5 minutes” and an “Estimated Response Completion Time: 2 minutes,” but the “Estimated Response Time” and “Estimated Response Completion Time” of the second remote control of “Vehicle 2” have increased to “10 minutes.” In this case, the time required for "Vehicle 2" to return has increased from "8 minutes" to "12 minutes."

[0189] 23, the remote control system 1 according to the present disclosure performs emergency measures, such as interrupting the first remote control of "Vehicle 1," switching the target of remote control, and moving "Vehicle 2" backward to create space so that the second remote control of "Vehicle 2" becomes less difficult. After lowering the priority of "Vehicle 2" through the emergency measures, the operator resumes the first remote control of "Vehicle 1." In this case, the time required for "Vehicle 2" to return to normal operation is reduced from 12 minutes to 10 minutes through the emergency measures.

[0190] Thus, according to the remote control system 1 of the present disclosure, although the time required for "vehicle 1" to return to autonomous driving is delayed due to the implementation of emergency measures on "vehicle 2," the emergency measures make it easier to perform the second remote control of "vehicle 2," thereby shortening the total time required for "vehicle 1" and "vehicle 2" to return to autonomous driving. Furthermore, the first remote control of "vehicle 1" may be suspended when the operator temporarily suspends the first remote control and assesses the situation of surrounding traffic participants, for example, in a situation where the first remote control cannot be performed due to a large number of oncoming vehicles. In this case, the overall loss in the return time is further reduced. In other words, according to the remote control system 1 of the present disclosure, the time required for returning to autonomous driving is shortened, thereby improving operational efficiency.

[0191] The remote control system 1 according to the present disclosure may be configured to interrupt remote control to improve efficiency and shorten the average waiting time. For example, suppose a first remote request and a second remote request are issued in the order of "Vehicle 1" and "Vehicle 2." Here, the first remote control in response to the first remote request is a remote control with an "estimated response time" (required time), which is the estimated time required for remote intervention, of "3 minutes." The "estimated response time" of this first remote control is equal to the "estimated response completion time" if the request is before the start of the first remote control, and is the time obtained by subtracting the elapsed time from the start of the first remote control from the "estimated response completion time" if the request is after the start of the first remote control. Furthermore, suppose the second remote control in response to the second remote request is a remote control with an "estimated response time" (required time) of "15 seconds."

[0192] For example, consider "Case 1," in which the first remote control is not interrupted and is executed in order. In this case, the second remote control is started after the first remote control is completed. In this "Case 1," the waiting time for "Vehicle 1" is "0 seconds," and the waiting time for "Vehicle 2" is "3 minutes," so the average waiting time for these is "90 seconds."

[0193] For example, consider "Case 2," in which the first remote control is executed in order with an interruption in between. "Case 2" is, for example, a case in which a second remote control request occurs 30 seconds after the first remote control request. In this case, the first remote control is started and then interrupted, the second remote control is executed while the first remote control is interrupted, and the first remote control is resumed after the second remote control is completed or interrupted. In this "Case 2," the waiting time for "Vehicle 1" is 15 seconds, and the waiting time for "Vehicle 2" is 0 seconds, so the average waiting time for these is 7.5 seconds.

[0194] Therefore, "Case 2" provides higher overall efficiency than "Case 1." Therefore, in the above example, the remote control system 1 according to the present disclosure may calculate the average waiting time and determine or suggest to the operator which case to implement so as to shorten the average waiting time. In the above example, the remote control system 1 according to the present disclosure adopts "Case 2," which executes the first remote control in order with an interruption therebetween.

[0195] In this way, the remote control system 1 according to the present disclosure may, upon the occurrence of a second remote request, calculate and compare the average waiting time when the first remote control is interrupted and the average waiting time when the first remote control is not interrupted, and if the average waiting time when the first remote control is interrupted is shorter than the average waiting time when the first remote control is not interrupted, temporarily interrupt the first remote control and prioritize execution of the second remote control. In other words, upon the occurrence of a second remote request, if the time required to complete the first remote control for the currently ongoing first remote request is longer than the time required for the second remote control for the second remote request (or is longer than a predetermined threshold), temporarily interrupt the first remote control and prioritize execution of the second remote control.

[0196] If the first remote control is interrupted too late, a waiting time for the second remote control occurs, and the average waiting time becomes longer. Therefore, in order to shorten the average waiting time, it is preferable to interrupt the first remote control as soon as possible.

[0197] (Second Application Example) This application example illustrates another use case in the case of avoiding roadside parking.

[0198] FIG. 24 is a diagram for explaining the second application example.

[0199] First, as illustrated in the display screen 614 of Fig. 24 , while the operator is responding to the first remote control of "vehicle 3," "vehicle 2" discovers a roadside parking object 807 (surrounding object), issues a second remote request, and waits for the second remote control of "roadside parking avoidance." Also, assume that an approaching vehicle 805 (surrounding object) is approaching "vehicle 2" from behind while waiting for the second remote control. In the situation illustrated in the display screen 614 of Fig. 24 , since the road on which "vehicle 2" is traveling has one lane in each direction, if "vehicle 2" stops on the spot to wait for the second remote control, it may cause inconvenience to the approaching vehicle 805.

[0200] 24 , in the remote control system 1 according to the present disclosure, for example, the first remote control of "vehicle 3" is suspended, and an emergency measure is taken to remotely move "vehicle 2" to a shelter 809 so that "vehicle 2" does not get in the way of the approaching vehicle 805. Note that the suspension of the first remote control of "vehicle 3" may be performed, for example, when the operator temporarily suspends the first remote control and assesses the situation of surrounding traffic participants, such as when there are many oncoming vehicles and the first remote control cannot be performed. After "vehicle 2" has been evacuated, the operator resumes the first remote control of "vehicle 3."

[0201] As such, unlike the case of performing normal second remote control in which the first remote control of "Vehicle 3" is temporarily suspended and "Street Parking Avoidance" of "Vehicle 2" is performed as illustrated on the display screen 616 in FIG. 24 , the remote control system 1 according to the present disclosure performs an emergency response that involves evacuation to the evacuation location 809, resulting in a longer time to return to autonomous driving. With this configuration, while the time to return to autonomous driving is longer, "Vehicle 2" can be moved more quickly than with normal second remote control so as not to interfere with the approaching vehicle 805. In other words, the remote control system 1 according to the present disclosure can achieve a response that does not inconvenience surrounding traffic participants, or that causes inconvenience for a shorter period of time, compared to the case of performing normal second remote control. Therefore, unlike a configuration in which remote control is simply performed sequentially in response to remote requests with higher priority, remote control can be achieved that takes into account safety aspects such as the risk of collision with surrounding traffic participants, thereby improving the safety of remote control.

[0202] Furthermore, in a situation such as the one illustrated on the display screen 614 of FIG. 24 , it is possible to increase the priority of the second remote control and perform the second remote control to avoid causing inconvenience to surrounding traffic participants. However, in order to avoid causing inconvenience to surrounding traffic participants or to minimize the duration of the inconvenience, it is preferable to perform emergency measures without wasting time. However, performing emergency measures requires determining a location where the vehicle will not cause inconvenience, and depending on the situation, it may be necessary to control the vehicle to move closer to the approaching vehicle 805. For this reason, it is difficult to perform emergency measures using autonomous control without wasting time and without causing inconvenience to surrounding traffic participants. In this situation, the remote control system 1 according to the present disclosure allows the operator to perform emergency measures remotely, thereby moving "vehicle 2" out of the way of the approaching vehicle 805 more quickly than when performing the normal second remote control.

[0203] (Third Application Example) This application example illustrates a use case in which vehicles pass each other.

[0204] FIG. 25 is a diagram for explaining the third application example.

[0205] For example, assume that an autonomous mobile robot is used as vehicle 30. While "Vehicle 1" is responding to a first remote request for "avoiding roadside parking" with "Priority: 3" by performing a first remote control, "Vehicle 2" receives a second remote request for "approaching pedestrian" with "Priority: 5." In this case, the newly generated second remote request from "Vehicle 2" has a higher priority than the first remote request for "avoiding roadside parking" currently being responded to by "Vehicle 1," because a failure to respond to the second remote request could result in a collision with a pedestrian. For example, if the second remote control is implemented in response to the second remote request based on the priority, the robot will remotely avoid a collision with a pedestrian or change its driving route to avoid a collision with a pedestrian, and the required time for the second remote control will be "10 minutes or more."

[0206] For example, as illustrated in Fig. 25 , information 56 indicating the relationship between first aid measures taken in response to approaching pedestrians and the priority after the measures is temporarily stored in, for example, the memory of the server device 10. For example, as illustrated in Fig. 25 , priorities such as "3," "4," "2," "2," "1," "2," "1," etc. are preset for each of first aid measures such as "stop," "call out," "pull over to the shoulder," "stop + call out," "stop + pull over to the shoulder," "call out + pull over to the shoulder," "stop + call out + pull over to the shoulder," etc. Furthermore, the information 56 in Fig. 5 further stores the required time for first aid measures taken in response to a second remote request, calculated from the relationship between the first aid measures taken in response to approaching pedestrians and the estimated response time.

[0207] 25, the emergency action content for "pedestrian approach" of "vehicle 2" includes candidate emergency action 561, which has a lower priority than the first remote control currently being handled (priority: 3). In this case, based on information 57 indicating the situation of "vehicle 1," which is "vehicle approaching from behind in 5 minutes," "can be interrupted," and "interruption time limit: 4 minutes," the emergency action of "temporarily stop + call out" is adopted, which can be handled within the interruption time limit of "4 minutes" of the first remote control of "vehicle 1."

[0208] In this way, in a situation where the first remote control would be interrupted for the time required for the second remote control of "10 minutes or more" if the second remote control were to be performed as is based on priority, the remote control system 1 according to the present disclosure can shorten the interruption of the first remote control to the time required for the emergency response of "2 minutes" by lowering the priority of the second remote request through emergency response. Therefore, the remote control system 1 according to the present disclosure can realize remote control that takes into consideration safety aspects such as the risk of collision with surrounding traffic participants, thereby increasing the safety of remote control.

[0209] (Fourth Application Example) This application example illustrates a use case in which remote control or emergency treatment is performed in consideration of service efficiency.

[0210] For example, a mobile object such as a vehicle 30 is configured to be able to perform various tasks, including autonomous driving, related to the "advertising" service provided by the remote control system 1. In this case, the mobile object may travel, for example, toward a crowded area as an "advertising" and may encounter approaching pedestrians while traveling, causing the mobile object to pass by them.

[0211] As a response to passing situations (remote control) according to priority, for example, control to avoid approaching pedestrians or change of driving route is performed.

[0212] However, considering that the service is "advertising," there is a demand to present the advertisement to approaching pedestrians as well. Therefore, in the remote control system 1 according to the present disclosure, for example, when the service is "advertising," an emergency measure is taken, such as moving the mobile object to a position where the advertisement is easily visible to approaching pedestrians and where the mobile object does not interfere with other surrounding traffic participants, and then temporarily stopping the mobile object. In other words, in the case of a predetermined service such as "advertising," the remote control system 1 according to the present disclosure takes an emergency measure instead of remote control.

[0213] On the other hand, in the case of other specified services such as "delivery," the remote control system 1 according to the present disclosure may prioritize remote control over emergency measures, such as using remote control to avoid approaching pedestrians, in order to prioritize arriving at the destination as quickly as possible.

[0214] In this way, the remote control system 1 according to the present disclosure may be configured to determine whether to perform remote control or emergency repair depending on the type and category of the service to be provided, which can improve the efficiency of the services provided.

[0215] (Fifth Application Example) This application example illustrates a use case in which emergency measures are taken in response to a first remote request from the vehicle 30 that has occurred earlier, in order to perform a second remote control for a second remote request that will occur next.

[0216] For example, it may be possible that a high-priority remote request will occur or that there is a high possibility of this occurring based on information about past remote requests. In such a case, the remote control system 1 according to the present disclosure performs emergency response instead of remote control when a first remote request with a low priority occurs. Then, it performs remote control for the next high-priority second remote request.

[0217] As an example, suppose that it is known from past remote requests that passing pedestrians on a narrow road increases the possibility of a collision, and that there are points where high-priority remote control is likely to occur. For example, suppose that a "vehicle A" is traveling on the narrow road and is approaching a point where past information indicates that a remote control request is likely to occur, and then a first remote control request from a "vehicle B" with a lower priority occurs first. In other words, suppose that a second remote control request from a "vehicle A" with a higher priority is expected to occur within a predetermined period based on past information, but the first remote control request from a "vehicle B" with a lower priority occurs first.

[0218] In such a case, since the operator is not currently remotely controlling "vehicle B" at the time the first remote request for "vehicle B," which has a lower priority, is made, it is possible to perform the first remote control for "vehicle B" in response to the first remote request. However, the remote control system 1 according to the present disclosure takes into consideration the possibility that a second remote request for "vehicle A," which has a higher priority, may be made based on the situation of "vehicle A," and performs emergency response for "vehicle B," and then performs second remote control for "vehicle A" in response to the second remote request for "vehicle A," which has a higher priority, that is made subsequently. Then, after the second remote control for "vehicle A" is completed, the remote control system 1 performs the first remote control for "vehicle B," for which emergency response was performed.

[0219] Here, the emergency measure for the first remote control request from "vehicle B," which has a lower priority, may be, for example, lowering the priority of the first remote control request, but is not limited to this. The emergency measure may be any measure that allows "vehicle B," which has a lower priority, to wait while the second remote control in response to the second remote request from "vehicle A," which has a higher priority, is executed first. In other words, the emergency measure may be any measure that allows "vehicle B" to wait until the first remote control is started after the second remote control is completed or interrupted. In other words, the emergency measure may be any measure that maintains the priority of the first remote request and does not necessarily have to lower the priority.

[0220] In addition, if the second remote request of "Vehicle A," which has a higher priority, does not occur for a predetermined period of time or longer, the first remote control may be implemented in response to the first remote request of "Vehicle B" instead of emergency measures.

[0221] It should be noted that this method may be applied not only to locations where it is known from past information that remote requests are likely to occur, but also to locations with a high environmental priority (related to safety), such as tunnels or locations with strong crosswinds.

[0222] The predetermined period is, for example, a threshold value of time that is determined in advance and stored in the memory of the server device 10, but is not limited to this. A threshold value related to the distance to a location that is known from past information to be a place where remote requests are likely to occur may also be used.

[0223] When implementing emergency measures instead of remote control in response to the first remote request of "vehicle B" with a low priority, the remote control system 1 according to the present disclosure may notify the operator that emergency measures will be implemented instead of remote control in response to the remote request. Furthermore, the remote control system 1 may notify the operator of the reason for implementing emergency measures instead of remote control.

[0224] Furthermore, the remote control system 1 may allow the operator to select whether to perform remote control or emergency measures. For example, the remote control system 1 may display a selection screen that allows the operator to select whether to perform remote control or emergency measures. This selection screen may also display selection support information that assists the operator in making a selection. The selection support information may be, for example, information on the time required for remote control or the time required for the second remote request (the time required to stop the vehicle for emergency measures).

[0225] According to this configuration, for example, in a situation where a high-priority remote request is expected to be issued based on information about past remote requests, it is possible to prevent a delay in the start of a second remote control for a high-priority second remote request due to a first remote control for a first remote request with a lower priority. Furthermore, a response to a low-priority first remote request can be implemented as an emergency measure without having to wait until a high-priority second remote request is actually issued and the second remote control is completed. Therefore, according to the remote control system 1 according to the present disclosure, even when multiple remote requests are issued from multiple moving objects within a predetermined period, it is possible to prevent a decrease in safety, such as interference with surrounding traffic participants or the risk of a collision.

[0226] In the remote control system 1 according to the present disclosure, multiple operators may perform remote control. For example, when a high-priority remote request is received, the remote control system 1 may determine which of the multiple operators to interrupt the remote control of when the remote request is received, based on the timing of the remote request. As an example, the remote control system 1 may determine to interrupt the remote control of the operator who is responding to the lowest-priority remote request among the multiple operators currently performing remote control. Note that an operator making a low-priority remote request may not necessarily be in a position to interrupt the remote control. For this reason, the remote control system 1 may determine to interrupt the remote control of the operator who is responding to a high-priority remote request among the multiple operators currently performing remote control, but who is capable of interrupting the remote control, the operator whose interruption of the remote control will have a small impact on the vehicle 30, or the operator who can interrupt the remote control earliest and respond.

[0227] In this way, in the remote control system 1 according to the present disclosure, an operator monitoring the autonomously driving vehicle 30 assists the autonomous driving of the vehicle 30 by performing remote control that intervenes in the control of the vehicle 30, such as remote operation, in response to a remote request from the vehicle 30.

[0228] For example, with remote control that responds to remote requests in order of occurrence, even if another remote request occurs while responding to a remote control, each remote control can be responded to without interrupting the remote control being responded to. However, there is a risk that responses to remote requests with high priority, such as remote controls that are highly related to safety, may be delayed. In addition, it is difficult to support services that improve efficiency by changing the response order.

[0229] Furthermore, there is known a technology for remote control based on priorities that change according to the vehicle's driving environment. Remote control that responds to remote requests based on priorities prioritizes remote requests with higher priorities, making it possible to respond to remote control that is highly related to safety. Furthermore, by using priorities that take service efficiency into consideration, it is possible to respond to services that improve efficiency by changing the order of responses.

[0230] In such a situation, in remote control that takes into account the priority of multiple autonomously traveling mobile objects, it is possible to imagine a case where multiple remote control requests are issued from multiple mobile objects within a predetermined period of time. For example, it is possible to imagine a case where a new remote control request with a higher priority is issued from another mobile object while a remote control request from a certain mobile object is being responded to. Also, for example, it is possible to imagine a case where a remote control request with a lower priority is issued from another mobile object in a situation where a remote control request with a higher priority is likely to be issued from a certain mobile object.

[0231] However, with regard to remote control that takes into account the priorities of multiple autonomously traveling mobile objects, sufficient consideration has not been given to how to deal with cases in which multiple remote requests with different priorities are made within a predetermined period of time. For example, when multiple remote requests are made within a predetermined period of time, no consideration has been given to how to determine which remote request to respond to, or how to respond to one remote request when responding to another remote request.

[0232] For example, while a mobile object that has issued a remote request is being remotely controlled, a mobile object that has issued another remote request and is waiting for the start of remote control may experience a decrease in safety and convenience due to changes in the surrounding circumstances. For example, if remote control of a mobile object that is currently responding to a remote control request with a new high priority is simply interrupted to perform remote control of the other mobile object, changes in the surrounding circumstances while the remote control is interrupted may result in a decrease in safety, such as interference with surrounding traffic participants or a risk of collision. Furthermore, for example, if remote control is performed in response to a low-priority remote request from another mobile object in a situation where a high-priority remote request from a certain mobile object is likely to be received, the start of remote control of the high-priority remote request may be delayed, resulting in a decrease in safety, such as interference with surrounding traffic participants or a risk of collision. Furthermore, if the surrounding circumstances change while remote control is interrupted or while waiting for the start of remote control, it may become more difficult to perform remote control after the interruption or start, which may increase the time required to return to autonomous driving and reduce operational efficiency.

[0233] Therefore, with regard to remote control that takes into account the priority of multiple autonomously moving vehicles, there is room for improvement in terms of how to deal with cases where multiple remote requests occur within a specified period, for example from the standpoint of safety and efficiency.

[0234] In this situation, the remote control system 1 of the present disclosure is configured such that, when a first remote request of first priority and a second remote request of second priority occur within a specified period, emergency measures are taken for either the first remote request or the second remote request to lower its priority, and then remote control is performed for the other remote request.

[0235] According to the above configuration, it is possible to lower the priority of one of the remote requests by implementing emergency measures instead of remote control, and then implement one of the other remote controls. Therefore, even if multiple remote requests are made within a predetermined period of time, safe and efficient remote control can be realized while taking priority into consideration. In other words, according to the above configuration, with regard to remote control of multiple autonomously traveling mobile objects taking priority into consideration, it is possible to appropriately respond even if multiple remote requests with different priorities are made within a predetermined period of time. Therefore, according to the above configuration, safe and efficient remote control can be performed while taking priority into consideration.

[0236] For example, even when a vehicle approaches from behind, emergency measures can be implemented to improve safety, such as by not interfering with surrounding traffic participants. For example, even when a vehicle is trapped between roadside parking spaces, emergency measures can be implemented to improve the overall efficiency of remote control responses. For example, even when a remote request with a higher safety priority than the remote control being handled occurs, such as when a pedestrian passes by during remote control of roadside parking avoidance, emergency measures can be implemented to improve safety. For example, even when a remote request with a lower safety priority than the remote control being handled changes due to changes in surrounding conditions, such as when a vehicle approaches a waiting vehicle from behind during remote control of roadside parking avoidance, safety can be improved by emergency measures. For example, even when a vehicle approaches a waiting vehicle from behind and parks on the road, causing the vehicle to be trapped between roadside parking spaces, emergency measures can be implemented to improve safety even when it takes a long time to recover from the overall remote control, such as when a vehicle approaches a waiting vehicle from behind and parks on the road, causing the vehicle to be trapped between roadside parking spaces. For example, when a vehicle is being remotely controlled to avoid parking on the road and an oncoming vehicle is approaching, the remote control must be interrupted until all oncoming vehicles have passed. Even if the vehicle is being remotely controlled and the remote control must be interrupted due to changes in the surrounding situation, emergency measures can be taken to improve safety.

[0237] The remote control system 1 according to the present disclosure is also configured to determine whether the currently active first remote control can be interrupted based on the status of surrounding traffic participants. If the currently active first remote control can be interrupted, the remote control system 1 is configured to calculate the allowable interruption time and switch between remote control and emergency measures as a response based on the calculated allowable interruption time.

[0238] This configuration allows a high-priority remote request to be addressed while minimizing the impact of the interruption on an interrupted remote request. In other words, with the above configuration, with regard to a second remote control for a high-priority second remote request that is highly related to safety, it is possible to determine whether to perform the second remote control or to lower the priority of the second remote request by performing emergency measures for the second remote request, taking into account whether the currently-being-addressed first remote control can be interrupted and the allowable interruption time. Here, emergency measures are measures (responses) for a remote request that are different from remote control and that can lower the priority of the remote request more easily or in less time than remote control. In other words, with the above configuration, it is possible to appropriately determine whether to perform emergency measures or remote control depending on the allowable interruption time. Furthermore, with the above configuration, it is possible to implement appropriate emergency measures depending on the allowable interruption time. In other words, with the above configuration, it is possible to prevent a situation in which a low-priority remote request is suspended indefinitely even if the allowable interruption time is known, unless the priority of the high-priority remote request is exceeded, and it is also possible to prevent a situation in which an event that does not exceed the priority cannot be addressed during that time.

[0239] The above-described embodiment can be arbitrarily combined with at least one of the above-described modifications and application examples. Also, two or more of the above-described modifications and application examples can be arbitrarily combined.

[0240] Note that some or all of the functions of each device in the remote control system 1 according to the above-described embodiment may be realized by other devices in the remote control system 1. For example, some of the functions of the vehicle 30 according to the above-described embodiment may be realized by at least one of the server device 10 and the operator terminal 20. Alternatively, in the remote control system 1 according to the above-described embodiment, the server device 10 and the operator terminal 20 may be configured as an integrated unit. Alternatively, in the remote control system 1 according to the above-described embodiment, any one of the multiple vehicles 30 and the server device 10 may be configured as an integrated unit.

[0241] In the remote control system 1 according to the above-described embodiment, the multiple vehicles 30 monitored by one operator via one operator terminal 20 include one or more vehicles 30 for which it is possible to determine whether or not emergency measures need to be taken based on the allowable interruption time and then decide on a response based on priority, as described above. In other words, the multiple vehicles (mobile bodies) to be monitored may be a mixture of the vehicles 30 according to the above-described embodiment and vehicles for which a response based on priority is decided without determining whether or not emergency measures need to be taken based on the allowable interruption time.

[0242] In the above-described embodiment, the determination of "whether it is A or not" may be realized by determining only that it is A, or by determining only that it is not A, or by determining both of these.

[0243] In the above embodiment, "any of A" means "at least one of A."

[0244] The programs executed by each device in the remote control system 1 according to the above-described embodiment may be provided by being recorded in an installable or executable file format on a computer-readable recording medium (Computer Program Product) such as a CD-ROM, FD, CD-R, or DVD.

[0245] The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be provided or distributed via a network such as the Internet.

[0246] Furthermore, the programs executed by the devices of the remote control system 1 according to the above-described embodiment may be configured to be provided by being pre-installed in a ROM or the like.

[0247] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0248] (Additional Notes) The above embodiments disclose the following technologies. (1) An information processing method executed by an information processing device that supports an operator in remote control of a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task, including remote monitoring and remote operation by the operator based on video data transmitted from each of the plurality of mobile objects, wherein when a first remote request of a first priority and a second remote request of a second priority higher than the first priority are generated within a predetermined period of time, the information processing method performs an emergency response for either the first remote request or the second remote request, lowers the priority of one remote request through the emergency response, and then performs remote control for the other remote request. (2) The information processing method described in (1) above, when the second remote request is generated while the operator is responding to the first remote control for the first remote request, interrupts the first remote control and performs the emergency response for the second remote request, lowers the priority of the second remote request through the emergency response for the second remote request, and then resumes the first remote control. (3) The information processing method according to (1) or (2), wherein, in a situation where the second remote request is expected to be issued, if the first remote request is issued first, the first emergency response is implemented for the first remote request, and after the first emergency response is implemented to the first remote request, the first mobile body that issued the first remote request is put into a state where it can wait while remote control for the second remote request is being implemented first, and then, when the second remote request is issued, the second remote control for the second remote request is implemented. (4) The information processing method according to (2), wherein, based on information indicating the presence or absence of a place to temporarily take refuge around the first mobile body that issued the first remote request, it is determined whether the first remote control being handled can be interrupted.(5) The information processing method according to (2) or (4), further comprising: calculating an allowable interruption time based on a predetermined rule; if the allowable interruption time of the first remote control is shorter than a required time for second remote control in response to the second remote request, interrupting the first remote control and implementing the emergency response for the second remote request; if the allowable interruption time of the first remote control is longer than a required time for second remote control in response to the second remote request, interrupting the first remote control and implementing the second remote control without implementing the emergency response for the second remote request. (6) The information processing method according to (5), further comprising: if a surrounding traffic participant is approaching the first moving body, calculating, based on information indicating the status of the surrounding traffic participants, including the number of surrounding traffic participants present around the first moving body that issued the first remote request and whether or not the surrounding traffic participant is approaching the first moving body, a predicted time until the surrounding traffic participant will come within a predetermined range around the first moving body, as the allowable interruption time. (7) The information processing method according to (5) or (6), calculating a required time for the second remote control based on an estimated response time that is set in advance for each type of remote request that occurs. (8) The information processing method according to any one of (2), (4) to (7), calculating a required time for the second remote control based on information indicating a relationship between the emergency measure that is set in advance for each type of remote request that occurs and a priority after the emergency measure, if the second priority of the second remote request would be lower than the first priority of the first remote request if the emergency measure is performed on the second remote request, interrupting the first remote control and performing the emergency measure on the second remote request, and if the second priority of the second remote request would not be lower than the first priority of the first remote request if the emergency measure is performed on the second remote request, interrupting the first remote control and performing the second remote control on the second remote request without performing the emergency measure on the second remote request.(9) The information processing method described in any one of (5) to (7), wherein the time required for the emergency measure for the second remote request is calculated based on information indicating a relationship between the emergency measure and an estimated time required for the emergency measure, which is set in advance for each type of remote request generated; if the time required for the emergency measure for the second remote request is shorter than the interruption time of the first remote control, the first remote control is interrupted and the emergency measure is performed for the second remote request; and if the time required for the emergency measure for the second remote request is longer than the interruption time of the first remote control, the first remote control is interrupted and the second remote control is performed without performing the emergency measure for the second remote request. (10) An information processing device that supports an operator in remotely controlling a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task, including remote monitoring and remote operation of the plurality of mobile objects based on video data transmitted from each of the plurality of mobile objects, the information processing device comprising at least one processor configured to, when a first remote request of a first priority and a second remote request of a second priority higher than the first priority occur within a predetermined period of time, implement emergency measures for either the first remote request or the second remote request, and lower the priority of one of the remote requests by the emergency measures, and then implement remote control for the other remote request. (11) A program for a computer that realizes an information processing device that supports an operator's remote control of a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task, including remote monitoring and remote operation of the plurality of mobile objects by the operator based on video data transmitted from each of the plurality of mobile objects, the program causing the computer to execute the following: when a first remote request with a first priority and a second remote request with a second priority higher than the first priority occur within a predetermined period of time, implement emergency measures for either the first remote request or the second remote request, lower the priority of one of the remote requests by the emergency measures, and then implement remote control for the other remote request.(12) An information processing device comprising at least one processor and at least one memory, wherein the at least one processor executes the information processing method according to any one of (1) to (9) above by executing a program stored in the at least one memory. (13) A program that causes a computer to execute the information processing method according to any one of (1) to (9) above, or a computer-readable non-transitory recording medium on which the program is stored.

[0249] 1 Remote control system 10 Server device (information processing device) 101 Remote request information acquisition unit 102 Priority calculation unit 103 Operator status acquisition unit 104 Operator status determination unit 105 Priority determination unit 106 Remote request response time calculation unit 107 Interruptibility determination unit 108 Interruptibility time calculation unit 109 Remote request response determination unit 110 Emergency treatment determination unit 111 Emergency treatment required time calculation unit 112 Emergency treatment content determination unit 113 Emergency treatment information transmission unit 114 Remote request information transmission unit 115 Automatic stop control information generation unit 116 Automatic stop control information transmission unit 20 Operator terminal (information processing device) 201 Remote request information reception unit 202 Remote request information notification unit 203 Display unit 204 Emergency treatment information reception unit 205 Emergency treatment information notification unit 206 Input unit 207 Operator response information transmission unit 30 Vehicle (mobile body) 301 Sensor 302 Remote request determination unit 303 Control information acquisition unit 304 Control unit 305 Drive unit 4 Information processing device 41 Processor 42 ROM 43 RAM 44 Device I / F unit 51 Information indicating the relationship between the remote request content and priority 52 Information indicating the relationship between the operator's response status and remote control 53 Information indicating the relationship between the remote request content and the estimated response time 54 Information indicating emergency measures for approaching pedestrians 55 Information indicating emergency measures for approaching pedestrians that have a lower priority than the remote request being responded to 56 Information indicating the emergency measures content 57 Information indicating the vehicle status 601 to 615 Display screens 701a to 701d Images 711 to 745 Notification 801 Falling object (surrounding object) 803 Pedestrian (surrounding object) 805 Approaching vehicle (surrounding object) 807 Parking on the road (surrounding objects) 809 Evacuation area N Network

Claims

1. An information processing method executed by an information processing device that supports an operator's remote control of a plurality of mobile objects, each configured to move autonomously and perform a specified task, including the operator's remote monitoring and remote operation of the plurality of mobile objects based on video data transmitted from each of the plurality of mobile objects, wherein, when a first remote request of a first priority and a second remote request of a second priority higher than the first priority occur within a specified period of time, the information processing method implements emergency measures for either the first remote request or the second remote request, and after lowering the priority of one of the remote requests by the emergency measures, implements remote control for the other remote request.

2. The information processing method of claim 1, wherein, if the second remote request occurs while the operator is responding to the first remote control in response to the first remote request, the first remote control is interrupted and the emergency response is performed on the second remote request, and after the priority of the second remote request is lowered by the emergency response on the second remote request, the first remote control is resumed.

3. The information processing method of claim 1, wherein, in a situation where the second remote request is expected to occur, if the first remote request occurs first, the first aid is carried out for the first remote request, and the first mobile body that issued the first remote request is placed in a state where it can wait while remote control for the second remote request is carried out first by the first aid for the first remote request, and then, when the second remote request occurs, the second remote control is carried out for the second remote request.

4. An information processing method as described in claim 2, wherein a determination is made as to whether the first remote control being handled can be interrupted based on information indicating whether there is a place to temporarily take refuge around the first mobile body that issued the first remote request.

5. The information processing method of claim 2, further comprising: calculating an interruption time based on a predetermined rule; interrupting the first remote control and implementing the emergency measures for the second remote request when the interruption time of the first remote control is shorter than the time required for the second remote control in response to the second remote request; and interrupting the first remote control and implementing the second remote control without implementing the emergency measures for the second remote request when the interruption time of the first remote control is longer than the time required for the second remote control in response to the second remote request.

6. The information processing method described in claim 5, wherein, based on information indicating the status of surrounding traffic participants, including the number of surrounding traffic participants present around the first moving body that issued the first remote request and whether or not any of the surrounding traffic participants are approaching the first moving body, if any surrounding traffic participants are approaching the first moving body, the predicted time until they approach within a predetermined range around the first moving body is calculated as the interruption time.

7. The information processing method according to claim 5, further comprising calculating the time required for the second remote control based on a predetermined estimated response time for each type of remote control request that occurs.

8. The information processing method of claim 2, further comprising: based on information indicating the relationship between the emergency measures set in advance for each type of remote request that occurs and the priority after the emergency measures are taken, if the second priority of the second remote request would be lower than the first priority of the first remote request if the emergency measures are taken for the second remote request, interrupting the first remote control and taking the emergency measures for the second remote request; and if the second priority of the second remote request would not be lower than the first priority of the first remote request if the emergency measures are taken for the second remote request, interrupting the first remote control and taking the second remote control for the second remote request without taking the emergency measures for the second remote request.

9. The information processing method of claim 5, further comprising: calculating a required time for the emergency measure for the second remote request based on information indicating a relationship between the emergency measure and an estimated required time for the emergency measure, the relationship being preset for each type of remote request generated; if the required time for the emergency measure for the second remote request is shorter than the allowable interruption time for the first remote control, interrupting the first remote control and implementing the emergency measure for the second remote request; and if the required time for the emergency measure for the second remote request is longer than the allowable interruption time for the first remote control, interrupting the first remote control and implementing the second remote control without implementing the emergency measure for the second remote request.

10. An information processing device that supports an operator in remotely controlling a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task, including remote monitoring and remote operation of the plurality of mobile objects based on video data transmitted from each of the plurality of mobile objects, the information processing device comprising at least one processor configured to, when a first remote request of a first priority and a second remote request of a second priority higher than the first priority occur within a predetermined period of time, implement emergency measures for either the first remote request or the second remote request, and after lowering the priority of one of the remote requests by the emergency measures, implement remote control for the other remote request.

11. A program for a computer that implements an information processing device that supports an operator's remote control of a plurality of mobile objects, each configured to move autonomously and perform a specified task, including remote monitoring and remote operation of the plurality of mobile objects by the operator based on video data transmitted from each of the plurality of mobile objects, the program causing the computer to execute the following: when a first remote request of a first priority and a second remote request of a second priority higher than the first priority occur within a specified period of time, implement emergency measures for either the first remote request or the second remote request, lower the priority of one of the remote requests by the emergency measures, and then implement remote control for the other remote request.

Images