Information processing method, terminal, and program
The information processing method in remote control systems facilitates efficient and safe operator replacement by managing responsibility and authority transfers in remote control systems for multiple moving objects, addressing inefficiencies and legal concerns by ensuring clear accountability during operator changes.
Patent Information
- Application Number
- PCT/JP2024/010057
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-03-14
- Publication Date
- 2024-11-14
AI Technical Summary
Existing remote control systems for multiple moving objects do not effectively manage operator changes, leading to inefficiencies and unclear responsibility when replacing operators, particularly in scenarios where air traffic control is involved, resulting in potential legal and safety issues.
An information processing method and system that allows for the seamless transfer of responsibility and operating authority between operators while ensuring clear accountability, even during autonomous travel of mobile objects, by acquiring and approving transfer requests and independently managing responsibility and authority transfer timing.
Enables efficient and safe operator replacement with clear responsibility transfer, reducing time loss and ensuring legal compliance by explicitly managing and displaying responsibility and authority changes, thus enhancing the operational efficiency and safety of remote control systems.
Smart Images

Figure JP2024010057_14112024_PF_FP_ABST
Abstract
Description
Information processing method, terminal and program
[0001] The present disclosure relates to an information processing method, a terminal, and a program.
[0002] Patent Document 1 discloses a system for remotely controlling a plurality of moving objects by a plurality of operators.
[0003] Patent No. 6980609
[0004] However, there are cases where the operator remotely controlling a moving object needs to be changed while the moving object is moving. In such cases, it is desirable that the operator change be performed appropriately. However, Patent Document 1 does not take into consideration the operator change.
[0005] Therefore, the present disclosure provides an information processing method, a terminal, and a program that enable appropriate operator changeover.
[0006] An information processing method according to one aspect of the present disclosure is an information processing method executed by an information processing system for one or more operators including a first operator to remotely control one or more mobile bodies including a first mobile body, which obtains a transfer request to transfer responsibility associated with the remote control of the first mobile body to the first operator or an autonomous mobile system of the first mobile body, obtains information from the first operator or the autonomous mobile system regarding whether or not to accept the obtained transfer request, and if information approving the transfer request is obtained, transfers the responsibility for the first mobile body to the first operator or the autonomous mobile system.
[0007] A terminal according to one aspect of the present disclosure is a terminal for one or more operators including a first operator to remotely control one or more mobile bodies including a first mobile body, and includes a first acquisition unit that acquires a transfer request to transfer responsibility associated with the remote control of the first mobile body to the first operator, a second acquisition unit that acquires an operation from the first operator indicating whether or not to accept the acquired transfer request, and a processing unit that, when a first operation accepting the transfer request is acquired from the first operator, transfers the responsibility for the first mobile body and the operating authority of the first mobile body to the first operator.
[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method.
[0009] According to one aspect of the present disclosure, it is possible to realize an information processing method or the like that allows appropriate operator changeover.
[0010] FIG. 1 is a diagram showing a schematic configuration of a remote control system according to an embodiment. FIG. 2 is a block diagram showing a functional configuration of a terminal according to an embodiment. FIG. 3 is a block diagram showing a functional configuration of a control server according to an embodiment. FIG. 4 is a diagram showing information stored in a storage device according to an embodiment. FIG. 5 is a block diagram showing a functional configuration of a moving object according to an embodiment. FIG. 6 is a diagram showing a form of remote control according to an embodiment. FIG. 7 is a flowchart showing operations of a side requesting a transfer in the remote control system according to an embodiment. FIG. 8 is a diagram showing a screen displayed on a display unit of the side requesting a transfer according to an embodiment. FIG. 9 is a flowchart showing operations of a side receiving a transfer request in the remote control system according to an embodiment. FIG. 10 is a diagram showing a screen displayed on a display unit of the side receiving a transfer request in the remote control system according to an embodiment. FIG. 11 is a flowchart showing operations of a side requesting a transfer in the remote control system according to a first modification of the embodiment. FIG. 12 is a flowchart showing operations of a side requesting assistance in the remote control system according to a second modification of the embodiment. FIG. 13 is a diagram showing a screen displayed on a display unit of the side requesting assistance in the remote control system according to the second modification of the embodiment. FIG. 14 is a flowchart showing operations of a side receiving an assistance request in the remote control system according to the second modification of the embodiment. FIG. 15 is a diagram showing a screen displayed on a display unit of a side receiving an assistance request in a remote control system according to a second modification of the embodiment. FIG. 16 is a flowchart showing the operation of a side requesting assistance in a remote control system according to a third modification of the embodiment. FIG. 17 is a flowchart showing the operation of a side requesting transfer in a remote control system according to a fourth modification of the embodiment. FIG. 18 is a diagram showing a first screen displayed on a display unit of a side requesting transfer in a remote control system according to the fourth modification of the embodiment. FIG. 19 is a diagram showing a second screen displayed on a display unit of a side requesting transfer in a remote control system according to the fourth modification of the embodiment. FIG. 20 is a flowchart showing the operation of a side receiving a transfer request in a remote control system according to the fourth modification of the embodiment. FIG. 21 is a diagram showing a screen displayed on a display unit of a side receiving a transfer request in a remote control system according to the fourth modification of the embodiment.FIG. 22 is a flowchart showing the operation of a side receiving an assistance request in a remote control system according to a fifth variation of the embodiment. FIG. 23 is a diagram showing driving task data according to the fifth variation of the embodiment. FIG. 24 is a diagram showing manual data according to the fifth variation of the embodiment. FIG. 25 is a diagram showing a screen displayed on a display unit of a side receiving an assistance request in a remote control system according to the fifth variation of the embodiment. FIG. 26 is a flowchart showing the operation of a side requesting assistance in a remote control system according to a sixth variation of the embodiment. FIG. 27 is a diagram showing moving object information according to the sixth variation of the embodiment. FIG. 28 is a diagram showing surrounding environment information according to the sixth variation of the embodiment. FIG. 29 is a diagram showing manual data according to the sixth variation of the embodiment. FIG. 30 is a flowchart showing the operation of a control server in a remote control system according to a seventh variation of the embodiment. FIG. 31 is a diagram showing driving task data according to the seventh variation of the embodiment. FIG. 32 is a diagram showing operator skill information according to the seventh variation of the embodiment. FIG. 33 is a diagram showing operator skill level information according to the seventh variation of the embodiment. FIG. 34 is a diagram showing required skill information according to the seventh variation of the embodiment. FIG. 35 is a flowchart showing the operation of the autonomous driving system in the remote control system according to Modification 8 of the embodiment. FIG. 36 is a diagram showing autonomous driving ODD information according to Modification 8 of the embodiment. FIG. 37 is a flowchart showing the operation of a side receiving an assistance request in the remote control system according to Modification 8 of the embodiment. FIG. 38 is a diagram showing a screen displayed on a display unit on the side receiving an assistance request in the remote control system according to Modification 8 of the embodiment. FIG. 39 is a flowchart showing the operation of a side requesting responsibility transfer in the remote control system according to Modification 9 of the embodiment. FIG. 40 is a diagram showing a screen displayed on a display unit on the side requesting responsibility transfer in the remote control system according to Modification 9 of the embodiment. FIG. 41 is a flowchart showing the operation of the autonomous driving system side in the remote control system according to Modification 9 of the embodiment. FIG. 42 is a sequence diagram showing a first example of the operation when the remote control system according to Modification 10 of the embodiment is unable to fulfill control responsibility.FIG. 43 is a sequence diagram showing a second example of the operation when the remote control system according to the tenth modification of the embodiment is unable to fulfill its control responsibility. FIG. 44 is a sequence diagram showing a third example of the operation when the remote control system according to the tenth modification of the embodiment is unable to fulfill its control responsibility. FIG. 45 is a flowchart showing the operation of the remote control system according to the eleventh modification of the embodiment. FIG. 46 is a diagram showing a screen displayed on the display unit of the administrator in the remote control system according to the eleventh modification of the embodiment. FIG. 47 is a flowchart showing the operation of the side requesting responsibility transfer in the remote control system according to the eleventh modification of the embodiment. FIG. 48 is a diagram showing a screen displayed on the display unit of the side requesting responsibility transfer in the remote control system according to the eleventh modification of the embodiment. FIG. 49 is a flowchart showing the operation of the side receiving the responsibility transfer request in the remote control system according to the eleventh modification of the embodiment. FIG. 50 is a diagram showing a screen displayed on the display unit of the side receiving the responsibility transfer request in the remote control system according to the eleventh modification of the embodiment.
[0011] (Background to the present disclosure) There are multiple control modes (see FIG. 6 described below) for remote control of several hundred mobile objects (e.g., autonomous mobile objects) shared among multiple operators, and the operators have driving responsibilities depending on the control mode.
[0012] Furthermore, the operator performing remote control will bear responsibility (control responsibility) associated with the remote control. Therefore, when changing the operator remotely controlling a mobile object, it is desirable to clarify who has control responsibility. In other words, when changing the operator, it is desirable that the change of operator be carried out appropriately from the perspective of clarifying who has control responsibility. Control responsibility is the responsibility associated with the remote assistance or remote operation of a mobile object, and is the responsibility that results in legal disadvantages or sanctions if an accident or other incident occurs while the mobile object is traveling. In the following, responsibility associated with remote control will also be referred to simply as responsibility or control responsibility.
[0013] Furthermore, when an operator has control responsibilities (e.g., driving responsibilities), it is assumed that the mobile object will be stopped and then handed over to another operator, but this is inefficient. Therefore, an information processing method that allows for operator change while the mobile object is autonomously traveling, for example, is desired. In other words, when changing operators, it is desirable to execute the operator change appropriately from the viewpoint of improving the efficiency of the operator change. This allows for seamless transfer of control responsibilities while the mobile object is traveling, thereby reducing the time lost due to the operator change.
[0014] The above-mentioned Patent Document 1 does not disclose a technology for appropriately executing such an operator change. Therefore, the inventors of the present application have conducted extensive research into an information processing method etc. that can appropriately execute an operator change, and have devised the information processing method etc. described below.
[0015] An information processing method according to a first aspect of the present disclosure is an information processing method executed by an information processing system for one or more operators including a first operator to remotely control one or more mobile bodies including a first mobile body, which obtains a transfer request to transfer responsibility associated with the remote control of the first mobile body to the first operator or an autonomous mobile system of the first mobile body, obtains information from the first operator or the autonomous mobile system regarding whether or not to accept the obtained transfer request, and if information approving the transfer request is obtained, transfers the responsibility for the first mobile body to the first operator or the autonomous mobile system.
[0016] As a result, if the first operator or the autonomous mobile system accepts the transfer request, the transfer is actually carried out. In other words, the first operator or the autonomous mobile system can be more reliably transferred responsibility. Therefore, since the location of responsibility for the mobile object when an operator is changed becomes clear, the change of operator, etc. can be carried out appropriately.
[0017] Also, for example, the information processing method according to the second aspect may be an information processing method according to the first aspect, and when information approving the transfer request is obtained, the operating authority of the first moving body may be transferred to the first operator or the autonomous mobile system independently of the responsibility of the first moving body.
[0018] This allows the operation authority of the first moving object to be transferred to the first operator or the autonomous mobile system independently of the responsibility, which clarifies the location of the responsibility and operation authority of the moving object when the operator is changed, and the change of the operator can be carried out more appropriately.
[0019] Also, for example, the information processing method according to the third aspect may be an information processing method according to the second aspect, in which the responsibility for the first moving body and the operating authority of the first moving body are transferred to the first operator or the autonomous mobile system at different times.
[0020] This allows for appropriate handover of operators, etc., even if responsibility and operating authority for the first moving body are transferred at different times.
[0021] Also, for example, the information processing method according to the fourth aspect may be an information processing method according to the third aspect, in which the responsibility of the first moving body is transferred to the first operator or the autonomous mobile system at an earlier timing than the operating authority of the first moving body.
[0022] This allows for a proper change of operator, etc., even if responsibility for the first moving body is transferred before the operating authority is transferred.
[0023] Also, for example, the information processing method according to the fifth aspect may be an information processing method according to the third aspect, in which the operating authority of the first moving body is transferred to the first operator or the autonomous mobile system at a timing earlier than the responsibility of the first moving body.
[0024] This allows for a proper change of operator, etc., even when the operating authority of the first moving body is transferred before the responsibility is transferred.
[0025] Furthermore, for example, the information processing method according to the sixth aspect may be an information processing method according to any one of the second to fifth aspects, and may present to the first operator in different ways that the responsibility for the first mobile body has been transferred to the first operator and that the operating authority for the first mobile body has been transferred to the first operator.
[0026] As a result, the fact that responsibility has been transferred and the fact that operating authority has been transferred are displayed in different ways, making it easy to check the transfer status.
[0027] Also, for example, an information processing method according to a seventh aspect may be an information processing method according to any one of the second to sixth aspects, wherein the transfer request is a request to transfer part of the responsibility of the first moving body, and when an operation accepting the transfer request is obtained, part of the responsibility of the first moving body and part of the operating authority of the first moving body may be transferred to the first operator or the autonomous mobile system.
[0028] This allows a single moving object to be remotely controlled by multiple operators. Some moving objects are special in that they cannot be remotely controlled without multiple operators. The information processing method of the present disclosure can also be applied to such special moving objects.
[0029] Furthermore, for example, an information processing method according to an eighth aspect may be an information processing method according to any one of the first to seventh aspects, and when the transfer request is acquired, the details of the remote control of the first moving body may be presented, and after the details are presented, an operation to decide whether or not to accept the transfer request may be acquired from the first operator.
[0030] This allows the first operator to check the reference information for determining whether to approve the transfer request before making a decision, thereby supporting the first operator's decision.
[0031] Furthermore, for example, an information processing method according to a ninth aspect may be an information processing method according to any one of the first to eighth aspects, which acquires skill information indicating the skills in the remote control of each of the one or more operators, determines whether the first operator has the skills required for the remote control of the first mobile body based on the acquired skill information, and, if it is determined that the first operator has the skills required for the remote control of the first mobile body, acquires a transfer request to transfer the responsibility of the first mobile body to the first operator.
[0032] As a result, if the first operator has the skills appropriate for the first mobile body, a transfer request is made to the first operator, so that remote control can be carried out appropriately after the operator is changed.
[0033] Also, for example, an information processing method according to a tenth aspect is an information processing method according to any one of the first to ninth aspects, wherein the one or more operators include a second operator remotely controlling the first moving object, an operation indicating that the responsibility for the first moving object is to be transferred is obtained from the second operator, the request for transfer to the first operator is notified by the operation being obtained, and information indicating that the request for transfer of the responsibility for the first moving object has been made is presented to the second operator.
[0034] This allows the transfer of responsibility from the second operator to the first operator to be carried out appropriately.
[0035] Also, for example, an information processing method according to an eleventh aspect is an information processing method according to any one of the first to tenth aspects, and the transfer request to the first operator may be notified from the autonomous mobile system.
[0036] This allows the transfer of responsibility from the autonomous driving system to the first operator to be carried out appropriately.
[0037] Furthermore, for example, an information processing method according to a twelfth aspect may be an information processing method according to any one of the second to eleventh aspects, in which the transfer request is sent to terminals of two or more operators including the first operator, and if the first operator among the two or more operators performs an operation to accept the transfer request first, the responsibility for the first mobile body and the operating authority for the first mobile body may be transferred to the first operator.
[0038] This allows the responsibility to be transferred to the first operator in a shorter time, thereby further reducing the time lost due to the change of operator.
[0039] Furthermore, for example, an information processing method according to a thirteenth aspect may be an information processing method according to any one of the second to twelfth aspects, wherein the one or more mobile bodies include a second mobile body, and the one or more operators include a third operator who remotely controls the second mobile body, and the information processing method may acquire a third operation from the first operator requesting the transfer of the responsibility associated with the remote control of the second mobile body, and based on the acquired third operation, send a transfer request to the terminal of the third operator requesting the transfer of the responsibility for the second mobile body, and when a fourth operation indicating that the third operator has accepted the transfer request is acquired, the responsibility for the second mobile body and the operating authority for the second mobile body may be transferred from the third operator to the first operator.
[0040] This allows for proper operator handover even when the first operator takes over responsibility, and therefore allows proper handover from one operator to another.
[0041] Furthermore, for example, an information processing method according to a fourteenth aspect may be an information processing method according to any one of the second to thirteenth aspects, wherein the one or more moving bodies include a third moving body that is not remotely controlled by the one or more operators, and the information processing method may present information for remotely controlling the third moving body to the first operator, acquire a fifth operation from the first operator indicating that remote control of the third moving body will begin, and transfer the responsibility for the third moving body and the operating authority for the third moving body to the first operator based on the acquired fifth operation.
[0042] This allows the responsibility for the third mobile object that is not remotely controlled to be appropriately transferred to the first operator.
[0043] Furthermore, for example, an information processing method according to a fifteenth aspect is an information processing method according to any one of the first to fourteenth aspects, wherein the one or more moving bodies include a fourth moving body, and the information processing method presents to the first operator an operation unit for accepting an operation indicating that the remote control of the fourth moving body is to be started, and the operation unit may become effective when the remote control skills possessed by the first operator satisfy the remote control skills required to continue the autonomous movement of the fourth moving body.
[0044] This allows the fourth moving object to be remotely controlled by the first operator who has the necessary skills, and therefore operator change can be appropriately performed from the viewpoint of switching to an operator who has the necessary skills.
[0045] Furthermore, for example, an information processing method according to the 16th aspect is an information processing method according to any one of the 2nd to 15th aspects, and when the responsibility for the first moving body and the operating authority for the first moving body are transferred to the first operator, other operators among the one or more operators, excluding the first operator, may not be permitted to bear the responsibility for the first moving body unless given permission by the first operator.
[0046] This can prevent responsibility for the first mobile object from being transferred against the will of the first operator currently bearing that responsibility.
[0047] Furthermore, for example, an information processing method according to a seventeenth aspect may be an information processing method according to any one of the first to sixteenth aspects, and when an operation to reject the transfer request is acquired from the first operator, the information processing method may cause the first moving body to execute MRM (Minimal Risk Maneuver) to evacuate the first moving body.
[0048] This allows the first moving body to be safely stopped if the transfer request is rejected.
[0049] Furthermore, for example, an information processing method according to an 18th aspect is an information processing method according to any one of the 2nd to 17th aspects, and after the responsibility for the first mobile body and the operating authority of the first mobile body are transferred to the first operator, if an abnormality is detected in communication between the terminal owned by the first operator and the first mobile body, or an abnormality in a system including the terminal and the first mobile body, MRM may be executed on the first mobile body.
[0050] This allows the first moving body to be safely stopped in the event of a communication or system abnormality.
[0051] A terminal according to a 19th aspect of the present disclosure is a terminal for one or more operators including a first operator to remotely control one or more moving bodies including a first moving body, the terminal including: a first acquisition unit that acquires a transfer request to transfer responsibility for remote control of the first moving body to the first operator or an autonomous mobile system of the first moving body, a second acquisition unit that acquires information on whether to accept the acquired transfer request from the first operator or the autonomous mobile system, and a processing unit that transfers the responsibility for the first moving body to the first operator or the autonomous mobile system when information accepting the transfer request is acquired from the first operator or the autonomous mobile system. A program according to a 20th aspect of the present disclosure is a program for causing a computer to execute the information processing method according to the 1st to 18th aspects.
[0052] This provides the same effect as the above-described information processing method.
[0053] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.
[0054] Hereinafter, the embodiments will be specifically described with reference to the drawings.
[0055] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.
[0056] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.
[0057] Furthermore, in this specification, numerical values and numerical ranges are not expressions that express only the strict meaning, but are expressions that mean that they also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).
[0058] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.
[0059] (Embodiment) Hereinafter, a remote control system according to the present embodiment will be described with reference to Figs.
[0060] [1. Configuration of the Remote Control System] First, the configuration of a remote control system according to this embodiment will be described with reference to Figures 1 to 6. Figure 1 is a diagram showing the schematic configuration of a remote control system 1 according to this embodiment.
[0061] The remote control system operates in a remote control center or server device where one or more operators 50 are located, and is a system in which each operator 50 remotely monitors, and in some cases remotely assists or remotely controls, one or more mobile objects 40. Each operator 50 uses a terminal 10 such as a personal computer to monitor the status of each mobile object 40 displayed on a display unit 110a (see FIG. 8, which will be described later, etc.).
[0062] Furthermore, when a request for remote assistance or remote control is received from the mobile body 40 or when it is determined that the mobile body 40 is in a state where remote assistance or remote control is required, each operator 50 performs remote assistance or remote control of the mobile body 40. Furthermore, each operator 50 can hand over the remote control of the mobile body 40 to another operator 50 as needed. In the remote control system 1, when handing over the remote control of the mobile body 40 to another operator 50, a process is executed to ensure that the responsibility (control responsibility) associated with the remote control is reliably taken over by the other operator 50.
[0063] Hereinafter, remote assistance or remote operation will also be referred to as assistance / operation or remote support.
[0064] 1, the remote control system 1 includes a terminal 10, a control server 20, and a storage device 30. The remote control system 1 may further include one or more mobile objects 40. The control server 20 and the mobile objects 40 are communicatively connected via an external network (e.g., the Internet). The remote control system 1 is an information processing system that allows one or more operators to remotely control one or more mobile objects 40.
[0065] The terminal 10 is an operation terminal for operation by an operator 50, and is provided for each operator 50, for example. Fig. 2 is a block diagram showing the functional configuration of the terminal 10 according to this embodiment. The terminal 10 has a CPU (Central Processing Unit), a memory, etc., and the CPU executes a program stored in the memory to realize each function of the terminal 10.
[0066] 2, the terminal 10 includes an output device 110 that presents information to the operator 50, and an input device 210 that accepts input (operations) from the operator 50. The terminal 10 also includes a display unit 110a that includes a display device such as a liquid crystal display device and displays information to the operator 50. The terminal 10 may also include a component that presents information by voice or the like.
[0067] The output device 110 has a control status presentation unit 111, a transfer request presentation unit 112, a transfer request presentation unit 113, a control requirement presentation unit 114, a mobile object information presentation unit 115, a surrounding environment information presentation unit 116, a terminal communication information acquisition unit 117, an MRM (Minimal Risk Maneuver) instruction unit 118, a remote response request presentation unit 119, and a terminal monitoring unit 120.
[0068] The control status presentation unit 111 executes processing for presenting the status of remote control in the remote control system 1. For example, based on information managed by a control status management unit 121 (see FIG. 3 described later) of the control server 20, the control status presentation unit 111 presents information indicating which operator 50 is remotely controlling (monitoring / assisting / operating) which mobile object 40 to the operator 50 via the display unit 110a.
[0069] The transfer request presentation unit 112 executes processing for presenting information related to a request for transfer of the control responsibility when notified by the transfer request notification unit 122 (see FIG. 3 , which will be described later) of the control server 20. The transfer request presentation unit 112 presents information indicating the transfer request to the operator 50 via the display unit 110 a, for example.
[0070] The transfer request presentation unit 113 executes processing for presenting information regarding a request for transfer of control responsibility when notified by a transfer request notification unit 124 (see FIG. 3 described later) of the control server 20. The transfer request presentation unit 113 presents information indicating the transfer request to the operator 50 via the display unit 110 a, for example.
[0071] The control requirement presentation unit 114 executes processing for presenting information related to control requirements based on information from the instruction content determination unit 126 (see FIG. 3 described later) of the control server 20. The control requirement presentation unit 114 presents, for example, information indicating the control requirements to the operator 50 via the display unit 110a. The control requirements include at least one of the following: what the operator 50 should check when assisting / operating when taking over remote control, the operation content, information indicating the required license (skill), and the like.
[0072] The mobile object information presentation unit 115 executes processing for presenting information about the mobile object 40 that is the target of remote control, based on information sent from the mobile object information transmission unit 127 (see FIG. 3 described later) of the control server 20. The mobile object information presentation unit 115 presents information about the status of the mobile object 40, such as the vehicle speed, angular velocity, and system status of each mobile object 40, to the operator 50 via the display unit 110a.
[0073] The surrounding environment information presentation unit 116 executes processing for presenting surrounding environment information indicating the current surrounding environment of the mobile object 40, based on information about the surroundings of the mobile object 40 sent from a surrounding environment information transmission unit 128 (see FIG. 3 , which will be described later) of the control server 20. The surrounding environment information presentation unit 116 presents the surrounding environment information to the operator 50 via, for example, the display unit 110a. The surrounding environment information includes at least one of sensing information from a camera, LiDAR, radar, or the like mounted on the mobile object 40, targets obtained by processing sensing information through machine learning, signal processing, or the like, a grid map, and the like.
[0074] The terminal communication information acquisition unit 117 executes processing to acquire terminal communication information related to communication between the terminal 10 and the control server 20. The terminal communication information acquisition unit 117 acquires communication information with the control server 20, mobile object information sent from the control server 20, end-to-end delay information of surrounding environment information, and the like. The terminal communication information acquisition unit 117 is configured to include, for example, a communication circuit (communication module). Furthermore, the method for calculating the end-to-end delay information is not particularly limited, and the end-to-end delay information may be calculated based on timestamps attached to the mobile object information and surrounding environment information.
[0075] The MRM instruction unit 118 transmits an MRM instruction signal as an operation signal to the control server 20 or the mobile body 40 when, for example, the communication state between the terminal 10 and the mobile body 40 becomes a predetermined state. The predetermined state may be, for example, when communication in the terminal communication information acquisition unit 117 is disconnected, or when a delay (for example, a delay of a predetermined length or more) is observed in the mobile body information or the surrounding environment information. The MRM instruction unit 118 is configured to include, for example, a communication circuit (communication module). Furthermore, the MRM instruction unit 118 presents a display for accepting an operation to execute MRM to the operator 50 via the display unit 110a.
[0076] When notified by the remote support request notification unit 132 (see FIG. 3 described later) of the control server 20, the remote support request presentation unit 119 executes a process of presenting information for notifying the operator 50 that the mobile object 40 requires remote assistance / operation. The remote support request presentation unit 119 presents the information to the operator 50 via the display unit 110a, for example.
[0077] The terminal monitoring unit 120 monitors whether or not there is an abnormality in the terminal 10. For example, the terminal monitoring unit 120 monitors whether or not the time in the terminal 10 is being changed discontinuously. For example, the terminal monitoring unit 120 monitors whether or not a time jump has occurred due to a leap second or the like.
[0078] The input device 210 has a transfer request response input unit 211, a transfer request input unit 212, a transfer request response input unit 213, a transfer request input unit 214, a start / end input unit 215, an operation signal transmission unit 216, a moving object selection unit 217, an input device status acquisition unit 218, an MRM instruction unit 219, and a call unit 220. The input device 210 is configured to include a reception device that receives operations from the operator 50, such as a touch panel or buttons. The input device 210 may also be configured to receive operations by voice, gestures, or the like.
[0079] The transfer request response input unit 211 executes processing for receiving input from the operator 50 regarding whether or not to request the transfer of control responsibility. For example, when a notification is received from the transfer request notification unit 122, the transfer request response input unit 211 displays an indication (icon) on the display unit 110a for inputting whether or not to accept the notification, acquires input from the operator 50 via a touch panel or the like, and notifies the acquired result to the control status management unit 121. The result here includes, for example, whether the transfer request has been approved or rejected.
[0080] When the operator 50 has control responsibility, the transfer request input unit 212 executes processing for transferring the control responsibility to another operator 50 or the like. When the operator 50 has control responsibility, the transfer request input unit 212, for example, displays a message for transferring the control responsibility to another operator 50 or the like on the display unit 110a, acquires input from the operator 50 via a touch panel or the like, and notifies the acquired result to the transfer request notification unit 122. The result here includes, for example, that the operator 50 has performed an operation to transfer the control responsibility that has fallen on the operator 50 to the other operator 50.
[0081] The transfer request input unit 212 may notify the transfer request notification unit 122 of the result indicating whether or not to transfer the vehicle, regardless of an operation from the operator 50. The transfer request input unit 212 may automatically send a notification indicating that the vehicle will be transferred, depending on the status of at least one of the operator 50, the remote control system 1, and the mobile object 40, for example.
[0082] The transfer request response input unit 213 executes processing for receiving an input from the operator 50 as to whether or not to accept the request to transfer the control responsibility. For example, when a notification is received from the transfer request notification unit 124, the transfer request response input unit 213 displays a display on the display unit 110a for inputting whether or not to accept the request, acquires input from the operator 50 via a touch panel or the like, and notifies the acquired result to the control status management unit 121. The result here includes, for example, an approval of the transfer request or a denial of the transfer request.
[0083] The transfer request input unit 214 executes processing for receiving an input from the operator 50 as to whether or not to accept the control responsibility held by another operator 50. For example, when the transfer request input unit 214 receives an input from the operator 50 to accept the control responsibility of the mobile object 40 held by the other operator 50, the transfer request input unit 214 notifies the transfer request notification unit 124 of the result.
[0084] The transfer request input unit 214 may notify the transfer request notification unit 124 of information indicating whether or not to transfer the control signal, without an operation from the operator 50. The transfer request input unit 214 may automatically notify the transfer request notification unit 124 of information indicating whether or not to transfer the control signal, depending on the status of at least one of the operator 50, the remote control system 1, and the mobile object 40, for example.
[0085] The start / end input unit 215 executes processing for notifying the control status management unit 121 of information indicating the start or end of assistance / operation of the mobile body 40. The start / end input unit 215 displays a display on the display unit 110a for allowing the operator 50 to input the start or end of assistance / operation of the mobile body 40, acquires input from the operator 50 via a touch panel or the like, and notifies the acquired result to the control status management unit 121.
[0086] The start / end input unit 215 may determine whether a start or end condition is met based on at least one of ODD (Operational Design Domain) information and operator qualification information, and may transition to an inactive state if it is determined that the condition is not met. The inactive state is a state in which a start or end operation from the operator 50 is not accepted. For example, a case in which the mobile object 40 is traveling in an environment in which it cannot travel autonomously alone and assistance / operation cannot be ended is exemplified. Note that traveling is an example of movement.
[0087] The operation signal transmission unit 216 is a processing unit that transmits operation signals for controlling the movement of the mobile body 40. The operation signal transmission unit 216 transmits operation signals such as remote operation information from an operating device (input device) such as a steering wheel, an instruction to start / end autonomous traveling, an MRM instruction, an instruction to switch / cancel remote operation, an instruction to clear an error, etc. to the operation signal transmission unit 129 of the control server 20 or the mobile body 40.
[0088] The moving object selection unit 217 accepts a selection of a moving object 40 to be remotely controlled from the operator 50. For example, the moving object selection unit 217 accepts the selection when the operator 50 wants to control a specific moving object 40. The moving object selection unit 217 may be realized by, for example, a button.
[0089] The input device status acquisition unit 218 acquires the status of an operating device such as a steering wheel (for example, the input device 60 shown in FIG. 43 ). The status here includes a status related to a malfunction such as a breakdown. The operating device and the terminal 10 are connected so as to be able to communicate with each other.
[0090] The MRM instruction unit 219 executes a process of transmitting an MRM instruction signal as an operation signal to the mobile object 40. For example, when an operating device such as a steering wheel is abnormal, the MRM instruction unit 219 transmits the MRM instruction signal as an operation signal to the mobile object 40.
[0091] The communication unit 220 is a processing unit that realizes a function of communicating between the operators 50. The communication unit 220 enables communication between the operators 50 when, for example, a plurality of people simultaneously control the same mobile object 40.
[0092] Referring again to Fig. 1, the control server 20 is an information processing device that allows the operator 50 to execute processing for remotely controlling the mobile object 40. The control server 20 may be set, for example, in a remote location different from the building in which the terminal 10 is installed. The control server 20 is connected to, for example, a plurality of terminals 10 so as to be able to communicate with them. Fig. 3 is a block diagram showing the functional configuration of the control server 20 according to this embodiment.
[0093] 3 , the control server 20 includes a control status management unit 121, a transfer request notification unit 122, a transfer request recipient determination unit 123, a transfer request notification unit 124, a transfer acceptance determination unit 125, an instruction content determination unit 126, a mobile object information transmission unit 127, a surrounding environment information transmission unit 128, an operation signal transmission unit 129, a communication information acquisition unit 130, an MRM instruction unit 131, a remote response request notification unit 132, and an autonomous driving continuation acceptance determination unit 133. The control server 20 includes a CPU, a memory, etc., and each function of the control server 20 is realized by the CPU executing a program stored in the memory.
[0094] The control status management unit 121 manages the control status of each of the multiple operators 50. The control status management unit 121 manages which operator 50 is monitoring / assisting / operating which mobile object 40.
[0095] The transfer request notification unit 122 receives the notification from the transfer request input unit 212 and notifies the operator 50 of a request to transfer the control responsibility.
[0096] The transfer request recipient determination unit 123 executes a process of narrowing down the operators 50 that will request the transfer of control responsibility from among the multiple operators 50, based on at least one of the information stored in the storage device 30, the mobile body information (see Figure 27 described later), and the surrounding environment information of the mobile body 40 (see Figure 28 described later).
[0097] The transfer request notifying unit 124 receives the notification from the transfer request input unit 214 and executes a process of notifying the terminal 10 of a request to transfer the control responsibility.
[0098] The transfer possibility determination unit 125 determines whether the operator 50 is able to request the transfer of control responsibility based on at least one of the information stored in the storage device 30, the mobile body information, and the surrounding environment information of the mobile body 40.
[0099] The instruction content determination unit 126 determines the content of monitoring / operation to be performed by the operator 50 based on information stored in the traveling task information DB 32 and the manual information DB 33 (see FIG. 4 described later). The instruction content determination unit 126 determines, for example, requested monitoring / operation content.
[0100] The mobile object information transmission unit 127 executes a process of transmitting mobile object information sent from the mobile object information acquisition unit 42 (see Figure 5 described later) of the mobile object 40 to the terminal 10 based on information from the control status management unit 121.
[0101] The surrounding environment information transmission unit 128 transmits the surrounding environment information sent from the surrounding environment information acquisition unit 43 (see FIG. 5 described later) to the terminal 10 based on the information from the control status management unit 121 .
[0102] The operation signal transmission unit 129 executes a process of transmitting an operation signal from the terminal 10 associated with assistance / operation to the mobile object 40 based on information from the control status management unit 121 .
[0103] The communication information acquisition unit 130 acquires communication information between the terminal 10 and the control server 20, and between the control server 20 and the mobile object 40. The communication information includes information on the operating status of communication devices, communication delays, and the like.
[0104] The MRM instruction unit 131 executes a process of transmitting an MRM instruction to the mobile object 40 as an operation signal when a communication abnormality occurs between the terminal 10 and the control server 20 or when an abnormality occurs in the control server 20 .
[0105] The remote support request notifier 132 notifies the terminal 10 that remote assistance / operation is required when remote assistance / operation is required, based on at least one of the information stored in the storage device 30, the mobile body information, and the surrounding environment information of the mobile body 40. The information stored in the storage device 30 includes at least one of information from the ODD information DB 31, the traveling task information DB 32, and the operator qualification information DB 34.
[0106] The autonomous driving continuation determination unit 133 determines whether the autonomous driving of the mobile body 40 can continue based on the information in the ODD information DB 31, and if it is determined that the mobile body 40 cannot continue driving without operation / assistance, executes a process to request responsibility transfer through the transfer request presentation unit 112.
[0107] 1 again, the storage device 30 stores various information for the control server 20 to execute various processes. The storage device 30 is realized by a hard disk, a semiconductor memory, etc. Fig. 4 is a diagram showing information stored in the storage device 30 according to this embodiment.
[0108] As shown in FIG. 4, the storage device 30 includes an ODD information DB 31, a driving task information DB 32, a manual information DB 33, and an operator qualification information DB .
[0109] The ODD information DB 31 includes information indicating under what conditions the automatic driving system 46 (see FIG. 5 described later) installed in the mobile object 40 can operate (i.e., the conditions under which the mobile object 40 can travel autonomously). For example, the ODD information DB 31 stores road context conditions under which autonomous driving / remote operation can be performed under autonomous driving / remote assistance, communication state conditions, and skill information of the operator 50. Specific examples of ODD information will be described later using FIG. 36.
[0110] The traveling task information DB 32 stores schedules of traveling tasks to be executed by each mobile object 40, such as following a road, turning right at an intersection, etc. Specific examples of traveling task information will be described later with reference to FIG.
[0111] The manual information DB 33 stores manual information indicating monitoring items, operation items, etc. of the operator 50. The manual information DB 33 stores information on what the operator 50 checks and operates during remote assistance / operation (remote support) for each road context. Specific examples of the manual information will be described later using FIG. 24 etc.
[0112] The operator qualification information DB 34 stores driving tasks that can be remotely assisted / operated for each operator 50. A specific example of the operator qualification information will be described later with reference to FIG.
[0113] Referring again to FIG. 1 , the mobile object 40 includes an autonomous mobile robot, such as an automatic delivery robot. In addition to an autonomous mobile robot, the mobile object 40 may also include an autonomous mobile vehicle, such as an automobile or a motorcycle. In the embodiment, the mobile object 40 is not completely autonomous, but is remotely monitored by an operator 50 via a remote control system 1. In some cases, the mobile object 40 may move with remote assistance or remote monitoring by the operator 50 via the remote control system 1. In the following description, the mobile object 40 is assumed to be an automatic delivery robot, particularly an automatic monitoring robot that patrols and monitors a facility or premises.
[0114] The mobile body 40 is not limited to the autonomous mobile type described above, but may be, for example, a manually operated type that can be manually operated by a driver on board. The work performed by the mobile body 40 is not limited to the above-described monitoring, but may be other work such as cleaning, delivery, sales, or advertising. The mobile body 40 may also be an aircraft with a flying function (for example, an autonomously flying drone).
[0115] FIG. 5 is a block diagram showing the functional configuration of a moving object 40 according to this embodiment.
[0116] As shown in FIG. 5, the mobile object 40 has an MRM execution unit 41, a mobile object information acquisition unit 42, a surrounding environment information acquisition unit 43, a communication information acquisition unit 44, a mobile object monitoring unit 45, and an automatic driving system 46.
[0117] The MRM execution unit 41 executes MRM when an MRM instruction is received from the terminal 10 or the control server 20, when a communication abnormality occurs between the control server 20 and the mobile body 40, or when an abnormality occurs in the mobile body 40.
[0118] The mobile object information acquisition unit 42 acquires information about the state of the mobile object 40 , such as the vehicle speed, angular velocity, and system state, from various sensors mounted on the mobile object 40 .
[0119] The surrounding environment information acquisition unit 43 acquires surrounding environment information such as sensing information from cameras, LiDAR, radar, etc. mounted on the mobile body 40, and target or grid maps obtained by processing the sensing information using machine learning, signal processing, etc.
[0120] The communication information acquisition unit 44 acquires the communication state between the control server 20 and the mobile object 40 and delay information of the transmitted operation command. The communication information acquisition unit 44 may acquire the delay information based on, for example, a timestamp attached to the operation command.
[0121] The mobile object monitoring unit 45 monitors whether or not there is an abnormality in the mobile object 40. For example, the mobile object monitoring unit 45 monitors whether or not the time in the mobile object 40 is being discontinuously changed. For example, the mobile object monitoring unit 45 monitors whether or not a time jump has occurred due to a leap second or the like.
[0122] The automatic driving system 46 is a system that executes control for the autonomous driving of the mobile body 40. The automatic driving system 46 automatically performs operations such as acceleration / deceleration, steering, and braking of the mobile body 40. The automatic driving system 46 is an example of an autonomous mobile system.
[0123] Here, the form of remote control in the remote control system 1 according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing the form of remote control according to this embodiment.
[0124] As shown in Figure 6, there are three control modes: "monitoring," "assistance," and "driving." Monitoring means remote monitoring, assistance means remote assistance, and driving means remote operation.
[0125] In all three modes, the UI (for example, the display unit 110a) displays information about the moving object and the surrounding environment.
[0126] The control responsibility of the operator 50 is "no" for monitoring, but "yes" for assistance and operation. For example, the operator 50 who is remotely monitoring is not responsible for an accident or the like caused by the mobile object 40 that he is remotely monitoring.
[0127] The available operations indicate the operations that the operator 50 can perform on the moving body 40, with monitoring being "none," assistance being "safety stop / stop autonomous driving / resume autonomous driving / send command to change route," and operation being "longitudinal and lateral control / acceleration / brake." With assistance, the operator 50 can perform simple operations such as stopping and starting driving, and with operation, more complex operations can be performed. Stopping autonomous driving also includes pausing.
[0128] The simultaneous control constraint indicates a constraint when one mobile body 40 is remotely controlled simultaneously by multiple operators 50, and the monitoring is "none", and the assistance and operation are "only one operator 50 can assist / operate one vehicle (mobile body)." Note that with regard to remote assistance, one operator 50 may be able to assist multiple vehicles simultaneously.
[0129] The safety function activation conditions indicate conditions for not activating the safety functions of the moving body 40, with monitoring being "none," assistance being "there is a delay in the video / vehicle information transmission and display function and it is not operating correctly" and "there is a delay in the transmission / execution of instruction commands and it cannot be performed correctly," and operation being "there is a delay in the video / vehicle information transmission and display function and it is not operating correctly" and "there is a delay in the transmission / execution of operation commands and it cannot be performed correctly." With assistance and operation, when these conditions are no longer met, the safety functions of the moving body 40 can be activated.
[0130] The levels of autonomous driving that can be achieved are "Level 4" for monitoring, "Level 3, 2" for assistance, and "Level 2, 1" for operation. Level 4 is also called high-level automated driving, Level 3 is also called conditional automated driving, Level 2 is also called partial automated driving, and Level 1 is also called driving assistance.
[0131] As described above, there are multiple types of remote control, and depending on the type, the operator 50 may have driving responsibilities.
[0132] 2. Operation of the Remote Control System Next, the operation of the remote control system 1 configured as described above will be described with reference to Figures 7 to 10. Below, a process for changing the operator 50 controlling the moving object 40 while the moving object 40 is autonomously traveling will be described.
[0133] Fig. 7 is a flowchart showing the operation (information processing method) of the transfer requesting side in the remote control system 1 according to this embodiment. Fig. 7 shows the operation executed by the terminal 10 of the operator 50 when the operator 50 currently remotely assisting the moving object 40 transfers control responsibility. Fig. 8 is a diagram showing a screen displayed on the display unit 110a of the transfer requesting side according to this embodiment. Hereinafter, the operator 50 requesting the transfer will also be simply referred to as operator 50 or operator A, and the operator 50 receiving the transfer request will also be referred to as another operator 50 or operator B.
[0134] 7, the transfer request input unit 212 receives a responsibility transfer request from operator A (S11). As shown in (a) of Fig. 8, for example, the transfer request input unit 212 displays a button display e1 for receiving the transfer request from operator A, and when it detects that the button display e1 has been pressed, it determines that the responsibility transfer request has been received.
[0135] On the display unit 110a, the control status presentation unit 111 displays "Assisting," which is a status display b1 indicating that the operator 50 is responsible for control. The start / end input unit 215 displays a button display e2 for ending control. The mobile object information presentation unit 115 displays "Running," which is a status display c indicating that the mobile object 40 is traveling. The surrounding environment information presentation unit 116 displays video captured around the mobile object 40, sensing information, and a status display d indicating that a vehicle is approaching the mobile object 40. The operation signal transmission unit 216 displays an operation display a2 for temporarily stopping the mobile object 40. Note that, because the mobile object 40 is traveling, the button display e2 is inactive (inoperable state). Furthermore, the input device 210 displays an operation display a1 for safely stopping the mobile object 40. Furthermore, because remote assistance is currently being performed, the operator A can temporarily stop the mobile object 40 using the operation display a2.
[0136] 7, next, when the transfer request input unit 212 receives the responsibility transfer request in step S11, it issues a responsibility transfer notification to the terminal 10 of another operator 50 (operator B) (S12). Issuing means transmitting the responsibility transfer notification to the terminal 10 of operator B. In other words, when an operation indicating that the responsibility for the mobile object 40 is to be transferred is acquired from operator A, the responsibility transfer notification is sent. The responsibility transfer notification is a notification indicating that operator A is requesting the transfer of the control responsibility for the mobile object 40 that he is remotely assisting.
[0137] 8(b), the control status display unit 111 switches the display from the status display b1 to the status display b2 "Assisting (requesting changeover)" indicating that a request to transfer control responsibility is being made. In other words, the status display b2 is presented to the operator A. This notifies the operator A that a request to transfer control responsibility is being made.
[0138] 7 , next, the transfer request input unit 212 determines whether or not Operator B has accepted the control responsibility (S13). When the transfer request input unit 212 acquires (for example, receives) a notification indicating that the transfer request is accepted from the terminal 10 of Operator B, the transfer request input unit 212 determines that Operator B has accepted the control responsibility. The notification is an example of information indicating whether or not the transfer request is accepted.
[0139] Next, when the transfer request input unit 212 determines that Operator B has accepted the control responsibility (Yes in S13), it releases Operator A from the control responsibility state (S14) and notifies Operator A of the release of the control responsibility state (S15). Furthermore, when it is determined that Operator B has accepted the control responsibility, the terminal 10 of Operator A transmits information indicating (where) the control responsibility for the mobile object 40 and information indicating (the granting of) the operation authority for the mobile object 40 to the terminal 10 of Operator B. The information indicating (where) the control responsibility includes information indicating that the control responsibility will be transferred from Operator A to Operator B, and the information indicating (the granting of) the operation authority includes information indicating that the operation authority will be transferred from Operator A to Operator B.
[0140] As shown in (c) of FIG. 8 , the control status presentation unit 111 switches the display from the status display b2 to the status display b3, which indicates that the transition to the control responsibility state (the state in which control responsibility exists), has been completed, indicating "Operator B is assisting." This notifies Operator A that the control responsibility state has been released and control responsibility has been transferred to Operator B. At this time, the operation display a2 is switched to an inactive state (e.g., hidden). This prevents Operator A from operating the mobile object 40. In other words, the operation authority for the mobile object 40 is transferred. For example, Operator A's remote control mode switches from remote assistance to remote monitoring. Furthermore, the transfer request input unit 212 switches from the button displays e1 and e2 to display a button display e3 as a response button when a transfer request is received from Operator B. The operation authority is the authority to permit the mobile object 40 to perform an operation according to the control mode. The operation authority is a different concept from the control responsibility and may be transferred at a different timing from the management responsibility. In other words, the operating authority and the management responsibility can be transferred (and received) independently. Here, "independently" means that the operating authority and the management responsibility can be transferred individually.
[0141] In this way, the fact that responsibility for the mobile object 40 has been transferred to operator B and the fact that the operating authority for the mobile object 40 has been transferred to operator B are presented to operator A in different ways. For example, when responsibility for the mobile object 40 is transferred to operator B, a status display b3 indicating that operator B is remotely controlling the mobile object 40 is presented to operator A, and when the operating authority for the mobile object 40 is transferred to operator B, an operating display a2 (operating unit) for remotely operating the mobile object 40 is made unavailable (for example, hidden).
[0142] Furthermore, when the control status presentation unit 111 acquires (for example, receives) a notification from the terminal 10 of the operator B indicating that the acceptance of the transfer request is refused, the control status presentation unit 111 determines that the operator B has refused the control responsibility (No in S13), and notifies the operator A that the release of the control responsibility status has failed (S16). In this case, the operator A will continue to provide remote assistance to the mobile object 40, for example.
[0143] Next, the operation of the terminal 10 on the side receiving the transfer request will be described with reference to FIGS. 9 and 10 . FIG. 9 is a flowchart showing the operation (information processing method) of the side receiving the transfer request in the remote control system 1 according to this embodiment. FIG. 9 shows the operation executed by the terminal 10 possessed by another operator 50 (operator B) currently remotely monitoring the mobile object 40 when the operator B receives a request to transfer the control responsibility. FIG. 10 is a diagram showing a screen displayed on the display unit 110a on the side receiving the transfer request in the remote control system 1 according to this embodiment. It is assumed that the other operator 50 is remotely monitoring the mobile object 40 that is being remotely assisted by the operator 50 (operator A). The responsibility transfer notification may be sent to at least one of the other operators 50 remotely monitoring the mobile object 40 and the other operators 50 not remotely monitoring the mobile object 40.
[0144] 10A, during remote monitoring, the control status display unit 111 displays on the display unit 110a a status display b4 indicating that operator A is responsible for control, which is "Operator A is assisting,". The display unit 110a also displays an operation display a1, a button display e3, a status display c, and a situation display d. At this point, the display unit 110a does not display an operation display a2 for temporarily stopping the moving object 40.
[0145] 9, the terminal 10 of the operator B receives the responsibility transfer notification issued from the terminal 10 of the operator A in step S12 (S21). For example, the transfer request presentation unit 112 may function as a first acquisition unit that acquires the responsibility transfer notification.
[0146] As shown in FIG. 10B , upon receiving a responsibility transfer notification, the transfer request presentation unit 112 switches the display from status display b4 to status display b5, which indicates that a transfer request (switch request) has been notified, and displays "Switch Request (Operator A is Handling)." The transfer request response input unit 211 also displays a button display e4, "Switch Request Accepted," indicating that the transfer request has been accepted, and a button display e5, "Switch Request Rejected," indicating that the transfer request has been rejected. The transfer request response input unit 211 may, for example, display the buttons e4 and e5 in different modes. For example, the transfer request response input unit 211 may display the buttons e4 and e5 in different colors.
[0147] 9 , the transfer request response input unit 211 then determines whether or not the acceptance of the request for transfer of the control responsibility has been accepted (S22). The transfer request response input unit 211 determines that the acceptance has been accepted if the button display e4 is operated by the operator B, and determines that the acceptance has been rejected if the button display e5 is operated by the operator B or if no button operation is performed for a predetermined period of time. The transfer request response input unit 211 functions as a second acquisition unit that acquires from the operator A an operation indicating whether or not to accept the transfer request. The transfer request response input unit 211 also transmits the determination result of whether or not the acceptance of the request for transfer of the control responsibility has been accepted to the terminal 10 of the operator A.
[0148] Next, if it is determined that the acceptance has been accepted (Yes in S22), the transfer request response input unit 211 notifies the operator of the start of the control responsibility state (S23) and transitions to the control responsibility state (S24). For example, when the terminal 10 of operator B receives information indicating (the location of) control responsibility for the moving object 40 and information indicating (the granting of) the operation authority for the moving object 40 from the terminal 10 of operator A, the terminal 10 of operator B executes step S24. The transfer request response input unit 211 functions as a processing unit that executes processing to transfer the responsibility for the moving object 40 and the operation authority for the moving object 40 to operator B.
[0149] As shown in (c) of FIG. 10 , upon accepting the responsibility transfer notification, the control status presentation unit 111 switches the status display b5 to "Assisting," which is the status display b1 indicating that remote assistance is being performed and that Operator B has control responsibility. Furthermore, the operation signal transmission unit 216 displays an operation display a2 for temporarily stopping the moving object 40 that has accepted control responsibility. In other words, the operation display a2 becomes active. This notifies Operator B of the transition to a control responsibility state, and the operation authority (operation display a2) for the moving object 40 is transferred. Furthermore, the transfer request input unit 212 switches from button displays e4 and e5 to button displays e1 and e2.
[0150] For example, step S14 shown in FIG. 7 and step S24 shown in FIG. 9 may be executed simultaneously.
[0151] In this way, the fact that responsibility for the mobile object 40 has been transferred to operator B and the fact that the operation authority for the mobile object 40 has been transferred to operator B are presented to operator B in different ways. For example, when responsibility for the mobile object 40 is transferred to operator B, a status display b3 indicating that the mobile object 40 is being remotely controlled is presented to operator B, and when the operation authority for the mobile object 40 is transferred to operator B, a display indicating that an operation display a2 (operation unit) for remotely operating the mobile object 40 has become available is presented to operator B. For example, the operation display a2, which was previously hidden, is displayed. Also, for example, the status display b3 and the operation display a2 are simultaneously displayed on the display unit 110a. Also, for example, the status display b3 and the operation display a2 are arranged spaced apart on the display unit 110a. Also, the status display b3 and the operation display a2 are displays including text information, but may also be displays including illustrations, etc.
[0152] As a result, through the three steps of requesting responsibility transfer, approving acceptance of responsibility transfer, and responsibility transfer, operator A's control responsibility can be reliably transferred to operator B. Furthermore, because explicit acceptance of responsibility is made, it becomes clear who is responsible when the operator 50 is replaced, so that, for example, control responsibility can be clearly handed over from operator A to operator B. Furthermore, according to the above method, seamless transfer of control responsibility is possible even while the mobile object 40 is traveling, and it is possible to reduce the time loss caused by the replacement of the operator 50.
[0153] Note that when the responsibility for the mobile object 40 and the authority to operate the mobile object 40 are transferred from operator A to operator B, unless operator B gives permission, other operators 50 other than operator B among the one or more operators 50 may not be permitted to assume responsibility for the mobile object 40. In other words, the responsibility for the mobile object 40 held by operator B may not be transferred to other operators 50 other than operator B unless operator B gives permission (for example, unless button display e1 is operated).
[0154] (First Modification of the Embodiment) The remote control system 1 according to this modification will be described below with reference to FIG. 11 . The following description will focus on differences from the embodiment, and descriptions of the same or similar content as the embodiment will be omitted or simplified. FIG. 11 is a flowchart showing the operation (information processing method) of the transfer requesting side in the remote control system 1 according to this modification. FIG. 11 shows an example in which, if the control responsibility cannot be transferred, the moving object 40 is automatically evacuated (MRM).
[0155] 11 , when the control status notification unit 111 receives a notification from the terminal 10 of another operator 50 (operator B) indicating that operator B has refused to accept responsibility, or when a certain period of time has passed without receiving a notification (Yes in S13a), the control status notification unit 111 notifies the operator 50 (operator A) that the control responsibility status has been released (S16). Furthermore, the MRM instruction unit 219 instructs the mobile unit 40 to execute MRM (S17). For example, the MRM instruction unit 219 transmits a signal to the mobile unit 40 to execute MRM. This allows the mobile unit 40 to be forcibly stopped, thereby releasing the control responsibility status of operator A (S14).
[0156] (Second Modification of the Embodiment) The following describes a remote control system 1 according to this modification with reference to FIGS. 12 to 15. The following description will focus on differences from the embodiment, and will omit or simplify descriptions of content that is the same as or similar to the embodiment. FIG. 12 is a flowchart illustrating the operation (information processing method) of the side requesting assistance in the remote control system 1 according to this modification. FIG. 12 describes the operation performed by the terminal 10 when an assistant control person (cooperator) is added. In the example of FIG. 12, when a single operator 50 cannot handle all the remote assistance (or remote operation) of the mobile object 40, multiple operators 50 perform simultaneous control. FIG. 13 is a diagram illustrating a screen displayed on the display unit 110a of the side requesting assistance in the remote control system 1 according to this modification. Hereinafter, the operator 50 requesting assistance will be referred to simply as operator 50 or operator A, and the operator 50 receiving the assistance request will be referred to as another operator 50 or operator B.
[0157] 12, the transfer request input unit 212 receives a request for assistance from an operator A (S31). As shown in (a) of Fig. 13, for example, the transfer request input unit 212 displays a button display e6 for receiving the request for assistance from the operator A, and when it is detected that the button display e6 has been pressed, it determines that the request for assistance has been received.
[0158] 12 , next, when the transfer request input unit 212 accepts the assistance request in step S31, it issues an assistance request notice to the terminal 10 of operator B (S32). Issuing an assistance request notice means transmitting the assistance request notice to the terminal 10 of operator B. The assistance request notice is a notice requesting operator B to join as an assistant (or operator) of the mobile object 40.
[0159] As shown in (b) of Figure 13, the control status presentation unit 111 switches the display from the status display b1 to "Assisting (Requesting Control Assistance)," which is a status display b6 indicating that a request for control assistance is being made. This notifies the operator A that assistance is being requested. In addition, the transfer request input unit 212 displays "Cancel Assistance Request," which is a button display e7 for canceling the assistance request.
[0160] 12, next, the transfer request input unit 212 determines whether the operator B has accepted the specified number of assistants (S33). The specified number of assistants may be determined based on the mobile object 40 or the state of the mobile object 40, or the operator 50 may decide the number of assistants.
[0161] The transfer request input unit 212 may make the determination in step S33 by counting the number of operators B who have accepted assistance.
[0162] Next, when the transfer request input unit 212 determines that a predetermined number of operators B have accepted the assistance (Yes in S33), it shares part of the control responsibility with or transfers part of the control responsibility to the operators B who accepted the assistance (S34), and notifies operator A that the control responsibility has been shared or transferred (S35). For example, when it is determined that a predetermined number of operators B have accepted the assistance, the terminal 10 of operator A transmits information indicating part (the location of) the control responsibility for the mobile object 40 and information indicating part (the granting of) the operation authority for the mobile object 40 to the terminal 10 of operator B who accepted the assistance.
[0163] 13(c), the control status presentation unit 111 switches the display from the status display b6 to the status display b7, "Assisting, you are in charge (Operator B is supporting)," which indicates that part of the control responsibility has been shared or transferred. This notifies Operator A that Operator B has joined as another operator 50 as an assistant control operator. At this time, the communication unit 220 displays a call display f on the display unit 110a for communication with Operator B, the assistant control operator.
[0164] Furthermore, if it is determined that the specified number of operators B have not accepted assistance (No in S33), the control status notification unit 111 notifies operator A that the sharing / transfer of control responsibility has failed (S36). In this case, operator A will continue to provide remote assistance to the mobile object 40, for example.
[0165] Next, the operation of the terminal 10 on the side receiving the assistance request will be described with reference to Figs. 14 and 15. Fig. 14 is a flowchart showing the operation (information processing method) on the side receiving the assistance request in the remote control system 1 according to this modified example. Fig. 14 shows the operation executed by the terminal 10 possessed by another operator 50 when the other operator 50 who is currently remotely monitoring the mobile body 40 receives the assistance request. Fig. 15 is a diagram showing a screen displayed on the display unit 110a on the side receiving the assistance request in the remote control system 1 according to this modified example.
[0166] As shown in (a) of Figure 15, during remote monitoring, the control status presentation unit 111 displays on the display unit 110a the status display b4 "Operator A is assisting," which indicates that operator A is responsible for control.
[0167] As shown in FIG. 14, the terminal 10 of the other operator 50 receives the assistance request notice issued from the terminal 10 of the operator A in step S32 (S41).
[0168] As shown in (b) of FIG. 15 , upon receiving an assistance request notification, the transfer request presentation unit 112 switches the display from the status display b4 to "Control Assistance Request (Operator A is Handling)," which is a status display b8 indicating that an assistance request has been notified. Furthermore, the transfer request response input unit 211 displays a button display e8 indicating "Assistance Request Accepted," which indicates that the assistance request is accepted, and a button display e9 indicating that the assistance request is rejected. The transfer request response input unit 211 may, for example, differentiate the display modes of the button displays e8 and e9. For example, the transfer request response input unit 211 may differentiate the display colors of the button displays e8 and e9.
[0169] 14, the transfer request response input unit 211 then determines whether or not the acceptance of the assistance request has been accepted (S42). If the button display e8 is operated by the operator B, the transfer request response input unit 211 determines that the acceptance has been accepted, and if the button display e9 is operated by the operator B or if no button is operated for a predetermined period of time, the transfer request response input unit 211 determines that the acceptance has been rejected.
[0170] Next, if the transfer request response input unit 211 determines that the acceptance has been approved (Yes in S42), it notifies the start of the control responsibility state (S23) and transitions to the control responsibility state (S24).
[0171] As shown in FIG. 15C , when the assistance request is accepted, the control status presentation unit 111 switches the display from the status display b8 to the status display b9 indicating "Assisting (Operator A is primarily in charge)," which indicates that Operator B is assisting in control. The status display b9 can also be considered to notify that the transition to a state in which partial control responsibility has been assumed has been completed. Furthermore, the operation signal transmission unit 216 displays the operation display a2 for temporarily stopping the moving body 40 that accepted the assistance request. In other words, the operation display a2 becomes active. This notifies Operator B of the transition to a control responsibility state (e.g., a control assistance state), and the operation authority (operation display a2) of the moving body 40 is shared. Furthermore, the transfer request input unit 212 switches from the button displays e8 and e9 to the button display e2.
[0172] In this way, the transfer request is a request to transfer part of the responsibility of the mobile unit 40, and part of the responsibility of the mobile unit 40 and part of the operating authority of the mobile unit 40 may be transferred from operator A to operator B.
[0173] 13 and 15, an example in which both operators A and B can pause has been described, but the present invention is not limited to this, and the operations that operators A and B can perform may be different. In other words, the assistance or operation content may be shared between operators A and B.
[0174] The control responsibility may be held by operator A who is the main person in charge, or may be allocated to operators A and B according to their operating authority. For example, the control responsibility may be allocated using a table in which operating authority is associated with how the control responsibility is allocated.
[0175] This makes it possible to more reliably clarify who is responsible for control and to allocate operating authority, even when multiple people simultaneously control the mobile body 40, through the three steps of starting control, requesting assistance, and accepting and approving the assistance request.
[0176] (Variation 3 of the embodiment) The remote control system 1 according to this variation will be described below with reference to FIG. 16. Note that the following description will focus on differences from variation 2 of the embodiment, and description of content that is the same as or similar to variation 2 of the embodiment will be omitted or simplified. FIG. 16 is a flowchart showing the operation (information processing method) of the assistance requesting side in the remote control system 1 according to this variation. FIG. 16 describes an example in which it is necessary to add an assistant (another operator 50) to assist the operator 50 (operator A) who provides remote assistance or remote operation, and MRM is executed if the specified number of assistants cannot be gathered.
[0177] 16 , when it is determined that the prescribed number of other operators 50 (operator B) have not accepted assistance (No in S33), the MRM instructing unit 219 instructs the mobile body 40 to execute MRM (S37), and the process proceeds to step S36. For example, the MRM instructing unit 219 transmits a signal to the mobile body 40 to execute MRM.
[0178] (Fourth Modification of the Embodiment) The following describes a remote control system 1 according to this modification, with reference to Figures 17 to 21. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified.
[0179] 17 is a flowchart showing the operation (information processing method) of the side requesting transfer in the remote control system 1 according to this modified example. In FIG. 17, an example will be described in which an available operator 50 takes over a mobile object 40 for which another operator 50 has control responsibility. Being available means that the operator is not currently assisting / operating the mobile object 40, for example, not being responsible for the mobile object 40.
[0180] 18 is a diagram showing a first screen displayed on the display unit 110a on the side requesting transfer in the remote control system 1 according to this modified example. Fig. 18 shows an example of the display of the number of controlled vehicles / controlled vehicle number when operator B is remotely controlling three vehicles and one mobile object 40 is requesting assistance. It is assumed that there is no operator 50 remotely controlling the mobile object 40 requesting assistance.
[0181] FIG. 19 is a diagram showing the second screen displayed on the display unit 110a of the side requesting the transfer in the remote control system 1 according to this modified example.
[0182] As shown in FIG. 17, the terminal 10 held by an available operator 50 (operator A) displays the number of vehicles / vehicles under control of each operator 50 on the display unit 110a (S51).
[0183] As shown in FIG. 18, the display unit 110a displays a map display section A11 showing the current positions of a plurality of mobile objects 40, and a list display section A13 showing the relationship between the mobile objects 40 and the operators 50, etc.
[0184] The map display section A11 displays a mark B9 indicating the position of the moving body 40 having the moving body ID "X9", a mark B10 indicating the position of the moving body 40 having the moving body ID "X10", a mark B11 indicating the position of the moving body 40 having the moving body ID "X11", and a mark B12 indicating the position of the moving body 40 having the moving body ID "X12". The map display section A11 may further display the movement route of each moving body 40.
[0185] The list display section A13 displays the mobile unit ID, an image of the mobile unit 40, a first display h1 displayed by the mobile unit selection section 217, a second display h2 and a third display h3 displayed by the control status presentation section 111, and a fourth display h4 displayed by the remote response request presentation section 119.
[0186] The first display h1 is displayed on each of the multiple moving bodies 40, and when pressed by the operator 50, causes a transition to a control screen (for example, a screen shown in FIG. 19(a) described later) of the moving body 40 corresponding to the pressed first display h1. When two or more moving bodies 40 are selected by the operator 50, it may be possible to simultaneously control the multiple moving bodies 40.
[0187] The second display h2 indicates which operator 50 is responsible for controlling which moving body 40. In Fig. 18, an example is shown in which operator B is responsible for controlling three moving bodies 40 with moving body IDs "X9" to "X11".
[0188] The third display h3 indicates how many mobile objects 40 each operator 50 is assisting / operating. Fig. 18 shows an example in which operator B is assisting / operating three mobile objects 40, and operator C is not assisting / operating any mobile objects 40.
[0189] The fourth display h4 indicates that the mobile object 40 is not being remotely controlled by any operator 50 and requires remote control. In Fig. 18, assistance is required for the mobile object 40 with the mobile object ID "X12". The assistance request here may be, for example, a request from the mobile object 40 with the mobile object ID "X12".
[0190] Furthermore, the list display section A13 displays an image indicating the status of each mobile unit 40. The images indicating the status of each mobile unit 40 include, from the left, an image indicating that there is a notification from the mobile unit 40, an image indicating whether or not communication with the mobile unit 40 is possible, an image indicating the communication status with the mobile unit 40, and an image indicating the remaining battery charge of the mobile unit 40.
[0191] 18 is displayed, an available operator 50 can check the screen and determine which mobile object 40 he or she is responsible for controlling. In other words, the terminal 10 can assist the operator 50 in determining which mobile object 40 he or she is responsible for controlling.
[0192] For example, when an operator 50 selects one of the first displays h1 in FIG. 18 , the display transitions to the screen shown in FIG. 19A, which is the control screen for the corresponding mobile object 40. FIG. 19A shows an example in which a mobile object 40 for which operator B is responsible for control is selected, and "Operator B is providing assistance" is displayed as the status display b3. At this time, the takeover request input unit 214 displays a button display e3 for requesting that operator B take over the mobile object 40 for which operator B is responsible for control. The button display e3 can also be considered a display for accepting assistance / operation. The control status display unit 111 also displays a vehicle number display g1 indicating the number of vehicles controlled by each operator 50.
[0193] 17 again, next, the transfer request input unit 214 receives a press of the control responsibility transfer button (button display e3 shown in FIG. 19(a)) from the operator 50 (S52). For example, it is assumed that the transfer request input unit 214 receives a press of the first display h1 corresponding to the moving object 40 whose moving object ID is "X9."
[0194] Next, the transfer request input unit 214 issues a transfer request notice to the terminal 10 of operator B (S53). When an operation requesting the transfer of responsibility for the mobile unit 40 is acquired from operator A, the transfer request notice is transmitted to operator B based on the operation. The transfer request notice is, for example, a notice indicating that operator A is requesting that operator A take over control responsibility for the mobile unit 40 whose mobile unit ID is "X9".
[0195] As shown in FIG. 19B , the control status presentation unit 111 switches the display from the status display b3 to "Changeover requested (Operator B is handling)," which is a status display b10 indicating that a request to transfer control responsibility is being made. This notifies Operator A that Operator B is being requested to transfer control responsibility for the mobile object 40 whose mobile object ID is "X9." The transfer request input unit 214 displays a button display e10 indicating "Cancel acceptance," which indicates that the transfer request is to be canceled. When Operator 50 presses the button display e10, the transfer request input unit 214 executes processing to cancel the transfer request to Operator B. For example, the transfer request input unit 214 may notify the terminal 10 of Operator B that the transfer request has been canceled.
[0196] 17 , next, the transfer request input unit 214 determines whether the corresponding operator 50 (here, operator B) has accepted the transfer request (S54). When the transfer request input unit 214 receives a notification from the terminal 10 of operator B indicating that the transfer request has been accepted, the transfer request input unit 214 determines that the transfer request has been accepted.
[0197] Next, if it is determined that Operator B has accepted the transfer request (Yes in S54), the control state notification unit 111 transitions to a control responsibility state (S55) and notifies Operator A that it is in the control responsibility state (S56). As a result, the control responsibility and operation authority for the moving object 40 are transferred from Operator B to Operator A.
[0198] As shown in (c) of FIG. 19 , the control status presentation unit 111 switches the display from the status display b10 to "assisting," which is the status display b1 indicating that the operator is in a control responsibility state. This allows the operator A to be notified that the operator B has taken over control responsibility for the mobile unit 40 with the mobile unit ID "X9" and that remote assistance is required (that the operator A is currently responding). At this time, the operation signal transmission unit 216 also displays an operation display a2 for temporarily stopping the taken-over mobile unit 40. The control status presentation unit 111 also displays an updated number display g2 after the takeover. The number of vehicles under operator B's control is updated from three to two.
[0199] Furthermore, if it is determined that Operator B has not accepted the transfer request (No in S54), the control status presentation unit 111 notifies Operator A of the transfer failure (S57). In this case, for example, the first screen shown in FIG. 18 may be displayed again.
[0200] The taking over may be by remote assistance or remote control.
[0201] Next, the operation of the terminal 10 that receives the transfer request will be described with reference to Figs. 20 and 21. Fig. 21 is a flowchart showing the operation (information processing method) of the side that receives the transfer request in the remote control system 1 according to this modified example. Fig. 20 shows the operation that is executed by the terminal 10 possessed by operator B when a request to transfer control responsibility is received from an available operator 50 (operator A). Fig. 21 is a diagram showing a screen displayed on the display unit 110a that receives the transfer request in the remote control system 1 according to this modified example.
[0202] As shown in (a) of Figure 21, before receiving the transfer request, the display unit 110a displays "assisting", which is a status display b1 indicating that the operator is responsible for control. Also, a number display g3 indicating the number of vehicles controlled by operators 50 other than operator B is displayed. The number of vehicles controlled by the available operator 50 (operator A who requested the transfer in Figure 17) is 0. Also, the transfer request input unit 212 displays "transfer request", which is a button display e11 for making a transfer request.
[0203] As shown in FIG. 20, the terminal 10 of the operator B receives a transfer request notification from the terminal 10 of the operator A (S61).
[0204] 21(b), when a transfer request notification is received, the control status display unit 111 switches the display from the status display b1 to a status display b12 indicating that a transfer request (switch request) has been notified, which is "Switch Request." In addition, the transfer request response input unit 213 displays a button display e4 indicating that the transfer request is accepted, which is "Switch Request Accepted," and a button display e5 indicating that the transfer request is rejected, which is "Switch Request Rejected."
[0205] 20 , next, the transfer request response input unit 213 determines whether the acceptance of the request to transfer control responsibility has been accepted and whether a timeout has occurred (S62). The transfer request response input unit 213 determines that the acceptance has been accepted if the button display e4 is operated by the operator B or if a timeout has occurred (if the button display e4 or e5 is not operated by the operator B for a predetermined time or longer), and determines that the acceptance has been rejected if the button display e5 is operated by the operator B. By determining that the acceptance has been accepted if a timeout has occurred, it becomes possible to have the operator A transfer responsibility for the mobile object 40, for example, even in a situation where the operator B is unable to respond due to poor health or the like.
[0206] Next, if the takeover request response input unit 213 determines that the acceptance has been accepted (Yes in S62), it cancels the remote assistance / operation (S63) and notifies Operator B of the cancellation of the remote assistance / operation (S64). Furthermore, if it is determined that the acceptance has been accepted, for example, the terminal 10 of Operator A transmits information indicating (where) the responsibility for controlling the mobile object 40 and information indicating (the granting of) the authority to operate the mobile object 40 to the terminal 10 of Operator B. The information indicating (where) the responsibility for controlling the mobile object 40 includes information indicating that the responsibility for controlling the mobile object 40 will be transferred from Operator A to Operator B, and the information indicating (the granting of) the authority to operate the mobile object 40 includes information indicating that the authority to operate the mobile object 40 will be transferred from Operator A to Operator B.
[0207] As shown in (c) of Figure 21, when the transfer request notification is accepted, the control status presentation unit 111 switches the display of the status display b12 to "Operator A is assisting," which is a status display b7 indicating that operator B is currently assisting. The display of the status display b7 corresponds to notifying operator B that remote assistance / operation has been canceled. At this time, the operation signal transmission unit 216 also hides (disables) the operation display a2. The control status presentation unit 111 also displays the updated number of vehicles display g4 after the transfer.
[0208] This allows operator A to reliably take over control responsibility from operator B through the three steps of starting control, requesting a transfer, and accepting and approving the transfer request. In addition, because the acceptance and approval of the transfer request is explicitly performed, the responsibility at the time of handover to operator A is made clear, so that, for example, operator B can clearly hand over control responsibility to operator A.
[0209] (Fifth Modification of the Embodiment) The following describes a remote control system 1 according to this modification, with reference to Figures 22 to 25. The following description focuses on differences from the embodiment, and omits or simplifies descriptions of content that is the same as or similar to the embodiment. This modification describes a terminal 10 that, when the travel route of a mobile object 40 is set in advance, presents information that can assist an operator 50 who has received a transfer request in determining whether or not to accept the transfer request.
[0210] 22 is a flowchart showing the operation (information processing method) of the side receiving the transfer request in the remote control system 1 according to this modification. In this modification, an example will be described in which supplemental information for determining whether or not to accept the transfer request is presented to the operator 50 receiving the transfer request. Note that each process shown in FIG. 22 may be executed by any processing unit of the terminal 10.
[0211] 22 , the terminal 10 of the operator A receives a transfer request notification from another operator 50 or the automated driving system 46 (i.e., the mobile object 40) (S61). The transfer request notification may include, for example, the mobile object ID of the mobile object 40 requesting the transfer.
[0212] Next, the terminal 10 acquires driving task data for the mobile object 40 for which the transfer request has been received from the driving task information DB 32 of the storage device 30 (S71). The driving task data here indicates the current status of multiple driving tasks on the driving route of the mobile object 40, when the driving route of the mobile object 40 is set in advance.
[0213] FIG. 23 is a diagram showing driving task data according to this modified example.
[0214] As shown in FIG. 23, the running task data includes a sequence, a running task type, and a state.
[0215] The order indicates the order in which the driving tasks occur, with driving tasks occurring in the order of 1 to 5.
[0216] The driving task type indicates what type of driving task will occur, including, but not limited to, following the road, crossing a pedestrian crossing, driving in an area where pedestrians jump out, and driving at an intersection.
[0217] The status indicates the current status of each traveling task, and includes completed, running, not running, etc.
[0218] In the example of Figure 23, the moving body 40 is currently performing (driving) the pedestrian jumping out section driving of order 4, has not yet performed the intersection driving of order 5, and will perform the intersection driving after the pedestrian jumping out section driving.
[0219] Referring again to FIG. 22, next, the terminal 10 extracts monitoring / operation items that will occur when a transfer request is received based on the driving task data acquired in step S71 and the manual data acquired from the manual information DB 33 (S72).
[0220] FIG. 24 is a diagram showing manual data according to this modification.
[0221] 24, the manual data is a table in which travel task types, monitoring items, and operation items are associated with each other. The monitoring items indicate targets that the operator 50 must monitor for each travel task type, and include, but are not limited to, no monitoring required (no target to monitor), pedestrians ahead, approaching vehicles, etc. The operation items indicate operations that the operator 50 must perform for each travel task type, and include, but are not limited to, no monitoring required (no operation required), start travel, stop travel in a dangerous location, remote operation, etc.
[0222] The terminal 10 extracts an ongoing driving task included in the driving task data, and extracts monitoring items and operation items corresponding to the extracted driving task from the manual data. In the examples of Figures 23 and 24, "driving in an area where a pedestrian has jumped out" is extracted as the ongoing driving task, "pedestrians ahead" is extracted as the monitoring item corresponding to the extracted driving task, and "stopping driving in case of danger" is extracted as the operation item.
[0223] In addition, the terminal 10 may extract unexecuted driving tasks instead of or in addition to the driving tasks being executed, and extract monitoring items and operation items for the unexecuted driving tasks.
[0224] Next, the terminal 10 displays information based on the monitoring / operation items extracted in step S73 on the display unit 110a (S73).
[0225] FIG. 25 is a diagram showing a screen displayed on the display unit 110a on the side receiving the assistance request in the remote control system 1 according to this modified example.
[0226] 25 , an auxiliary display h including the items extracted in step S72 is displayed as a takeover request, superimposed on an area displaying an image of the periphery of the mobile object 40. Note that the auxiliary display h may be displayed, for example, in an area that does not overlap with the area displaying the image. The transfer request response input unit 211 also displays button displays e12 and e13 as response buttons when a takeover request is accepted from another operator 50. For example, the button display e12 displays "Accept takeover request," indicating that the transfer request (takeover request) is accepted, and the button display e13 displays "Reject takeover request," indicating that the transfer request is not accepted.
[0227] The operator 50 can refer to the auxiliary display h displayed as auxiliary information to determine whether or not to accept the transfer request, thereby making it possible to make a more accurate determination.
[0228] In this manner, in this modified example, when a transfer request is received, the details of the remote control (remote assistance or remote operation) of the mobile body 40 are presented to the other operator 50, and after the details are presented, an operation to decide whether or not to accept the transfer request is obtained from the other operator 50.
[0229] In the above, an example has been described in which the terminal 10 extracts both monitoring items and operation items using driving task data and manual data, but this is not limited to this, and it is sufficient if at least one of the monitoring items and operation items is extracted and presented.
[0230] Next, the transfer request response input unit 211 determines whether or not the acceptance of the request for transfer of control responsibility has been accepted (S74). If the transfer request response input unit 211 receives a notification from the terminal 10 of the other operator 50 indicating that the button display e12 has been operated by the other operator 50, the transfer request response input unit 211 determines that the acceptance has been accepted, and if the transfer request response input unit 211 receives a notification from the terminal 10 of the other operator 50 indicating that the button display e13 has been operated by the other operator 50, or if there has been no button operation for a predetermined period of time, the transfer request response input unit 211 determines that the acceptance has been rejected. If the determination in step S74 is Yes, the process proceeds to step S23, and if the determination in step S74 is No, the process ends.
[0231] (Sixth Modification of the Embodiment) A remote control system 1 according to this modification will be described below with reference to Figures 26 to 29. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. This modification describes a terminal 10 that can present information that can assist an operator 50 who has received a transfer request in determining whether or not to accept the transfer request when the travel route of the mobile object 40 has not been set in advance.
[0232] FIG. 26 is a flowchart showing the operation (information processing method) of the assistance requesting side in the remote control system 1 according to this modified example.
[0233] As shown in Figure 26, when operator B's terminal 10 receives a transfer request notification from operator A's terminal 10 (S61), it acquires mobile object information from the mobile object information transmission unit 127 of the control server 20 and acquires surrounding environment information from the surrounding environment information transmission unit 128 of the control server 20 (S81).
[0234] 27 is a diagram showing moving body information according to this modification, in which (a) and (b) of FIG. 27 show moving body information for two moving bodies 40.
[0235] As shown in Fig. 27, the mobile object information includes items such as a mobile object ID, a state, a speed, and a position. Fig. 27(a) shows an example in which a mobile object 40 with a mobile object ID of "101" is traveling at a speed of 5 km / h on a crosswalk (longitude 35.681731°, longitude: 139.763495°). Fig. 27(b) shows an example in which a mobile object 40 with a mobile object ID of "102" is traveling at a speed of 0 km / h and is stopped on a sidewalk (longitude 35.681386°, longitude: 139.763369°).
[0236] Fig. 28 is a diagram showing surrounding environment information according to this modification, in which (a) and (b) of Fig. 28 show surrounding environment information for two moving bodies 40.
[0237] As shown in Fig. 28, the surrounding environment information includes items such as a moving object ID, a surrounding environment state, and the number of detected objects. Fig. 28(a) shows an example in which two vehicles are approaching a moving object 40 with a moving object ID of "101." Fig. 28(b) shows an example in which two pedestrians are approaching a moving object 40 with a moving object ID of "102."
[0238] Referring again to FIG. 26, next, the terminal 10 extracts monitoring / operation items that will occur when a transfer request is received based on the mobile object information and surrounding environment information acquired in step S81 and the manual data acquired from the manual information DB 33 (S72).
[0239] FIG. 29 is a diagram showing manual data according to this modification.
[0240] As shown in FIG. 29, the manual data is a table in which vehicle conditions, surrounding environmental conditions, monitoring items, and operation items are associated with each other.
[0241] In the example of Figure 29, when the moving body 40 is traveling and a vehicle is approaching, the monitoring item is "approaching vehicle" and the operation item is "stop traveling in case of danger" operation.
[0242] In step S72, the terminal 10 extracts monitoring items and operation items corresponding to the state of the mobile body 40 included in the mobile body information and the surrounding environment state of the mobile body 40 included in the surrounding environment information. For example, for a mobile body 40 with a mobile body ID of "101", the terminal 10 extracts that the monitoring item is an approaching vehicle and the operation item is to stop driving in case of danger, and for a mobile body 40 with a mobile body ID of "102", the terminal 10 extracts that the monitoring item is a pedestrian ahead and the operation item is to start driving.
[0243] As a result, even if the mobile body 40 does not have a pre-set driving route, the terminal 10 can present information to assist the operator 50 who has received a transfer request in determining whether or not to accept the transfer request, based on information about the current mobile body 40 or the surrounding area of the mobile body 40.
[0244] In the above, an example has been described in which the terminal 10 extracts both monitoring items and operation items using both mobile body information and surrounding environment information, but this is not limited to this, and it is sufficient that at least one of the monitoring items and operation items is extracted and presented based on at least one of the mobile body information and surrounding environment information.
[0245] (Seventh Modification of the Embodiment) The remote control system 1 according to this modification will be described below with reference to Figures 30 to 34. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. In this modification, a case will be described in which the control server 20 narrows down the operators 50 to whom a transfer request is made.
[0246] 30 is a flowchart showing the operation (information processing method) of the control server 20 in the remote control system 1 according to this modification. Note that each process shown in FIG. 30 may be executed by any processing unit of the control server 20.
[0247] As shown in FIG. 30, the control server 20 receives a transfer request notification from the terminal 10 (S61a).
[0248] Next, the control server 20 acquires driving task data for the mobile object 40 for which the transfer request has been received from the driving task information DB 32 in the storage device 30 (S71a). The driving task data here indicates the current status of multiple driving tasks on the driving route of the mobile object 40, when the driving route is set in advance.
[0249] FIG. 31 is a diagram showing driving task data according to this modified example.
[0250] As shown in Fig. 31, the traveling task data includes a sequence, a traveling task type, and a status, similar to Fig. 23. The example in Fig. 31 indicates that the moving object 40 is currently performing the following road traveling of sequence 3.
[0251] 30 again, next, the transfer request recipient determination unit 123 executes the processes of steps S91 to S93 for each of the target transfer recipient operators 50. The target transfer recipient operators 50 may be all of the other operators 50 except for the operator 50 of the terminal 10 that sent the transfer request notification.
[0252] The transfer request destination determination unit 123 acquires skill information indicating the remote control skills of the operator 50 from the operator qualification information DB 34 in the storage device 30 (S91).
[0253] Fig. 32 is a diagram showing the skill information of the operator 50 according to this modified example. Fig. 33 is a diagram showing the skill level information of the operator 50 according to this modified example.
[0254] As shown in Fig. 32, the skill information of the operators 50 indicates the skills of each operator 50. In the example of Fig. 32, the skill (qualification) of operator A is "verification skill," and the skill of operator B is "advanced driving skill."
[0255] As shown in Fig. 33, the skill level information of the operator 50 includes a level and a skill. In the example of Fig. 33, the skill corresponding to level "3" is an advanced driving skill, the skill corresponding to level "2" is a driving skill, and the skill corresponding to level "1" is a confirmation skill. The level (skill) of each of the multiple operators 50 among the levels (skills) shown in Fig. 33 is acquired in advance, and the skill information shown in Fig. 32 is generated.
[0256] 30 again, next, the transfer request recipient determination unit 123 determines whether or not the skill level of the operator 50 is sufficient (S92). Whether or not the skill level of the operator 50 is sufficient means, for example, whether or not the operator 50 has the skill corresponding to the remote control (remote assistance or remote operation) of the mobile body 40 that is the subject of the transfer request.
[0257] FIG. 34 is a diagram showing required skill information according to this modification.
[0258] As shown in Fig. 34, the required skill information is a table in which travel tasks are associated with required skills. In the example of Fig. 34, the required skill when the travel task type is road following is driving skill (level 2), the required skill when the travel task type is narrow road traveling is advanced driving skill (level 3), and the required skill when the travel task type is other travel task type is checking skill (level 1).
[0259] The transfer request recipient determination unit 123 extracts at least one of an ongoing and an unexecuted driving task based on the driving task data, and identifies a required skill for executing the driving task based on the extracted driving task and the required skill information. For example, if following a road is extracted as an ongoing driving task based on the driving task data, the required skill is identified as a driving skill based on the required skill information.
[0260] The transfer request recipient determination unit 123 executes the determination of step S92 by determining whether the skill of the operator 50 based on the skill information is equal to or higher than the specified skill. When the specified skill is a driving skill, the transfer request recipient determination unit 123 determines that the skill level of the operator 50 is sufficient if the skill of the operator 50 based on the skill information is equal to or higher than the driving skill (e.g., level 2 or higher), and determines that the skill level of the operator 50 is insufficient if the skill of the operator 50 based on the skill information is lower than the driving skill (e.g., level 1).
[0261] Next, if the transfer request recipient determination unit 123 determines that the skill level of the operator 50 is sufficient (Yes in S92), it notifies (e.g., transfers) a transfer request to the terminal 10 of the operator 50 whose skill level is determined to be sufficient (S93). In other words, if it is determined that the operator 50 has the skills required for remote control of the mobile object 40 to be transferred, a transfer request to transfer responsibility for the mobile object 40 is notified to the operator 50 (e.g., the operator 50 receives the transfer request).
[0262] In addition, if the transfer request recipient determination unit 123 determines that the skill level of the operator 50 is insufficient (No in S92), it does not notify the transfer request to the terminal 10 of the operator 50 whose skill level is determined to be insufficient.
[0263] This allows a request for transfer to be notified to an operator 50 who can execute the driving task. For example, when an operation requires a high level of skill, it is possible to prevent remote control from being handed over to an operator 50 with low skill.
[0264] (Variation 8 of the Embodiment) The remote control system 1 according to this variation will be described below with reference to Figures 35 to 38. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. In this variation, a case will be described in which control responsibility is transferred from an automated driving system 46 installed in a moving object 40 to an operator 50.
[0265] FIG. 35 is a flowchart showing the operation (information processing method) of the automatic driving system 46 in the remote control system 1 according to this modified example.
[0266] As shown in Fig. 35 , the autonomous driving system 46 acquires the next driving task (S101). When driving task data is stored in the mobile object 40, the autonomous driving system 46 acquires the next driving task from the driving task data. For example, in the driving task data shown in Fig. 31 , driving in a section where a pedestrian has entered is acquired as the next driving task. Note that in step S101, all remaining driving tasks (driving in a section where a pedestrian has entered and driving through an intersection in the case of Fig. 31 ) may be acquired in addition to or instead of the next driving task.
[0267] Next, the automatic driving system 46 acquires the autonomous driving ODD information from the storage device 30 (S102). The autonomous driving ODD information may be created for each type of moving body 40, for example.
[0268] FIG. 36 is a diagram showing autonomous driving ODD information according to this modification.
[0269] As shown in FIG. 36 , the autonomous driving ODD information is a table in which driving task types are associated with driving conditions. The driving conditions indicate conditions that require remote control operation for the mobile object 40 to drive autonomously. Since road-following driving and narrow road driving do not have any driving conditions, the operator 50 can drive autonomously without remote control operation. For example, autonomous driving is possible with just remote monitoring by the operator 50. Since the driving conditions for crosswalk driving and intersection driving are remote assistance, autonomous driving is possible with remote assistance from the operator 50. Since the driving condition for pedestrian jumping-out section driving is remote operation, autonomous driving is possible with remote operation by the operator 50.
[0270] Next, the autonomous driving system 46 determines whether or not remote control or remote assistance is included in the driving conditions of the mobile object 40 (S103). For example, the autonomous driving system 46 identifies driving conditions corresponding to the next driving task from the autonomous driving ODD information, and determines whether or not remote control or remote assistance is included in the identified driving conditions.
[0271] Next, if the automated driving system 46 determines that remote operation or remote assistance is required (Yes in S103), it issues a notification of a responsibility transfer request (S104). The automated driving system 46 requests the control server 20 to transfer control responsibility. In this way, the transfer request to the operator 50 is notified from the automated driving system 46. Note that if remote assistance is required, a notification of a remote assistance request is issued as a responsibility transfer request, and if remote operation is required, a notification of a remote operation request is issued as a responsibility transfer request.
[0272] Next, the automated driving system 46 determines whether the operator 50 has accepted responsibility (accepted the responsibility transfer request) before the transition to the next driving task (S105). If the automated driving system 46 receives a response from the terminal 10 or the control server 20 acknowledging the responsibility transfer request, the automated driving system 46 determines "Yes" in step S105.
[0273] If the automated driving system 46 determines that the operator 50 has accepted responsibility before the transition to the next driving task (Yes in S105), it transfers control responsibility to the operator 50 who has accepted responsibility (S106). For example, if the automated driving system 46 determines that the operator 50 has accepted responsibility, it transmits information indicating (where the responsibility for) control of the mobile object 40 and information indicating (the granting of) operation authority for the mobile object 40 to the terminal 10 of the operator 50 who has accepted responsibility. Furthermore, if it is determined that the operator 50 has not accepted responsibility before the transition to the next driving task (No in S105), the MRM execution unit 41 executes MRM (S107).
[0274] Furthermore, if the automatic driving system 46 determines that there is no remote operation or remote assistance (No in S103), it continues the autonomous driving.
[0275] Next, the operation of the terminal 10 receiving the assistance request (control assistance request) will be described with reference to FIGS. 37 and 38 . FIG. 37 is a flowchart showing the operation (information processing method) of the terminal receiving the assistance request in the remote control system 1 according to this modification. FIG. 38 is a diagram showing a screen displayed on the display unit 110a of the terminal receiving the assistance request in the remote control system 1 according to this modification. Assume that the operator 50 receiving the assistance request is remotely monitoring the mobile object 40 that issued the assistance request (issued the notification of the responsibility transfer request in step S104). Therefore, as shown in FIG. 38 (a), the control status display unit 111 displays "Autonomous Driving" as the status display b13 indicating that the mobile object 40 is autonomously driving. Furthermore, the start / end input unit 215 displays "Start Control" as a button display e14 (operation unit) for starting control. The button display e14 is an example of information (display) indicating the start of remote control.
[0276] Note that the button display e14 may be set to be inoperable unless a predetermined condition is met. For example, the button display e14 may be enabled when the remote control skills of the operator 50 satisfy the remote control skills required to continue the autonomous traveling of the moving body 40 for which remote control is to be started. Here, being enabled means that the function of the button display e14 is enabled, and may also mean, for example, that the button display e14 can be operated.
[0277] As shown in FIG. 37, the terminal 10 receives an assistance / operation request notification (for example, a notification of a responsibility transfer request in step S104) (S111).
[0278] As shown in (b) of Figure 38, when an assistance / operation request notification is received, the transfer request presentation unit 112 switches the display from status display b13 to status display b14 indicating that a transfer request has been notified, which is "Control request (autonomous driving)."
[0279] 37, the terminal 10 determines whether the operator 50 has accepted the request (S112). When the start / end input unit 215 detects that the button display e14 has been pressed, it determines that the operator 50 has accepted the request.
[0280] Next, when it is determined that the operator 50 has accepted the request (Yes in S112), the start / end input unit 215 notifies the operator 50 of the start of the control responsibility state (S23) and transitions to the control responsibility state (S24). In this way, based on the operation of the button display e14 indicating the start of remote control, the responsibility for the moving object 40 and the authority to operate the moving object 40 are transferred to the operator 50 who operated the button display e14.
[0281] 38(c), when the operator 50 accepts the request for assistance, the control status presentation unit 111 switches the display from the status display b14 to "assisting," which is the status display b1 indicating that the operator 50 has control responsibility. This notifies the operator 50 that control responsibility has been transferred and that the operator 50 has control responsibility.
[0282] The transfer request input unit 212 displays a button display e1, the start / end input unit 215 displays a button display e2, and the operation signal transmission unit 216 displays an operation display a2 corresponding to air traffic control assistance.
[0283] Furthermore, if the start / end input unit 215 determines that the acceptance has not been approved (No in S112), it ends the process.
[0284] (Ninth Variation of the Embodiment) A remote control system 1 according to this variation will be described below with reference to FIGS. 39 to 41. The following description will focus on differences from the embodiment, and will omit or simplify descriptions of content that is the same as or similar to the embodiment. This variation describes a case in which control responsibility is transferred from an operator 50 to an automated driving system 46 installed in a mobile object 40. FIG. 40 is a diagram showing a screen displayed on the display unit 110a on the side requesting responsibility transfer in the remote control system 1 according to this variation.
[0285] FIG. 39 is a flowchart showing the operation (information processing method) of the side requesting responsibility transfer in the remote control system 1 according to this modified example.
[0286] 39 , when the transfer request input unit 212 receives a responsibility transfer request in step S11, it issues a responsibility transfer notification to the mobile object 40 (e.g., the automated driving system 46) (S12). Issuing the notification means transmitting the responsibility transfer notification to the mobile object 40.
[0287] As shown in (b) of FIG. 40, the control status presentation unit 111 switches the display from the status display b1 to "Control termination request (assisting)", which is a status display b15 indicating that a request for transfer of control responsibility is being made. In other words, the status display b15 is presented to the operator A. This notifies the operator A that a request for transfer of control responsibility is being made. Note that the status display b15 may include information indicating that a transfer request has been made to the automated driving system 46.
[0288] 39 , next, the transfer request input unit 212 determines whether the mobile object 40 (e.g., the automated driving system 46) has accepted control responsibility (S121). When the transfer request input unit 212 acquires (e.g., receives) a notification from the mobile object 40 indicating that the transfer request is accepted, the transfer request input unit 212 determines that the mobile object 40 has accepted control responsibility. The notification is an example of information indicating whether the transfer request is accepted.
[0289] Next, if the transfer request input unit 212 determines that the mobile object 40 has accepted control responsibility (Yes in S121), it releases the control responsibility state of Operator A (S14) and notifies Operator A of the release of the control responsibility state (S15). Furthermore, if it is determined that the mobile object 40 has accepted control responsibility, the terminal 10 of Operator A transmits to the mobile object 40 information indicating (where) the control responsibility of the mobile object 40 is located and information indicating (the granting of) the operation authority for the mobile object 40. The information indicating (where) the control responsibility is located includes information indicating that the control responsibility will be transferred from Operator A to the automated driving system 46, and the information indicating (the granting of) the operation authority includes information indicating that the operation authority will be transferred from Operator A to the automated driving system 46.
[0290] As shown in (c) of FIG. 40 , the control status presentation unit 111 switches the display from the status display b15 to "Autonomously Driving," which is the status display b13 indicating that the moving object 40 is autonomously driving. This notifies the operator A that the control responsibility state has been released and that control responsibility has been transferred to the automated driving system 46. At this time, the operation display a2 is switched to inactive (e.g., hidden). This prevents the operator A from operating the moving object 40. In other words, the operation authority for the moving object 40 is transferred. For example, the remote control mode of the operator A switches from remote assistance to remote monitoring. Furthermore, the transfer request input unit 212 switches from the button display e2 to display a button display e14 as a response button when a transfer request is received from the moving object 40 (e.g., the automated driving system 46).
[0291] In this way, the fact that responsibility for the mobile body 40 has been transferred to the automatic driving system 46 of the mobile body 40 and the fact that operation authority for the mobile body 40 has been transferred to the automatic driving system 46 of the mobile body 40 are presented to operator A in different ways.
[0292] Furthermore, when the control status presentation unit 111 acquires (for example, receives) a notification from the mobile object 40 indicating that it refuses to accept the transfer request, it determines that the mobile object 40 has refused control responsibility (No in S121) and notifies the operator A that the release of the control responsibility state has failed (S16). In this case, the operator A will continue to provide remote assistance to the mobile object 40, for example.
[0293] Next, the operation of the mobile body 40 (automated driving system 46) that receives the transfer request will be described with reference to Fig. 41. Fig. 41 is a flowchart showing the operation of the automatic driving system 46 in the remote control system 1 according to this modified example.
[0294] As shown in FIG. 41, the automatic driving system 46 receives a responsibility transfer request notification from the terminal 10 of the operator A (S111).
[0295] Next, the automatic driving system 46 performs the processes of steps S101 to S103 for a certain period of time.
[0296] The automatic driving system 46 acquires the next traveling task (S101). If traveling task data (see FIG. 31, for example) is stored in the moving object 40, the automatic driving system 46 receives the traveling task data and identifies the next traveling task from the received traveling task data.
[0297] Next, the autonomous driving system 46 acquires the autonomous driving ODD information from the storage device 30 (S102). The autonomous driving ODD information may be created for each type of moving body 40. The autonomous driving system 46 receives, for example, the autonomous driving ODD information shown in FIG. 36 .
[0298] The autonomous driving system 46 determines whether or not remote control or remote assistance is included in the driving conditions of the mobile object 40 (S103). For example, the autonomous driving system 46 identifies driving conditions corresponding to the next driving task from the autonomous driving ODD information, and determines whether or not remote control or remote assistance is included in the identified driving conditions.
[0299] Next, if the automated driving system 46 determines that remote operation or remote assistance is present (Yes in S103), it terminates the processing. In this case, the automated driving system 46 may, for example, send a notification to the terminal 10 of operator A indicating that acceptance of the responsibility transfer request is rejected. Alternatively, if the automated driving system 46 determines that remote operation or remote assistance is not present (No in S103), it notifies the terminal 10 of operator A that the responsibility transfer request has been accepted (S131) and initiates a control responsibility state (S132). The automated driving system 46 may send a notification to the terminal 10 of operator A indicating that the responsibility transfer request has been accepted, and may execute step S132 upon receiving information indicating (the location of) control responsibility for the mobile object 40 and information indicating (the granting of) operation authority for the mobile object 40 from the terminal 10 of operator A.
[0300] (Variation 10 of the Embodiment) The remote control system 1 according to this variation will be described below with reference to Figures 42 to 44. The following description will focus on differences from the embodiment, and description of content that is the same as or similar to the embodiment will be omitted or simplified. In this variation, various operation examples will be described for transitioning the mobile object 40 to a safe state when the operator 50 in the remote control system 1 is unable to assume control responsibility (for example, inability to monitor / operate, deterioration of control quality).
[0301] Fig. 42 is a sequence diagram showing a first example of the operation when the remote control system 1 according to this modified example is unable to fulfill its control responsibility. In Fig. 42, each pattern in which MRM is executed is shown enclosed in a dashed frame. For convenience, Fig. 42 shows four patterns in which MRM is executed. There is no temporal relationship between the four patterns.
[0302] 42, in the first pattern, the communication information acquisition unit 130 of the control server 20 first acquires the communication status from the terminal 10 (S211). The communication information acquisition unit 130 may acquire, for example, a malfunction of the communication device of the terminal 10, a communication delay in communication between the terminal 10 and the control server 20, or the like as the communication status.
[0303] Next, when the MRM instruction unit 131 detects a communication abnormality based on the communication state (S212), it generates an MRM instruction to cause the mobile body 40 to execute MRM, and the operation signal transmission unit 129 transmits the generated MRM instruction to the mobile body 40 (S213).
[0304] When the MRM execution unit 41 of the mobile object 40 acquires the MRM instruction from the MRM instruction unit 131, it executes MRM (S214) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S215). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0305] When the control server 20 acquires the MRM information from the mobile object 40, the control status management unit 121 releases the control responsibility of the operator 50 who had control responsibility for the mobile object 40 (S216). The control status management unit 121 may, for example, notify the terminal 10 of the operator 50 that the control responsibility has been released.
[0306] In this way, in the first pattern, if a communication abnormality occurs between the terminal 10 and the control server 20, the mobile body 40 can be made to execute MRM and the control responsibility for the mobile body 40 can be released.
[0307] Next, in the second pattern, the terminal monitoring unit 120 of the terminal 10 first monitors whether or not a time jump has occurred in the terminal 10 (S221).
[0308] Next, if the terminal monitoring unit 120 detects that a time jump has occurred (S222), the MRM instruction unit 118 generates an MRM instruction for the mobile body 40, and the operation signal transmission unit 216 transmits the generated MRM instruction to the mobile body 40 (S223).
[0309] When the MRM execution unit 41 of the mobile object 40 acquires the MRM instruction from the operation signal transmission unit 216, it executes the MRM (S224) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S225). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0310] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S226).
[0311] In this way, in the second pattern, when a time jump occurs in the terminal 10, the mobile unit 40 can be made to execute MRM and the control responsibility for the mobile unit 40 can be released.
[0312] Next, in the third pattern, first, the communication information acquisition unit 44 of the mobile object 40 acquires the communication status from the control server 20 (S231). The communication information acquisition unit 44 may acquire, for example, a communication delay in the communication between the control server 20 and the mobile object 40 as the communication status.
[0313] Next, when a communication abnormality is detected based on the communication state (S232), the MRM execution unit 41 executes MRM (S233) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S234). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0314] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S235).
[0315] In this way, in the third pattern, if a communication abnormality occurs between the control server 20 and the mobile body 40, the mobile body 40 can execute MRM and release the control responsibility for the mobile body 40.
[0316] Next, in the fourth pattern, the moving object monitoring unit 45 first monitors whether a time jump has occurred within the moving object 40 (S241).
[0317] Next, when the mobile object monitoring unit 45 detects that a time jump has occurred (S242), the MRM execution unit 41 executes MRM (S243) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S244). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0318] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S245).
[0319] In this way, in the fourth pattern, when a time jump occurs in the moving body 40, the moving body 40 can execute MRM and can release control responsibility for the moving body 40.
[0320] Fig. 43 is a sequence diagram showing a second example of the operation when the remote control system 1 according to this modification is unable to fulfill its control responsibility. For convenience, Fig. 43 also shows four patterns for executing MRM. There is no temporal relationship between the four patterns.
[0321] The remote control system 1 further includes an input device 60. The input device 60 is a device used by the operator 50 to remotely control the mobile object 40, such as a device for operating a steering wheel, a brake, and the like.
[0322] 43 , in the fifth pattern, the mobile object 40 first acquires surrounding environment information and assigns the current time (time A) to the acquired surrounding environment information (S250). For example, the surrounding environment information may be acquired via the surrounding environment information transmission unit 128 of the control server 20. Then, the mobile object 40 transmits the surrounding environment information with the current time assigned to the terminal 10.
[0323] Next, the terminal 10 acquires surrounding environment information from the mobile object 40 (S251).
[0324] Next, the terminal communication information acquisition unit 117 of the terminal 10 determines whether or not a delay occurs in the surrounding environment information based on the acquired surrounding environment information (S252). The terminal communication information acquisition unit 117 may perform the determination in step S252 based on, for example, whether or not the following formula 1 is satisfied:
[0325] Current time - Latest time > Default value (Formula 1)
[0326] The latest time is the time when the latest surrounding environment information was acquired. If Expression 1 is satisfied, the terminal communication information acquisition unit 117 determines that a delay has occurred, and if Expression 1 is not satisfied, it determines that a delay has not occurred.
[0327] When the terminal communication information acquisition unit 117 detects a delay in the surrounding environment information (S253), the MRM instruction unit 118 generates an MRM instruction for the mobile body 40, and the operation signal transmission unit 216 transmits the generated MRM instruction to the mobile body 40 (S254).
[0328] When the MRM execution unit 41 of the mobile object 40 acquires the MRM instruction from the operation signal transmission unit 216, it executes the MRM (S255) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S256). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0329] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S257).
[0330] In this way, in the fifth pattern, if there is a delay in the surrounding environment information acquired by the terminal 10, the mobile body 40 can be made to execute MRM and the control responsibility for the mobile body 40 can be released.
[0331] Next, in the sixth pattern, first, the input device status acquisition unit 218 of the terminal 10 acquires alive information from the input device 60 (S261). The alive information is information indicating the status of the input device 60, and if an abnormality occurs in the input device 60, includes information indicating that the abnormality has occurred.
[0332] If the input device status acquisition unit 218 detects an abnormality in the input device 60 (S262), the MRM instruction unit 219 generates an MRM instruction for the mobile body 40, and the operation signal transmission unit 216 transmits the generated MRM instruction to the mobile body 40 (S263).
[0333] When the MRM execution unit 41 of the mobile object 40 acquires the MRM instruction from the operation signal transmission unit 216, it executes the MRM (S264) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S265). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0334] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S266).
[0335] In this way, in the sixth pattern, when an abnormality occurs in the input device 60, the MRM can be executed for the moving body 40, and the control responsibility for the moving body 40 can be released.
[0336] Next, in the seventh pattern, the operation signal transmitting unit 216 of the terminal 10 first acquires an operation signal from the input device 60 (S271), assigns time A (time stamp) to the operation signal (S272), and transmits the operation signal with time A assigned to the mobile object 40 (S273).
[0337] When the communication information acquisition unit 44 of the mobile object 40 acquires the operation signal, it stores the reception time (time B) that is the time when the operation signal was acquired (S274), and determines whether or not a delay has occurred in the operation signal (S275). The communication information acquisition unit 44 may make the determination in step S275 based on, for example, whether or not the following formula 2 is satisfied:
[0338] Current time - Time A > Specified value (Formula 2)
[0339] When the communication information acquisition unit 44 detects a delay in the operation signal (S276), the MRM execution unit 41 executes MRM (S277) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S278). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0340] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S279).
[0341] Thus, in the seventh pattern, if the delay in transmitting an operation signal from the terminal 10 to the mobile body 40 is longer than a specified value, the mobile body 40 can be made to execute MRM and the control responsibility for the mobile body 40 can be released.
[0342] In the eighth pattern, after time B is held in step S274, the process waits for the next operation signal (S281). During the wait, the communication information acquisition unit 44 determines whether or not a delay in the operation signal has occurred (S282). The communication information acquisition unit 44 may perform the determination in step S282 based on, for example, whether or not the following formula 3 is satisfied:
[0343] Current time - Time B > Specified value (Equation 3)
[0344] That is, the communication information acquisition unit 44 determines whether a time longer than a specified value has elapsed since the previous acquisition of the operation signal. If the left side of Equation 3 is greater than the specified value, the communication information acquisition unit 44 determines that a delay in the operation signal has occurred.
[0345] When the communication information acquisition unit 44 detects a delay in the operation signal (S283), the MRM execution unit 41 executes MRM (S284) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S285). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0346] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S286).
[0347] In this way, in the eighth pattern, if the time interval for acquiring an operation signal at the mobile body 40 becomes longer than a specified value, MRM can be executed on the mobile body 40 and control responsibility for the mobile body 40 can be released.
[0348] Figure 44 is a sequence diagram showing a third example of the operation when the remote control system 1 according to this modification is unable to fulfill its control responsibility. The operations of steps S250 to S273 are the same as those in Figure 43, and therefore description thereof will be omitted. It is also assumed that the operation signal of step S273 is continuously transmitted from the terminal 10 to the mobile object 40 (for example, at predetermined time intervals). The operations of steps S250 to S273 and the operations of steps S291 to S296 are executed in parallel, for example (see "per" in Figure 44).
[0349] As shown in FIG. 44, in the ninth pattern, the remote control system 1 repeatedly executes the following process until the remote operation ends (S291).
[0350] The communication information acquisition unit 44 determines whether a delay occurs in the operation signal (S292). The communication information acquisition unit 44 may make the determination in step S292 based on whether the above formula 2 is satisfied, for example.
[0351] When the communication information acquisition unit 44 detects a delay in the operation signal (S293), the MRM execution unit 41 executes MRM (S294) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S295). The mobile object information transmission unit 127 of the control server 20 acquires the MRM information.
[0352] When the control status management unit 121 of the control server 20 acquires the MRM information from the moving object 40, it releases the control responsibility of the operator 50 who had control responsibility for the moving object 40 (S296).
[0353] In this way, in the ninth pattern, if the operation signal is continuously transmitted, it is possible to determine the delay of the operation signal in one determination process, thereby reducing the amount of processing required for the determination process.
[0354] As described above, MRM may be executed when a communication abnormality occurs or delay cannot be measured correctly between the terminal 10, the control server 20, and the mobile body 40. Furthermore, MRM may be executed when the time from acquiring information from the mobile body 40 to presenting the surrounding environment information and mobile body information of the mobile body 40 to the operator 50, and the time from operation by the operator 50 to reflection in the mobile body 40, cannot be made shorter than a certain time. Furthermore, MRM may be executed when an abnormality is detected in the remote control system 1 including the terminal 10, the control server 20, and the mobile body 40, as described above.
[0355] (Eleventh Variation of the Embodiment) The following describes a remote control system 1 according to this variation, with reference to Figures 45 to 50. The following description will focus on differences from the embodiment, and will omit or simplify descriptions of content that is the same as or similar to the embodiment. In this variation, a case will be described in which the administrator (system administrator) of the remote control system 1 sets the operator who will make a transfer request and the operator who will receive the transfer request.
[0356] The administrator is a person who manages multiple operators 50. The administrator manages the remote control status of the multiple operators 50. The administrator may be, for example, a person who does not actually perform remote control. The administrator may also be, for example, located in a remote control center where the multiple operators 50 are located, or may be located in a facility away from the remote control center.
[0357] Fig. 45 is a flowchart showing the operation (information processing method) of the remote control system 1 according to this modified example. Fig. 45 shows a flowchart for the case where operation is performed upon receiving input from an administrator. The operation shown in Fig. 45 is executed by, for example, the control server 20, but is not limited to this. Note that an information processing device, input device, and display device (not shown) used by the administrator are communicatively connected to the control server 20. The information processing device is, for example, a PC, the input device is, for example, a keyboard, mouse, touch panel, sound collection device, etc., and is connected to the information processing device, and the display device is, for example, a liquid crystal display device, etc., and is connected to the information processing device.
[0358] 45, the control server 20 receives a responsibility transfer request setting from an administrator (S301). The control server 20 acquires, for example, via communication, information indicating the responsibility transfer request setting entered by the administrator into an input device. The information includes information indicating the operator 50 making the responsibility transfer request and information indicating the operator 50 receiving the responsibility transfer request.
[0359] Fig. 46 is a diagram showing a screen displayed on the display unit 300 of the administrator in the remote control system 1 according to this modification. Fig. 46 shows a screen in which the first display h1 in the screen shown in Fig. 18 has been changed to a fifth display h5.
[0360] The fifth display h5 is displayed on each of the multiple moving bodies 40, and when pressed by an administrator, a process is executed to transfer control responsibility for the moving body 40 corresponding to the pressed fifth display h5 to another operator 50. For example, when the fifth display h5 of a moving body 40 whose moving body ID is "X9" is pressed, a process is executed to transfer control responsibility for the moving body 40 from operator B to another operator 50.
[0361] When the fifth display h5 is pressed, the display unit 300 may display information for selecting a transferee operator 50. The information may include a list of the other operators 50, or may include the remote control status of the other operators 50.
[0362] Hereinafter, the operator 50 from which the data is transferred will also be simply referred to as the operator 50, and the operator 50 to which the data is transferred will also be referred to as the other operator 50.
[0363] 45 again, next, the control server 20 issues a responsibility transfer request notice (S302). The transfer request notifier 122 of the control server 20 transmits a responsibility transfer request notice to the terminal 10 of the operator 50 making the responsibility transfer request, which has been set by the administrator. The transfer request notifier 122 also transmits a responsibility transfer request notice to the terminal 10 of the operator 50 (the other operator 50) receiving the responsibility transfer request, which has been set by the administrator, to take over the control responsibility from the current operator 50.
[0364] Next, the operation of the terminal 10 requesting responsibility transfer will be described with reference to Figs. 47 and 48. Fig. 47 is a flowchart showing the operation (information processing method) of the side requesting responsibility transfer in the remote control system 1 according to this modification. Fig. 47 shows the operation executed by the terminal 10 possessed by the operator 50 when transferring the control responsibility of the operator 50 who is currently remotely assisting the moving body 40. Fig. 48 is a diagram showing a screen displayed on the display unit 110a of the side requesting responsibility transfer in the remote control system 1 according to this modification. Fig. 48 shows an example of a screen displayed on the display unit 110a to be confirmed by the operator 50.
[0365] As shown in (a) of Figure 48, on the display unit 110a, the control status presentation unit 111 displays the status display b1 "assisting", which indicates that the operator 50 is responsible for control, and the start / end input unit 215 displays the button display e2 for ending control.
[0366] As shown in FIG. 47, the terminal 10 receives the responsibility transfer request notification transmitted by the transfer request notification unit 122 of the control server 20 (S311).
[0367] As shown in (b) of FIG. 48, upon receiving the responsibility transfer request notification, the control status presentation unit 111 switches the display from the status display b1 to the status display b2 "Assisting (requesting changeover)" indicating that a request for transfer of control responsibility is being made. In other words, the status display b2 is presented to the operator 50. This notifies the operator 50 that a request for transfer of control responsibility is being made by the administrator. Note that the status display b2 may also display information indicating that a transfer request has been made by the administrator.
[0368] Furthermore, the transfer request response input unit 211 of the terminal 10 displays a button display e4 "Accept Change Request" indicating that the transfer request is accepted, and a button display e5 "Reject Change Request" indicating that the transfer request is rejected. Here, the button display e4 is a button for accepting the transfer of control responsibility to another operator 50 set by the administrator, and the button display e5 is a button for not accepting the transfer of control responsibility to the other operator 50.
[0369] Next, the terminal 10 determines whether or not to approve the transfer of control responsibility for the mobile object 40 set by the administrator (S312). When the terminal 10 receives an input from the operator 50 indicating that the transfer is approved (for example, the button display e4 is operated), the terminal 10 determines that the transfer is approved (Yes in S312) and proceeds to step S313. When the terminal 10 receives an input from the operator 50 indicating that the transfer is not approved (for example, the button display e5 is operated), or when the terminal 10 does not receive an input indicating that the transfer is approved for a predetermined period of time (No in S312), the processing ends.
[0370] If the answer is No in step S312, the terminal 10 may transmit information indicating that the delivery has not been approved to the control server 20. This allows the administrator to be notified that the delivery has not been approved. If the answer is No in step S312, the terminal 10 may transmit information indicating that the delivery has not been approved to the terminal 10 of the other operator 50. This allows the other operator 50 to be notified that the delivery has not been approved.
[0371] Next, if the terminal 10 determines Yes in step S312, it further determines whether the other operator 50 has accepted responsibility (S313). If the terminal 10 acquires information indicating that the other operator 50 has accepted responsibility from the terminal 10 of the other operator 50, it determines that the other operator 50 has accepted responsibility (Yes in S313) and proceeds to step S14. If the terminal 10 acquires information indicating that the other operator 50 has not accepted responsibility from the terminal 10 of the other operator 50, it determines that the other operator 50 has not accepted responsibility (No in S313) and proceeds to step S16.
[0372] As shown in (c) of FIG. 48, the control status presentation unit 111 switches the display from the status display b2 to "Operator B is assisting," which is a status display b3 indicating that the transition to the control responsibility state (a state in which there is control responsibility) has been completed. This notifies the operator 50 that the control responsibility state has been released and control responsibility has been transferred to another operator 50 (operator B). At this time, the operation display a2 is switched to inactive (for example, hidden). This prevents the operator 50 from operating the moving object 40. In other words, the operation authority for the moving object 40 is transferred to the other operator 50.
[0373] Next, the operation of the terminal 10 that receives the responsibility transfer request will be described with reference to Figs. 49 and 50. Fig. 49 is a flowchart showing the operation (information processing method) of the terminal that receives the responsibility transfer request in the remote control system 1 according to this modified example. Fig. 50 is a diagram showing a screen displayed on the display unit 110a that receives the responsibility transfer request in the remote control system 1 according to this modified example. Fig. 50 shows an example of a screen displayed on the display unit 110a that is checked by another operator 50.
[0374] As shown in (a) of Figure 50, during remote monitoring, the control status presentation unit 111 displays on the display unit 110a the status display b4 "Operator A is assisting," which indicates that the operator 50 (operator A) is responsible for control.
[0375] As shown in FIG. 49, the terminal 10 receives a responsibility transfer request notification transmitted by the transfer request notification unit 122 of the control server 20 (S321).
[0376] As shown in (b) of Figure 50, when a responsibility transfer request notification is received, the transfer request presenting unit 112 switches the display from the status display b4 to "Switch Request (Operator A is handling)", which is a status display b5 indicating that a transfer request (switch request) has been notified. This makes it possible to notify other operators 50 that a request to transfer control responsibility is being made. Note that the status display b5 may also display information indicating that a transfer request has been made by the administrator.
[0377] Furthermore, the transfer request response input unit 211 displays a button display e4 "Accept Change Request" indicating that the transfer request set by the manager is accepted, and a button display e5 "Reject Change Request" indicating that the transfer request set by the manager is rejected. Here, the button display e4 is a button for accepting the transfer of control responsibility to another operator 50 set by the manager, and the button display e5 is a button for not accepting the transfer of control responsibility to the other operator 50.
[0378] Next, the terminal 10 determines whether or not to accept the control responsibility for the mobile object 40 set by the administrator (S322). If the terminal 10 receives an input indicating acceptance from another operator 50 (for example, the button display e4 is operated), the terminal 10 determines that acceptance is accepted (Yes in S322) and proceeds to step S323. If the terminal 10 receives an input indicating non-acceptance of acceptance from another operator 50 (for example, the button display e5 is operated), or if the terminal 10 does not receive an input indicating acceptance for a predetermined period of time (No in S322), the processing ends.
[0379] If the answer is No in step S322, the terminal 10 may transmit information indicating that the acceptance has not been approved to the control server 20. This allows the administrator to be notified that the acceptance has not been approved. If the answer is No in step S322, the terminal 10 may transmit information indicating that the acceptance has not been approved to the terminal 10 of the operator 50. This allows the operator 50 to be notified that the acceptance has not been approved.
[0380] Next, if the determination in step S322 is Yes, the terminal 10 further determines whether the current person in charge (here, the operator 50) has approved the delivery (S323). If the terminal 10 acquires information indicating that the operator 50 has approved the delivery from the terminal 10 of the operator 50, the terminal 10 determines that the operator 50 has approved the delivery (Yes in S323) and proceeds to step S23. Furthermore, if the terminal 10 acquires information indicating that the operator 50 has not approved the delivery from the terminal 10 of the operator 50, the terminal 10 determines that the operator 50 has not approved the delivery (No in S323) and ends the process.
[0381] As shown in (c) of FIG. 50 , upon accepting the responsibility transfer request notification, the control status presentation unit 111 switches the display of the status display b5 to "assisting," which is the status display b1 indicating that remote assistance is being performed and that the other operator 50 has control responsibility. Furthermore, the operation signal transmission unit 216 displays an operation display a2 for temporarily stopping the moving object 40 that has accepted control responsibility. In other words, the operation display a2 becomes active. This notifies the operator 50 that the moving object 40 has transitioned to a control responsibility state, and the operating authority (operation display a2) for the moving object 40 is transferred. Furthermore, the transfer request input unit 212 switches from button displays e4 and e5 to button displays e1 and e2.
[0382] (Other Embodiments) While the remote control system etc. according to one or more aspects has been described above based on the embodiments etc., the present disclosure is not limited to these embodiments etc. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art can conceive of to the present embodiment and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.
[0383] For example, the configurations shown in the modified examples of the above-described embodiments may be realized as, for example, a stand-alone device. For example, the remote control systems shown in Figures 42 to 44 may not have the configuration for executing the process related to the transfer of control responsibility shown in the embodiments. For example, such an information processing method executes MRM on the first mobile object when an abnormality in communication between a terminal owned by a first operator and a first mobile object, or an abnormality in a system including the terminal and the mobile object, is detected. Furthermore, for example, such an information processing system includes an acquisition unit that acquires information indicating that an abnormality in communication between a terminal owned by a first operator and the first mobile object, or an abnormality in a system including the terminal and the mobile object, has been detected, and an MRM instruction unit (or an MRM execution unit) that causes the first mobile object to execute MRM when either abnormality is detected.
[0384] Furthermore, in the above-described embodiments, if multiple operators attempt to assist / operate the same mobile object, the terminal or control server may reject the operator who attempts to assist / operate later. For example, if multiple operators simultaneously approve a transfer request or simultaneously perform an operation to start assisting / operating, the terminal or control server may reject the operation of the operator who performed the approval or start operation later. For example, the terminal or control server may reject the operation of the operator who performs an operation indicating that transfer of assistance / operation is possible after the second operator or after a predetermined number of operators. For example, the terminal or control server may transfer responsibility for the mobile object and the operating authority of the mobile object to the operator who first performed an operation to accept the transfer request.
[0385] In the above embodiments, the terminal or control server may automatically request another operator to transfer the responsibility if the period (or cumulative period) for which the operator is responsible is equal to or longer than a certain period. The terminal or control server may also monitor the vital signs of the operator, and if an abnormality in the operator is detected, may automatically request another operator to transfer the responsibility, or may execute MRM.
[0386] Furthermore, in the above-described embodiments, the terminal or the control server may preferentially transmit the transfer request notification to an operator who controls a small number of vehicles among a plurality of operators. For example, the terminal or the control server may preferentially transmit the transfer request notification to an operator who controls a predetermined number of vehicles or less, or to a predetermined number of lower-ranking operators who control a small number of vehicles.
[0387] In the above-described embodiments, the destination of the transfer request may be determined by a system administrator of the remote control system. In this case, the remote control system includes a reception unit that receives input from the system administrator. The reception unit may be a touch panel, a button, or may be configured to receive input by voice, gesture, or the like.
[0388] Furthermore, in the above-described embodiments, if a transfer request is approved, the control responsibility may be transferred before the operation authority of the moving object is transferred, or the control responsibility may be transferred after the operation authority of the moving object is transferred. Furthermore, for example, when the control responsibility is transferred from operator A to operator B, there may be a temporary period during which operators A and B each have operation authority for one moving object. In this case, the control responsibility may be held by only one of operators A and B. The same applies to the timing when the control responsibility and operation authority of the moving object are transferred from one of the automated driving system and the operator to the other.
[0389] In the remote control system according to the above-described embodiments, the terminal and the mobile body may communicate directly, or the terminal and the mobile body may communicate via a control server. For example, if the terminal and the mobile body can communicate directly, the remote control system does not need to include a control server. Furthermore, at least some of the functions of the control server may be provided by the terminal or the mobile body.
[0390] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.
[0391] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.
[0392] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.
[0393] Furthermore, the terminal or control server according to the above-described embodiments may be realized as a single device or may be realized by multiple devices. When the terminal or control server is realized by multiple devices, the components of the terminal or control server may be distributed in any manner among the multiple devices. When the terminal or control server is realized by multiple devices, the communication method between the multiple devices is not particularly limited, and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.
[0394] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, the term "LSI" is used, but depending on the level of integration, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After LSI fabrication, a field programmable gate array (FPGA) that can be programmed or a reconfigurable processor that can reconfigure the connections or settings of circuit cells within the LSI may also be used. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.
[0395] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip. Specifically, it is a computer system that includes a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.
[0396] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method (remote control method) shown in any of Figures 7, 9, 11, 12, 14, 16, 17, 20, 22, 26, 30, 35, 37, 39, 40, 42 to 45, 47, and 49.
[0397] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.
[0398] The present disclosure is useful for a remote control system for remotely controlling a moving object, etc.
[0399] 1 Remote control system 10 Terminal 20 Control server 30 Storage device 31 ODD information DB 32 Travel task information DB 33 Manual information DB 34 Operator qualification information DB 40 Mobile body 41 MRM execution unit 42 Mobile body information acquisition unit 43 Surrounding environment information acquisition unit 44, 130 Communication information acquisition unit 45 Mobile body monitoring unit 46 Automatic driving system (autonomous mobile system) 50 Operator 60 Input device 110 Output device 110a, 300 Display unit 111 Control status presentation unit 112 Transfer request presentation unit (first acquisition unit) 113 Transfer request presentation unit 114 Control requirement presentation unit 115 Mobile body information presentation unit 116 Surrounding environment information presentation unit 117 Terminal communication information acquisition unit 118, 131, 219 MRM instruction unit 119 Remote response request presentation unit 120 Terminal monitoring unit 121 Control status management unit 122 Transfer request notification unit 123 Transfer request recipient determination unit 124 Transfer request notification unit 125 Transfer acceptance determination unit 126 Instruction content determination unit 127 Mobile object information transmission unit 128 Surrounding environment information transmission unit 129 Operation signal transmission unit 132 Remote response request notification unit 133 Autonomous driving continuation acceptance determination unit 210 Input device 211 Transfer request response input unit (second acquisition unit, processing unit) 212 Transfer request input unit 213 Transfer acceptance request response input unit 214 Transfer request input unit 215 Start / end input unit 216 Operation signal transmission unit 217 Mobile object selection unit 218 Input device status acquisition unit 220 Call unit a1 Operation display a2 Operation display (operation section) A11 Map display section A13 List display section b1, b2, b3, b4, b5, b6, b7, b8, b9, b10, b12, b13, b14, b15, c Status display d Status display e1, e2, e3, e4, e5, e6, e7, e8, e9, e10, e11, e12, e13, e14 Button display f Call display g1, g2, g3, g4 Quantity display h Auxiliary display h1 1st display h2 2nd display h3 3rd display h4 4th display
Claims
1. An information processing method executed by an information processing system for one or more operators including a first operator to remotely control one or more moving bodies including a first moving body, comprising: acquiring a transfer request to transfer responsibility associated with the remote control of the first moving body to the first operator or an autonomous mobile system of the first moving body; acquiring information from the first operator or the autonomous mobile system regarding whether or not to accept the acquired transfer request; and if information approving the transfer request is acquired, transferring the responsibility of the first moving body to the first operator or the autonomous mobile system.
2. The information processing method according to claim 1, further comprising the step of: if information approving the transfer request is acquired, transferring the operating authority of the first moving body to the first operator or the autonomous mobile system independently of the responsibility of the first moving body.
3. The information processing method according to claim 2, further comprising transferring the responsibility of the first moving body and the operating authority of the first moving body to the first operator or the autonomous mobile system at different times.
4. The information processing method according to claim 3, further comprising transferring the responsibility of the first moving body to the first operator or the autonomous mobile system at a timing earlier than the operating authority of the first moving body.
5. The information processing method according to claim 3, further comprising transferring the operating authority of the first moving body to the first operator or the autonomous mobile system at a timing earlier than the responsibility of the first moving body.
6. An information processing method according to any one of claims 2 to 5, wherein the fact that the responsibility of the first moving body has been transferred to the first operator and the fact that the operating authority of the first moving body has been transferred to the first operator are presented to the first operator in different manners.
7. An information processing method according to any one of claims 2 to 5, wherein the transfer request is a request to transfer part of the responsibility of the first moving body, and when an operation approving the transfer request is acquired, the part of the responsibility of the first moving body and the part of the operating authority of the first moving body are transferred to the first operator or the autonomous mobile system.
8. An information processing method according to any one of claims 1 to 5, further comprising the steps of: upon receiving the transfer request, presenting details of the remote control of the first moving body; and, after presenting the details, obtaining an operation from the first operator as to whether or not to approve the transfer request.
9. An information processing method as described in any one of claims 1 to 5, comprising: acquiring skill information indicating the skills in the remote control of each of the one or more operators; determining whether the first operator has the skills required for the remote control of the first moving body based on the acquired skill information; and, if it is determined that the first operator has the skills required for the remote control of the first moving body, acquiring the transfer request to transfer the responsibility of the first moving body to the first operator.
10. An information processing method as claimed in any one of claims 1 to 5, wherein the one or more operators include a second operator remotely controlling the first moving object, an operation indicating that the responsibility for the first moving object is to be transferred is obtained from the second operator, the request for transfer to the first operator is notified by obtaining the operation, and information indicating that a request for transfer of the responsibility for the first moving object has been made is presented to the second operator.
11. The information processing method according to any one of claims 1 to 5, wherein the request for transfer to the first operator is notified from the autonomous mobile system.
12. An information processing method according to any one of claims 2 to 5, wherein the transfer request is transmitted to terminals of two or more operators including the first operator, and when the first operator among the two or more operators performs an operation to approve the transfer request earliest, the responsibility for the first mobile body and the operating authority for the first mobile body are transferred to the first operator.
13. The information processing method according to any one of claims 2 to 5, wherein the one or more moving bodies includes a second moving body, and the one or more operators includes a third operator who remotely controls the second moving body, and the information processing method comprises: acquiring from the first operator a third operation requesting the transfer of the responsibility associated with the remote control of the second moving body; transmitting to a terminal of the third operator a transfer request requesting the transfer of the responsibility of the second moving body based on the acquired third operation; and when a fourth operation indicating that the third operator has accepted the transfer request is acquired, the responsibility of the second moving body and the operating authority of the second moving body are transferred from the third operator to the first operator.
14. The information processing method described in any one of claims 2 to 5, wherein the one or more moving bodies includes a third moving body that is not remotely controlled by the one or more operators, and the information processing method comprises: presenting information for remotely controlling the third moving body to the first operator; acquiring from the first operator a fifth operation indicating to start remote control of the third moving body; and transferring the responsibility for the third moving body and the operating authority for the third moving body to the first operator based on the acquired fifth operation.
15. An information processing method as claimed in any one of claims 1 to 5, wherein the one or more moving bodies include a fourth moving body, and the information processing method further comprises presenting to the first operator an operation unit for accepting an operation indicating to start the remote control of the fourth moving body, and the operation unit becomes effective when the remote control skills possessed by the first operator satisfy the remote control skills required to continue the autonomous movement of the fourth moving body.
16. An information processing method according to any one of claims 2 to 5, wherein when the responsibility for the first moving object and the operating authority for the first moving object are transferred to the first operator, no other operator among the one or more operators other than the first operator is permitted to bear the responsibility for the first moving object unless given the permission of the first operator.
17. The information processing method according to any one of claims 2 to 5, further comprising, when an operation to reject the transfer request is acquired from the first operator, executing MRM (Minimal Risk Maneuver) for the first moving body to evacuate the first moving body.
18. An information processing method as claimed in any one of claims 2 to 5, further comprising: executing MRM on the first mobile body if an abnormality is detected in communication between a terminal owned by the first operator and the first mobile body, or an abnormality in a system including the terminal and the first mobile body, after the responsibility for the first mobile body and the operating authority for the first mobile body have been transferred to the first operator.
19. A terminal for one or more operators including a first operator to remotely control one or more moving bodies including a first moving body, the terminal comprising: a first acquisition unit that acquires a transfer request to transfer responsibility associated with the remote control of the first moving body to the first operator or an autonomous mobile system of the first moving body; a second acquisition unit that acquires information from the first operator or the autonomous mobile system regarding whether or not to accept the acquired transfer request; and a processing unit that transfers the responsibility of the first moving body to the first operator or the autonomous mobile system when information approving the transfer request is acquired from the first operator or the autonomous mobile system.
20. A program for causing a computer to execute the information processing method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Operation system for vehicle occupant non-operation
JP2012108868A
Driver assistant system for automated logging of protocol data, and method
JP2017517787A
Vehicle remote operation assistance system
JP2020042764A
Information processing method, and information processing system
JP2020060841A
Method and device for remotely operating vehicle
JP2020205037A
