Delay Detection Method, Delay Detection Device, and Program

Abstract

A delay detection method according to the present invention is to be executed by a mobile body that can be remotely controlled by an operator. The delay detection method involves (S2) adding first time information to mobile body information that indicates the state of the mobile body (10) and transmitting the mobile body information to a terminal (30) of the operator, (S6) receiving an operation signal that includes remote control details from the operator with respect to the mobile body information and also includes second time information based on the first time information included in the mobile body information, and (S7) detecting the delay at a remote control system that includes the mobile body (10) and the terminal (30) on the basis of the second time information and a prescribed time.

Description

Technical Field

[0001] The present disclosure relates to a delay detection method, a delay detection device, and a program.

Background Art

[0002] In recent years, unmanned transportation solutions using autonomous vehicles have attracted attention. Also, even in cases other than autonomous driving, transportation services provided by vehicles capable of full remote control have drawn attention from the perspective of labor reduction.

[0003] In such vehicles, it is currently difficult to achieve the autonomous driving level (Lv4) that does not require human intervention. First, it has been considered that a human intervenes by remote operation as needed. The exchange of signals in remote operation needs to be performed over an unstable network such as wireless, and it is necessary to consider safety measures in the event of network disconnection or delay.

[0004] Patent Document 1 discloses a system for determining the communication delay between an industrial vehicle having a vehicle communication unit that performs wireless communication and a remote operation device having a remote communication unit that performs wireless communication with the vehicle communication unit and is used to remotely operate the industrial vehicle.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the technology of Patent Document 1, since the delay time corresponding to the difference between the reception period required for the vehicle communication unit to receive a plurality of remote operation signals and the generation period required for generating the plurality of remote operation signals is used, the delay of the entire system cannot be monitored. In performing remote operation, it may be desirable to monitor the delay of the entire system.

[0007] Therefore, the present disclosure provides a delay detection method, a delay detection device, and a program that can more accurately monitor the delay of the entire system.

Means for Solving the Problem

[0008] A delay detection method according to an aspect of the present disclosure is a delay detection method executed by a mobile body that can be remotely controlled by an operator, wherein first time information is attached to mobile body information indicating the state of the mobile body and transmitted to the operator's terminal, and an operation signal based on the operator's operation on the mobile body with respect to the mobile body information, the operation signal including second time information based on the first time information included in the mobile body information is received, and based on the second time information and a predetermined time, a delay in a remote control system including the mobile body and the terminal is detected.

[0009] A delay detection device according to an aspect of the present disclosure is a delay detection device provided in a mobile body that can be remotely controlled by an operator, including a transmission unit that attaches first time information to mobile body information indicating the state of the mobile body and transmits it to the operator's terminal, a reception unit that receives an operation signal based on the operator's operation on the mobile body with respect to the mobile body information, the operation signal including second time information based on the first time information included in the mobile body information, and a determination unit that detects a delay in a remote control system including the mobile body and the terminal based on the second time information and a predetermined time.

[0010] A program according to an aspect of the present disclosure is a program for causing a computer to execute the above-described delay detection method.

Advantages of the Invention

[0011] According to an aspect of the present disclosure, it is possible to realize a delay detection method or the like that can more accurately monitor the delay of the entire system.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15A

Figure 15B

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 20

Figure 21A

Figure 21B

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29A

Figure 29B

Figure 30

Figure 31

Figure 32

Figure 33

Embodiments for Carrying Out the Invention

[0013] (Background Leading to the Present Disclosure) Before explaining embodiments and the like of the present disclosure, the background leading to the present disclosure will be described with reference to FIG. 1.

[0014] As described in the "Background Art", it has been considered to remotely control an autonomous vehicle by an operator as necessary. When remotely controlling an autonomous vehicle, it is necessary for the operator to recognize the surrounding situation in real time and reflect the operator's operation based on the recognition result on the behavior of the moving body with a sufficiently short delay.

[0015] Therefore, in some cases, it may be desirable to monitor the transmission delay of the entire system including not only the operation signal (control signal) for remote control but also the transmission of vehicle information such as images necessary for operation determination.

[0016] Note that remote control includes remote operation and remote monitoring. Remote operation means operating the movement of a moving body using an input device such as a steering wheel. Further, remote assistance may be permitting an operation in the moving body without using an input device such as a steering wheel. In the case of remote assistance, the moving body performs autonomous driving. As an example of remote assistance, for example, it means outputting a travel start instruction when a stopped moving body starts to travel.

[0017] Here, the delay in a system including a moving body and an operator's terminal will be described with reference to FIG. 1. FIG. 1 is a diagram for explaining the types of delays.

[0018] As shown in FIG. 1, examples of the delay include imaging delay, transmission delay in the video transmission system, display delay, reaction delay, signal acquisition delay, transmission delay in the remote operation signal transmission system, and execution delay.

[0019] Imaging delay is the delay required for imaging by an imaging device such as a camera mounted on a moving body. The imaging delay is, for example, the time from when the imaging device starts imaging until the terminal receives the video (an example of moving body information) obtained by the imaging.

[0020] The transmission delay in the video transmission system is the delay in communication when transmitting moving body information including video from a moving body such as a vehicle to the operator's terminal. The transmission delay in the video transmission system is, for example, the time from when the moving body transmits the moving body information until the terminal receives the moving body information.

[0021] The display delay is the processing delay until the video included in the moving body information received at the terminal is displayed on the display unit. The display delay is, for example, the time from when the terminal receives the moving body information until the moving body information is displayed on the display unit.

[0022] The reaction delay is the delay until the operator who has confirmed the moving body information displayed on the display unit performs an operation for remote control. The reaction delay is, for example, the time from when the moving body information is displayed on the display unit until the operator performs an operation on the input device.

[0023] The signal acquisition delay is the delay until the terminal acquires an operation signal corresponding to the operation input by the operator. The signal acquisition delay is, for example, the time from when the operator inputs an operation to the input device until the terminal receives the operation signal.

[0024] Note that the operation signal is a signal based on the operator's operation on the moving body for remotely controlling the moving body via the terminal, and is a signal including the remote control content (for example, a control command) that the operator operated on the input device. The operation signal may include control values such as the rotation speed of the motor, speed (accelerator or brake), and steering angle, for example.

[0025] The transmission delay in the remote operation signal transmission system is the delay in communication when transmitting the operation signal from the terminal to the moving body. The transmission delay in the remote operation signal transmission system is, for example, the time from when the terminal transmits the operation signal until the moving body receives the operation signal.

[0026] The execution delay is the delay until the control according to the actual operation content is completed by the moving body that has received the operation signal. The execution delay is the time from when the moving body receives the operation signal until the control according to the operation signal is completed (for example, the rotational speed of the motor of the moving body reaches the rotational speed included in the operation signal).

[0027] Thus, various delays can exist in the system. In Patent Document 1, mainly the transmission delay in the remote operation signal transmission system is considered, but other delays are not considered.

[0028] Also, when measuring the delay in another system such as ping / pong, there is a concern that the detection of video delay / operation delay may be delayed when sudden delays occur.

[0029] Therefore, the inventors of the present application have intensively studied a delay detection method and the like that can more accurately monitor the delay of the entire system, and have devised the following delay detection method and the like.

[0030] The delay detection method according to the first aspect of the present disclosure is a delay detection method executed by a moving body that can be remotely controlled by an operator, wherein first time information is added to moving body information indicating the state of the moving body and transmitted to the operator's terminal, and an operation signal based on the operator's operation on the moving body with respect to the moving body information, the operation signal including second time information based on the first time information included in the moving body information is received, and based on the second time information and a predetermined time, the delay in the remote control system including the moving body and the terminal is detected.

[0031] As a result, at least considering the round-trip transmission delay in the remote control system, the moving body can detect the delay. Therefore, according to the delay detection method, compared with the case where only one of the round-trip transmission delays is used, the delay of the entire remote control system can be monitored more accurately. Also, since the moving body can detect the delay, it is possible to take safety measures according to the delay on the moving body side.

[0032] Also, for example, the delay detection method according to the second aspect is the delay detection method according to the first aspect, where the predetermined time is the current time, and in the detection of the delay, the difference between the second time information and the current time may be calculated, and the delay may be detected based on the calculated difference.

[0033] As a result, the delay can be detected without performing complicated calculations only by calculating the difference between the second time information and the current time.

[0034] Also, for example, the delay detection method according to the third aspect is the delay detection method according to the first aspect or the second aspect, and in the detection of the delay, the difference between the second time information included in the operation signal received most recently and the current time may be calculated periodically.

[0035] As a result, since the delay is detected periodically, even when the delay varies over time, the delay can be monitored more accurately.

[0036] Also, for example, the delay detection method according to the fourth aspect is the delay detection method according to any one of the first aspect to the third aspect, and when the delay is detected, a predetermined operation may be executed on the moving body.

[0037] As a result, by executing the predetermined operation, for example, it becomes possible to take safety measures according to the delay on the moving body side.

[0038] Further, for example, the delay detection method according to the fifth aspect is a delay detection method according to any one of the first to third aspects, and when the internal time of the moving body rewinds by a predetermined amount or more, the moving body may be caused to execute a predetermined operation.

[0039] Thereby, when the internal time of the moving body has rewound by a predetermined amount or more, that is, when the delay cannot be accurately detected, the moving body can be caused to execute a predetermined operation. For example, when the internal time of the moving body has rewound by a predetermined amount or more, the moving body can be put into a safe state.

[0040] Further, for example, the delay detection method according to the sixth aspect is a delay detection method according to the fourth or fifth aspect, and causing the moving body to execute the predetermined operation may include causing the moving body to execute an MRM (Minimal Risk Maneuver).

[0041] Thereby, the safety of the moving body can be ensured when a delay is detected or the like.

[0042] Further, for example, the delay detection method according to the seventh aspect is a delay detection method according to the fourth or fifth aspect, the operation signal includes a control command based on the operation by the operator, and the predetermined operation may include ignoring the control command included in the received operation signal.

[0043] Thereby, when a delay is detected or the like, it is possible to suppress the moving body from being controlled by an operation signal that is considered inappropriate for the state of the moving body at that time. This leads to ensuring the safety of the moving body.

[0044] Further, for example, the delay detection method according to the eighth aspect is a delay detection method according to any one of the first to seventh aspects, and the first time information may include a time obtained by correcting, based on the time until the predetermined information included in the moving body information is acquired, the time when the predetermined information was acquired.

[0045] Accordingly, delay determination can be performed in consideration of the time until a predetermined piece of information is acquired (for example, imaging delay). By including this time, a more accurate delay is calculated, so that the delay of the entire remote control system can be monitored more accurately.

[0046] Further, for example, the delay detection method according to the ninth aspect is a delay detection method according to any one of the first to eighth aspects, and the second time information may include a time obtained by correcting the first time information based on the time from receiving the moving body information at the terminal until it is displayed.

[0047] Accordingly, delay determination can be performed in consideration of the time from receiving the moving body information until it is displayed (for example, display delay). By including this time, a more accurate delay is calculated, so that the delay of the entire remote control system can be monitored more accurately.

[0048] Further, for example, the delay detection method according to the tenth aspect is a delay detection method according to any one of the first to ninth aspects, and based on the time from receiving the operation signal until the control corresponding to the operation signal in the moving body is completed, the second time information is corrected, and based on the corrected second time information and the predetermined time, the delay in the remote control system may be detected.

[0049] Accordingly, delay determination can be performed in consideration of the time until the control corresponding to the operation signal is completed (for example, execution delay). By including this time, a more accurate delay is calculated, so that the delay of the entire remote control system can be monitored more accurately.

[0050] Further, for example, the delay detection method according to the eleventh aspect is a delay detection method according to any one of the first to tenth aspects, and the moving body information is further provided with first signature information and transmitted to the operator's terminal, the operation signal further including second signature information is received, and the second signature information may be verified based on the first signature information.

[0051] As a result, the operator can confirm, using an electronic signature, that the operator is performing remote control after checking the mobile body information transmitted by the own device.

[0052] Also, for example, the delay detection method according to the 12th aspect is the delay detection method according to the 11th aspect, and the second signature information may be information transmitted from the terminal at the start of remote control.

[0053] As a result, at the start of remote operation, the operator can confirm that the operator is transmitting an operation signal after checking the mobile body information transmitted by the own device.

[0054] Also, for example, the delay detection method according to the 13th aspect is the delay detection method according to any one of the 1st to 12th aspects, and first counter information is further added to the mobile body information and transmitted to the operator's terminal, and the operation signal further including second counter information is received, and the second counter information may be verified.

[0055] As a result, it is possible to confirm, using a counter, that the mobile body information and the operation signal are transmitted in order.

[0056] Also, for example, the delay detection method according to the 14th aspect is the delay detection method according to the 6th aspect, and when the delay is eliminated during the execution of the MRM, the execution of the MRM may be cancelled.

[0057] As a result, when the delay is eliminated, the mobile body can automatically recover from the MRM.

[0058] Also, for example, the delay detection method according to the 15th aspect is the delay detection method according to any one of the 1st to 14th aspects, and when the operator does not perform remote control on the mobile body, a time signal including the second time information may be received from the terminal.

[0059] As a result, even when the operator is performing remote assistance or the like, a delay can be detected.

[0060] Further, for example, the delay detection method according to the 16th aspect is the delay detection method according to any one of the 1st to 15th aspects, wherein the first time information is encrypted using a first encryption key, the encrypted first time information is attached to the mobile body information and transmitted to the terminal, and the second time information included in the received operation signal is decrypted using a first decryption key corresponding to the first encryption key.

[0061] As a result, it is possible to confirm, using an encryption technique, that the operator is performing remote control after checking the mobile body information transmitted by the own device.

[0062] Further, for example, the delay detection method according to the 17th aspect is the delay detection method according to any one of the 1st to 16th aspects, wherein a second encryption key is further attached to the mobile body information and transmitted to the terminal, the operation signal encrypted using the second encryption key is received, and the received operation signal is decrypted using a second decryption key corresponding to the second encryption key.

[0063] As a result, it is possible to confirm that the operation signal has not been tampered with. It is possible to prevent the mobile body from being controlled by an illegal operation signal.

[0064] Further, for example, the delay detection method according to the 18th aspect is the delay detection method according to any one of the 1st to 17th aspects, wherein the time indicated by the first time information and the time indicated by the second time information may be the same time.

[0065] As a result, a delay can be detected in consideration of the round-trip transmission delay in the remote control system. Further, since it is not necessary to correct the first time information at the terminal, the processing amount of the terminal can be reduced.

[0066] Further, for example, the delay detection device according to the 19th aspect is a delay detection device provided in a mobile body that can be remotely controlled by an operator, and includes a transmission unit that attaches first time information to mobile body information indicating the state of the mobile body and transmits the information to the operator's terminal, a reception unit that receives an operation signal based on the operator's operation on the mobile body with respect to the mobile body information, the operation signal including second time information based on the first time information included in the mobile body information, and a determination unit that detects a delay in a remote control system including the mobile body and the terminal based on the second time information and a predetermined time. Further, for example, the program according to the 20th aspect is a program for causing a computer to execute the delay detection method according to any one of the 1st to 18th aspects.

[0067] These achieve the same effects as the above-described delay detection method.

[0068] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium. The program may be pre-stored in the recording medium or may be supplied to the recording medium via a wide area communication network including the Internet or the like.

[0069] Hereinafter, embodiments and the like will be specifically described with reference to the drawings.

[0070] Note that the embodiments and the like described below are all illustrative of comprehensive or specific examples. The numerical values, shapes, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0071] In addition, each figure is a schematic diagram and is not necessarily drawn precisely. Therefore, for example, scales etc. in each figure do not necessarily match. Also, in each figure, substantially the same configurations are denoted by the same reference signs, and duplicate explanations are omitted or simplified.

[0072] In this specification, terms indicating relationships between elements such as correspondence, as well as numerical values and numerical ranges, are not expressions representing only strict meanings, but are expressions meaning that they include substantially equivalent ranges, for example, differences of about several % (or about 10%).

[0073] In this specification, ordinal numbers such as "first" and "second" do not mean the number or order of components, unless otherwise specified, and are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0074] (Embodiment 1) Hereinafter, the delay detection method and the like according to this embodiment will be described with reference to FIGS. 2 to 11.

[0075] [1-1. Configuration of Remote Control System] First, the configuration of the remote control system that executes the remote delay method will be described with reference to FIG. 2. FIG. 2 is a block diagram showing the functional configuration of the remote control system 1 according to this embodiment.

[0076] As shown in FIG. 2, the remote control system 1 includes a moving body 10, a control server 20, and a terminal 30. The remote control system 1 is a system for remotely operating an automatically drivable moving body 10 from a distance as needed.

[0077] The moving body 10 is a vehicle capable of switching between automatic driving and driving by remote operation. The vehicle is, for example, an automobile, but may be a truck, a bus, etc., or may be a railway, etc. Also, the moving body 10 may be an aircraft such as a drone, or may be a ship, etc.

[0078] The mobile body 10 includes each functional component realized by a CPU (Central Processing Unit) or MPU (Micro Processor Unit) that executes processing, and a memory unit such as a ROM (Read Only Memory) and a RAM (Random Access Memory) that stores programs for causing each functional component to execute various processes. Further, as functional components, the mobile body 10 includes a mobile body information acquisition unit 11a, a mobile body information transmission unit 11b, an operation signal reception unit 11c, a delay determination unit 12, an operation signal execution unit 13, an MRM (Minimal Risk Maneuver) execution unit 14, an assignment unit 15, a verification unit 16, a mobile body information acquisition delay correction unit 17a, an operation signal execution delay correction unit 17b, a decoding unit 18a, a vehicle encryption unit 18b, and a time management unit 19.

[0079] The mobile body information acquisition unit 11a acquires sensor information such as camera images and radar information or information related to the mobile body such as vehicle speed as mobile body information from sensors or the like mounted on the mobile body 10. Further, the mobile body information acquisition unit 11a acquires the acquisition time of the mobile body information acquired from sensors or the like as the mobile body information acquisition time. Note that the mobile body information only needs to include the result of sensing the surroundings of the mobile body 10 or information indicating the running state of the mobile body 10. For example, it may include at least one of camera images, LiDAR information (measurement results by LiDAR), radar information (measurement results by radar), sonar information (measurement results by sonar), the speed of the mobile body, and the angular velocity of the mobile body.

[0080] The mobile body information transmission unit 11b attaches the mobile body information acquisition time to the mobile body information and transmits it to the terminal 30. The mobile body information transmission unit 11b is an example of a transmission unit.

[0081] The operation signal reception unit 11c receives an operation signal from the terminal 30 and acquires the mobile body information acquisition time included in the operation signal. The operation signal reception unit 11c is an example of a reception unit.

[0082] The delay determination unit 12 calculates the round-trip delay (delay time) based on the mobile body information acquisition time and the current time. The round-trip delay includes at least the transmission delay in the video transmission system shown in FIG. 1 and the transmission delay in the remote operation signal transmission system. Further, the delay determination unit 12 determines whether the delay is equal to or less than a threshold value. That is, the delay determination unit 12 determines whether there is a delay. Note that having a delay means that the delay is greater than a specified value (threshold value). The current time is an example of a predetermined time, and the delay determination unit 12 is an example of a determination unit.

[0083] The operation signal execution unit 13 causes the mobile body 10 to execute a specific operation corresponding to the operation signal based on the operation signal.

[0084] The MRM execution unit 14 causes the mobile body 10 to execute MRM when a delay is detected or the like. MRM is an example of a predetermined operation.

[0085] The providing unit 15 provides predetermined information to the mobile body information. The providing unit 15 may, for example, provide an electronic signature for verification using a decryption key held by the mobile body 10 as the predetermined information to the mobile body information. The electronic signature may be, for example, a hash value calculated using a hash function. Further, the providing unit 15 may provide a counter (counter value) as the predetermined information to the mobile body information. The predetermined information provided by the providing unit 15 is attached (replaced) to the operation signal by the terminal 30 and sent back to the mobile body 10.

[0086] The verification unit 16 executes verification using the predetermined information included in the received operation signal. When the predetermined information is an electronic signature, the verification unit 16 verifies whether the electronic signature included in the operation signal is the electronic signature issued by its own device (for example, whether the two electronic signatures match). Further, when the predetermined information is a counter, the verification unit 16 verifies whether the reception order of the received operation signal is in the order of the counter.

[0087] The mobile body information acquisition delay correction unit 17a estimates the time required for acquiring mobile body information and corrects the mobile body information acquisition time. The mobile body information acquisition delay correction unit 17a estimates the imaging delay shown in FIG. 1 and corrects the mobile body information acquisition time included in the mobile body information based on the estimated imaging delay.

[0088] The operation signal execution delay correction unit 17b estimates the time required for executing the operation signal and corrects the mobile body information acquisition time. The operation signal execution delay correction unit 17b estimates the execution delay shown in FIG. 1 and corrects the mobile body information acquisition time included in the operation signal based on the execution delay.

[0089] The decoding unit 18a decodes the operation signal based on the decoding key.

[0090] The vehicle encryption unit 18b generates a pair of an encryption key and a decoding key, encrypts the mobile body information acquisition time to be attached to the video, and transmits the encryption key to the terminal 30.

[0091] The time management unit 19 synchronizes with a GPS (Global Positioning System) or the like and continuously outputs a highly accurate time. Further, the time management unit 19 detects a time deviation between the highly accurate time and the internal time of the mobile body 10 and gives a notification.

[0092] Note that the mobile body 10 only needs to include at least the mobile body information transmission unit 11b, the operation signal reception unit 11c, and the delay determination unit 12. For example, the mobile body 10 may include a mobile body information acquisition unit 11a, a mobile body information transmission unit 11b, an operation signal reception unit 11c, a delay determination unit 12, an operation signal execution unit 13, and an MRM execution unit 14. Further, a delay detection device is realized by each component included in the mobile body 10. As the delay detection device, it only needs to include at least the mobile body information transmission unit 11b, the operation signal reception unit 11c, and the delay determination unit 12.

[0093] The control server 20 is an information processing device communicably connected to each of the mobile body 10 and the terminal 30. The control server 20 may be a cloud server or a local server.

[0094] The control server 20 includes each functional component realized by a CPU, MPU, etc. that execute processing, and a memory unit such as a ROM and a RAM that stores programs for causing each functional component to execute various processes. Further, the control server 20 includes, as functional components, a control state management unit 21, a mobile body information transmission unit 22, and an operation signal transmission unit 23.

[0095] The control state management unit 21 determines between which terminal 30 and mobile body 10 the mobile body information and the operation signal are to be exchanged. For example, the control state management unit 21 determines the destination terminal 30 based on the information indicating the destination included in the mobile body information, and determines the destination mobile body 10 based on the information indicating the destination included in the operation signal.

[0096] Based on the determination of the control state management unit 21, the mobile body information transmission unit 22 transmits the mobile body information to the designated terminal 30.

[0097] Based on the determination of the control state management unit 21, the operation signal transmission unit 23 transmits the operation signal to the designated mobile body 10.

[0098] The terminal 30 is an input device 40 (see FIG. 12 etc. described later) for inputting operations when an operator performs a remote operation, and a terminal device for remote operation that is connected to the control server 20 and transmits and receives information required for the remote operation. For example, the terminal 30 performs processing to display the video included in the mobile body information received from the mobile body 10 to the operator. Further, for example, the terminal 30 performs processing to transmit the operation signal from the input device 40 to the mobile body 10. In the present embodiment, the terminal 30 is characterized in that it adds the mobile body information acquisition time included in the mobile body information to the operation signal indicating the operation input by the operator to the input device 40 with respect to the mobile body information received from the mobile body 10 and transmits it to the mobile body 10.

[0099] The terminal 30 is configured to include each functional component realized by a CPU, MPU, or the like that executes processing, and a memory unit such as a ROM and a RAM that stores programs for causing each functional component to execute various processes. Further, as a functional configuration, the terminal 30 includes a mobile body information receiving unit 31a, an operation signal receiving unit 31b, an operation signal transmitting unit 32, a mobile body information display unit 33a, a delay time display unit 33b, a display delay correction unit 34a, an operation signal acquisition delay correction unit 34b, an operator reaction delay correction unit 34c, a remote operation switching unit 35, and an encryption unit 36.

[0100] The mobile body information receiving unit 31a receives mobile body information and extracts the mobile body information acquisition time included in the mobile body information.

[0101] The operation signal receiving unit 31b is communicably connected to the input device 40 and receives an operation signal from the input device 40.

[0102] The operation signal transmitting unit 32 attaches the mobile body information acquisition time extracted by the mobile body information receiving unit 31a to the operation signal and transmits it to the mobile body 10.

[0103] The mobile body information display unit 33a displays the mobile body information on the screen. In the present embodiment, the mobile body information display unit 33a displays video. The mobile body information display unit 33a is a monitor and is realized by, for example, a liquid crystal display device or the like, but is not limited thereto. Note that displaying is an example of presenting. Presenting may mean displaying, or may mean emitting sound or light.

[0104] The delay time display unit 33b displays the delay time (for example, the round-trip delay time) on the screen. The delay time display unit 33b is a monitor and is realized by, for example, a liquid crystal display device or the like, but is not limited thereto. Note that the delay time display unit 33b and the mobile body information display unit 33a may be realized by one display device.

[0105] The delay correction unit 34a estimates the delay from receiving the moving body information until it is displayed on the screen of the moving body information display unit 33a, and corrects the moving body information acquisition time extracted by the moving body information receiving unit 31a. This correction is performed to make a delay determination in consideration of the display delay shown in FIG. 1.

[0106] The operation signal acquisition delay correction unit 34b estimates the operation signal transmission processing delay of the input device 40, and corrects the moving body information acquisition time extracted by the moving body information receiving unit 31a. This correction is performed to make a delay determination in consideration of the signal processing delay shown in FIG. 1.

[0107] The operator reaction delay correction unit 34c estimates the operator's reaction delay, and corrects the moving body information acquisition time extracted by the moving body information receiving unit 31a. This correction is performed to make a delay determination in consideration of the reaction delay shown in FIG. 1.

[0108] The remote operation switching unit 35 receives a switching button signal of a controller (not shown), and transmits a remote control switching signal to the moving body 10. The controller may be included in the input device 40, for example. The remote control switching signal is a signal indicating a switch from automatic driving to driving by remote control (for example, remote operation).

[0109] The encryption unit 36 creates an encryption key and a decryption key, and encrypts the operation signal.

[0110] Further, the remote control system 1 may further include an input device 40 for the operator to input for remote operation. The input device 40 is communicably connected to the terminal 30, and includes at least one of, for example, a steering wheel, a brake, an accelerator, and the like.

[0111] [1-2. Operation of the Remote Control System] Next, the operation of the remote control system 1 configured as described above will be described with reference to FIGS. 3 to 11. First, the outline of the operation in the remote control system 1 will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining the outline of the operation (delay detection method) of the remote control system 1 according to the present embodiment.

[0112] As shown in FIG. 3, the mobile body 10 assigns the mobile body information acquisition time to the mobile body information (S1), and transmits the mobile body information with the mobile body information acquisition time assigned thereto to the terminal 30 (S2). The mobile body information acquisition time is an example of the first time information.

[0113] The terminal 30 receives the mobile body information transmitted through the mobile body information transmission system N (S3), replaces the mobile body information acquisition time with the operation signal from the input device 40 corresponding to the received mobile body information (S4), and transmits the operation signal (S5). The control server 20 transmits the operation signal from the terminal 30 to the mobile body 10 (S6). The mobile body 10 verifies the mobile body information acquisition time included in the operation signal (S7). The mobile body information acquisition time included in the operation signal is an example of the second time information.

[0114] In this way, by associating the mobile body information (for example, each frame of the video) with the operation signal and measuring the round-trip delay, it is possible to measure the total transmission delay of the operation signal based on the video frame actually seen by the operator. For example, it is possible to verify at what time before the operation signal received in step S6 is a signal for the video captured by the mobile body 10. For example, the mobile body 10 calculates the delay time (total transmission delay) in steps S2 to S6 from the mobile body information acquisition time included in the operation signal and the current time, and detects the delay depending on whether the delay time is equal to or less than the threshold value. It can be said that the mobile body 10 executes the delay determination. Further, since the mobile body 10 performs the delay determination using the mobile body information acquisition time given by the mobile body 10, for example, time synchronization between the mobile body 10 and the terminal 30 is unnecessary. The current time is an example of a predetermined time.

[0115] Hereinafter, the operation in the remote control system 1 will be further described with reference to FIG. 4 and the like. FIG. 4 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In FIG. 4, when the total of the time from acquiring information from the moving body 10 until the moving body information is displayed to the operator and the time from when the operator operates until it is reflected on the moving body 10 cannot be made less than a certain time, the operation when the MRM is executed is illustrated. Further, FIG. 4 shows a sequence diagram when a remote operation is being performed. In the subsequent sequence diagrams as well, unless otherwise specified, a sequence diagram when a remote operation is being performed is similarly shown.

[0116] As shown in FIG. 4, the moving body information acquisition unit 11a of the moving body 10 acquires moving body information from a camera or the like, and assigns the acquisition time of the moving body (moving body acquisition time) (time A), which is the time when the moving body information is acquired, to the moving body information (S11).

[0117] Next, the moving body information transmission unit 11b transmits the moving body information to which time A is assigned to the terminal 30 (S12). The moving body information transmission unit 11b transmits the moving body information to the terminal 30 by wireless communication.

[0118] Next, the moving body information reception unit 31a of the terminal 30 receives the moving body information transmitted from the moving body information transmission unit 11b, acquires time A from the received moving body information, and holds it (S13).

[0119] Next, the moving body information display unit 33a displays the moving body information (S14). The moving body information display unit 33a displays, for example, a video as the moving body information. The operator checks the video of the moving body information display unit 33a and performs an operation for remote control on the input device 40 as necessary.

[0120] Next, the operation signal reception unit 31b acquires an operation signal for the moving body 10 with respect to the video displayed in step S14 from the input device 40 (S15).

[0121] Next, the operation signal transmission unit 32 attaches the time A acquired in step S13 to the operation signal (S16) and transmits the operation signal (S17). The operation signal transmission unit 32 transmits the operation signal to the mobile body 10 by wireless communication.

[0122] Next, the operation signal reception unit 11c receives the operation signal from the terminal 30 and extracts the time A from the received operation signal (S18).

[0123] Next, the delay determination unit 12 determines whether the operation signal is delayed (whether an operation signal delay has occurred) based on the time A extracted from the operation signal (S19). The delay determination unit 12 determines the presence or absence of delay based on whether the difference time obtained by subtracting the time A from the current time is greater than a specified value (threshold value). When the difference time is greater than the specified value, the delay determination unit 12 determines that a delay has occurred; when the difference time is less than or equal to the specified value, the delay determination unit 12 determines that no delay has occurred.

[0124] Note that the delay determination unit 12 repeatedly executes the process of step S19 until the remote operation ends. The delay determination unit 12, for example, periodically calculates the difference between the mobile body acquisition time included in the most recently received operation signal and the current time.

[0125] Next, when the MRM execution unit 14 determines that a delay has occurred according to the delay determination unit 12, since the received operation signal may be an operation signal for an operation performed by the operator after viewing an image captured at an earlier time, for safety reasons, the MRM is executed (S20).

[0126] Also, when the operation signal execution unit 13 determines that the delay of the operation signal is less than or equal to the specified value according to the delay determination unit 12, the operation signal is executed (S21). The operation signal execution unit 13 causes the mobile body 10 to execute the operation indicated by the operation signal. The operation is, for example, a change in speed, rudder angle, etc., but is not limited thereto.

[0127] In addition, when the mobile body 10 can be remotely controlled from a plurality of terminals 30, the mobile body 10 may transmit an operator notification signal (see FIG. 11 described later) to the plurality of terminals 30 at an arbitrary timing between steps S11 to S21 shown in FIG. 4.

[0128] Subsequently, the outline (concept) of each process shown in FIG. 4 will be described with reference to FIGS. 5A to 5E. FIG. 5A is a flowchart showing a first operation (delay detection method) of the mobile body 10 according to the present embodiment. FIG. 5A shows the operation of the mobile body 10 in acquiring and transmitting mobile body information. FIG. 5A corresponds to steps S11 and S12 shown in FIG. 4.

[0129] As shown in FIG. 5A, the mobile body information acquisition unit 11a acquires mobile body information from a camera or the like (S31a), and assigns the mobile body information acquisition time (time A) at which the mobile body information is acquired to the mobile body information (S32a). Then, the mobile body information transmission unit 11b transmits the mobile body information to which time A is assigned to the terminal 30 (S33a).

[0130] FIG. 5B is a flowchart showing a second operation (delay detection method) of the terminal 30 according to the present embodiment. FIG. 5B shows the operation of the terminal 30 in receiving mobile body information. FIG. 5B corresponds to steps S13 and S14 shown in FIG. 4.

[0131] As shown in FIG. 5B, the mobile body information reception unit 31a receives the mobile body information from the mobile body 10 (S41a), extracts and holds the mobile body information acquisition time from the received mobile body information (S42a).

[0132] Next, the mobile body information display unit 33a displays the mobile body information on the screen (S43a).

[0133] FIG. 5C is a flowchart showing a third operation (delay detection method) of the terminal 30 according to the present embodiment. FIG. 5C shows the operation of the terminal 30 in transmitting an operation signal. FIG. 5C corresponds to steps S15 to S17 shown in FIG. 4.

[0134] As shown in FIG. 5C, the operation signal receiving unit 31b receives an operation signal from the input device 40 (S51a), and assigns the latest moving body information acquisition time to the operation signal (S52a). Then, the operation signal transmitting unit 32 transmits the operation signal to the moving body 10 (S33a).

[0135] FIG. 5D is a flowchart showing a fourth operation (delay detection method) in the moving body 10 according to the present embodiment. FIG. 5D shows the operation of the moving body 10 in receiving the operation signal. FIG. 5D corresponds to step S18 shown in FIG. 4.

[0136] As shown in FIG. 5D, the operation signal receiving unit 11c receives an operation signal from the terminal 30 (S61a), extracts and holds the moving body information acquisition time from the operation signal (S62a).

[0137] FIG. 5E is a flowchart showing a fifth operation (delay detection method) in the moving body 10 according to the present embodiment. FIG. 5E shows the operation of delay determination in the moving body 10. FIG. 5E corresponds to steps S20 and S21 shown in FIG. 4.

[0138] As shown in FIG. 5E, the delay determination unit 12 compares the latest held moving body information acquisition time with the current time (S71a).

[0139] When it is determined by the delay determination unit 12 that the current time - the moving body information acquisition time > the specified value (yes in S72a), the MRM execution unit 14 executes the MRM (S73a). Also, when the delay determination unit 12 determines that the current time - the moving body information acquisition time > the specified value is not satisfied (no in S72a), it returns to step S71a and continues the process until the remote operation ends.

[0140] Next, a specific example of each process shown in FIG. 4 will be described with reference to FIGS. 6A to 6E. FIG. 6A is a flowchart showing a specific example of the first operation (delay detection method) in the mobile body 10 according to the present embodiment. In FIGS. 6A to 6E, the mobile body information is camera video, and a specific example of the operation when WebRTC (Web Real-Time Communication) is used as the video transmission protocol of the camera video and Json (JavaScript Object Notation) or protocolBuffer is used for the serialization of the operation signal is illustrated.

[0141] FIG. 6A shows the operation of the mobile body 10 in acquiring and transmitting the video stream. FIG. 6A corresponds to steps S11 and S12 shown in FIG. 4.

[0142] As shown in FIG. 6A, the mobile body information acquisition unit 11a receives a video stream as mobile body information from a camera or the like (S31b), and updates the video acquisition timestamp on the memory to the reception time of step S31b (S32b). Updating the video acquisition timestamp on the memory means updating the video acquisition timestamp assigned to the mobile body information. The video acquisition timestamp is an example of the mobile body acquisition time.

[0143] Next, the mobile body information transmission unit 11b encodes the video stream (S33b), and adds the video acquisition timestamp to the frame data (S34b). Adding the video acquisition timestamp to the frame data is an example of assigning the mobile body acquisition time to the mobile body information.

[0144] Next, the mobile body information transmission unit 11b divides the frame data into RTP (Real-time Transport Protocol) packets (S35b), and transmits the video packets to the terminal 30 (S36b).

[0145] FIG. 6B is a flowchart showing a specific example of the second operation in the terminal 30 according to the present embodiment. FIG. 6B corresponds to steps S13 and S14 shown in FIG. 4.

[0146] As shown in FIG. 6B, the moving body information receiving unit 31a receives a video stream from the moving body 10 (S41b), constructs frame data (data for one screen) from the received RTP packets (S42b), extracts a time stamp from the frame data (S43b), and updates the time stamp in the memory (S44b). Updating the time stamp in the memory means updating the time stamp attached to the operation signal. Further, the moving body information receiving unit 31a decodes the frame data (S45b).

[0147] Next, the moving body information display unit 33a displays a video based on the decoded frame data (S46b). Thereby, the operator can be presented with the video, that is, the state of the moving body 10.

[0148] FIG. 6C is a flowchart showing a specific example of the third operation (delay detection method) in the terminal 30 according to the present embodiment. FIG. 6C corresponds to steps S15 to S17 shown in FIG. 4.

[0149] As shown in FIG. 6C, the operation signal receiving unit 31b receives an operation packet as an operation signal from the input device 40 (S51b), executes deserialization of the operation packet (S52b), and assigns the time stamp in the memory to the operation information (operation signal) (S53b). Then, the operation signal transmitting unit 32 executes serialization of the operation information (S54b), divides the operation information into packets (S55b), and transmits the operation packet to the moving body 10 (S56b).

[0150] FIG. 6D is a flowchart showing a specific example of the fourth operation (delay detection method) in the moving body 10 according to the present embodiment. FIG. 6D corresponds to step S18 shown in FIG. 4.

[0151] As shown in FIG. 6D, the operation signal receiving unit 11c receives an operation packet from the terminal 30 as an operation signal (S61b), deserializes the operation packet (S62b), extracts a time stamp from the operation signal (S63b), and updates the time stamp on the memory (S64b).

[0152] FIG. 6E is a flowchart showing a specific example of the fifth operation (delay detection method) in the moving body 10 according to the present embodiment. FIG. 6E corresponds to steps S20 and S21 shown in FIG. 4.

[0153] As shown in FIG. 6E, the delay determination unit 12 compares the time stamp on the memory with the current time (S71b).

[0154] When it is determined by the delay determination unit 12 that the current time - time stamp (time when moving body information is acquired) > specified value (yes in S72b), the MRM execution unit 14 executes the MRM (S73b). Also, when it is determined by the delay determination unit 12 that the current time - time stamp > specified value is not satisfied (no in S72b), the operation signal execution unit 13 executes the operation signal (S74b), returns to step S71b, and continues the process until the remote operation ends.

[0155] FIG. 7 is a diagram for explaining the details of the operation (delay detection method) of the remote control system 1 according to the present embodiment. FIG. 7 shows the flow when the time stamp is replaced at the terminal 30. Specifically, FIG. 7 shows a timing chart when there is no display delay and signal acquisition delay for video frame reception timing, video drawing, time stamp in the memory, operation signal transmission, and operation signal reception timing. Also, in FIG. 7, an example is shown in which frames are received in the order of frame A, frame B, frame C, and frame D.

[0156] When Frame A is received, Frame A is drawn, and the timestamp in the memory is updated to the timestamp of Frame A, and the timestamp of Frame A is retained until Frame B is received. When an operation signal is received from the input device 40 during the period from the reception of Frame A to the reception of Frame B, since the operation signal is considered to be a signal by the operation of the operator who has viewed the video of Frame A, the timestamp of Frame A is attached to the operation signal. Then, the operation signal to which the timestamp of Frame A is attached is transmitted to the moving body 10.

[0157] When Frame B is received, the video switches to the drawing of Frame B, and the timestamp in the memory is updated to the timestamp of Frame B.

[0158] Also, when a plurality of operation signals are received from the input device 40 during the period from the reception of Frame C to the reception of Frame D, since each of the plurality of operation signals is considered to be a signal by the operation of the operator who has viewed the video of Frame C, the timestamp of Frame C is attached to each of the plurality of operation signals.

[0159] Here, message examples of each signal will be described with reference to FIGS. 8 to 11. FIG. 8 is a diagram showing a message example of an operation signal according to the present embodiment. In FIGS. 8 to 11, an example in which RTCP (Real-time Transport Control Protocol) is used as a protocol is shown.

[0160] V = 2, P, IC, PT = APP = 204, and length L shown in FIG. 8 are commonly included items. In FIG. 8, for convenience, "PT = APP = 204" is illustrated as "PT = APP".

[0161] "SSRC of RTP packet sender" stores the identifier of the terminal 30 (video reception side). The identifier is used to reject operations from terminals 30 other than those having its own operation authority.

[0162] "RCMD (ASCII)" stores the message name, and here, information indicating that it is an operation signal is stored.

[0163] "Received frame border RTP timestamp" stores the transmission timestamp (unix time) of the received mobile body information (for example, the mobile body acquisition time given to the mobile body information in mobile body 10). For example, in an Insertable stream or the like, it stores the timestamp embedded in the RTP packet.

[0164] "SSRC of RTP packet sender" stores the identifier of mobile body 10 (the video transmission side). The identifier is used to determine whether the operation signal is from terminal 30 that has viewed the video transmitted by the own device.

[0165] "sequence number" stores the sequence number. The sequence number is used to confirm whether the messages are being sent in order.

[0166] "Control message" stores control values such as the accelerator value and the steering wheel angle.

[0167] Note that in RTCP, it is possible to arbitrarily set which items are included in the message. In the present disclosure, the item "Received frame border RTP timestamp" is included, and the timestamp (mobile body acquisition time) included in the mobile body information, that is, the timestamp given in mobile body 10, is stored in "Received frame border RTP timestamp".

[0168] FIG. 9 is a diagram showing a message example of the operation start signal according to the present embodiment. The operation start signal is a signal transmitted from terminal 30 to mobile body 10 when starting a remote operation.

[0169] "SSRC of packet sender" stores the identifier of the terminal 30 (video receiving side), and here, the identifier of the terminal 30 that requests the operation right of the mobile body 10 is stored.

[0170] "RREQ (ASCII)" stores the message name, and here, the information indicating the start of the remote operation is stored.

[0171] FIG. 10 is a diagram showing a message example of the operation end signal according to the present embodiment. The operation end signal is a signal transmitted from the terminal 30 to the mobile body 10 when the remote operation is ended.

[0172] "SSRC of packet sender" stores the identifier of the terminal 30 (video receiving side), and here, the identifier of the terminal 30 that abandons the operation right of the mobile body 10 is stored.

[0173] "REND (ASCII)" stores the message name, and here, the information indicating the end of the remote operation is stored.

[0174] FIG. 11 is a diagram showing a message example of the operator notification signal according to the present embodiment. This is a signal transmitted from the mobile body 10 to a plurality of terminals 30 when there are a plurality of terminals 30 having the right to remotely operate the mobile body 10 and the remote operation is being performed by any one of the plurality of terminals 30. Since the mobile body 10 can be remotely operated only from one terminal 30, when it has already received a remote operation from one terminal 30, an operator notification signal is transmitted to notify other terminals 30 that they have already received a remote operation.

[0175] "SSRC of packet sender" stores the identifier of the mobile body 10 (video transmitting side).

[0176] "ROPR (ASCII)" stores the message name, and here, the information indicating that it is a message for notifying which terminal 30 is remotely operating is stored.

[0177] "SSRC of control RTCP packet sender" stores the identifier of the operator, and here, information indicating which video receiver is remotely operating the mobile body 10 is stored. The video receiver means the operator corresponding to the terminal 30 that has received the mobile body information from the mobile body 10. It can also be said that information indicating which operator's operation packet the mobile body 10 is receiving is stored.

[0178] (Embodiment 2) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 12. In the following, the description will focus on the differences from Embodiment 1, and the description of the same or similar content as in Embodiment 1 will be omitted or simplified. In each of the embodiments and modification examples after Embodiment 2, the configuration of each component of the remote control system may be the same as that of the remote control system 1 according to Embodiment 1, and the description will be made using the reference numerals used in Embodiment 1.

[0179] FIG. 12 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In FIG. 12, an example in which the MRM is executed when no operation signal is received for a certain period after the operation signal is received will be described.

[0180] As shown in FIG. 12, the operation signal receiving unit 11c of the mobile body 10 holds the operation signal reception time (time B), which is the time when the operation signal is received (S81).

[0181] Then, until the next control signal (operation signal) comes or the MRM is executed, the delay determination unit 12 determines the operation signal delay based on time B (S82). The delay determination unit 12 makes this determination by whether the current time - time B > specified value is satisfied. When the current time - time B > specified value is satisfied, the delay determination unit 12 determines that the operation signal has been delayed, and when the current time - time B > specified value is not satisfied, the delay determination unit 12 determines that the operation signal has not been delayed.

[0182] When the delay determination unit 12 determines that a delay has occurred in the operation signal, the MRM execution unit 14 executes the MRM (S83).

[0183] As described above, in the present embodiment, when the total time from acquiring the moving body information from the moving body 10 to presenting the moving body information to the operator and the time from when the operator operates until it is reflected in the moving body 10 cannot be made less than a certain time (prescribed value), the control is performed so as to execute the MRM.

[0184] (Embodiment 3) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 13. FIG. 13 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In FIG. 13, an example will be described in which, when a delay occurs, instead of executing the MRM, an operation signal having a delay of a certain level or more is ignored on the moving body 10 side.

[0185] As shown in FIG. 13, when the delay determination unit 12 determines that the delay of the operation signal is equal to or less than the prescribed value, that is, when it is determined that the operation signal is not delayed, the operation signal execution unit 13 executes the operation signal (S21). The operation signal execution unit 13 causes the moving body 10 to execute the operation indicated by the operation signal.

[0186] In addition, when the delay determination unit 12 determines that the delay of the operation signal is greater than the prescribed value, that is, when it is determined that the operation signal is delayed, the operation signal execution unit 13 ignores the operation signal. For example, when the operation signal execution unit 13 determines that the operation signal is delayed, it ignores the control command included in the received operation signal. That is, when the operation signal execution unit 13 determines that the operation signal is delayed, it does not execute the control of the moving body 10 by the operation signal. Ignoring the operation signal is an example of a predetermined operation.

[0187] Note that the moving body 10 according to the present embodiment may not include the MRM execution unit 14.

[0188] (Embodiment 4) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIGS. 14 to 18. FIG. 14 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example in which the acquisition delay (imaging delay) of the moving body information and the display delay of the monitor are considered will be described.

[0189] As shown in FIG. 14, the moving body information acquisition delay correction unit 17a of the moving body 10 estimates the moving body information acquisition delay and corrects the time A1 (S91). The moving body information acquisition delay correction unit 17a corrects the time A1 to a time in the past by the amount of the moving body information acquisition delay time. For example, when the time A1 is 10:00 and the moving body information acquisition delay is 1 minute, the moving body information acquisition delay correction unit 17a corrects the time A1 to 9:59 to calculate the time A2. In reality, a delay of 1 minute is unlikely to occur, but for the sake of explanation, 1 minute is exemplified.

[0190] The time A2 in the present embodiment is an example of the first time information. The first time information may include the time A2 obtained by correcting the time A1 at which the moving body 10 acquires a predetermined information (for example, camera video) included in the moving body information based on the time (for example, imaging delay) until the predetermined information is acquired.

[0191] The moving body information receiving unit 31a of the terminal 30 receives the moving body information, acquires the time A2 from the received moving body information, and holds it (S92).

[0192] When the moving body information is displayed (S14), the display delay correction unit 34a estimates the display delay and corrects the time A2 (S93). The display delay correction unit 34a corrects the time A2 to a time in the past by the amount of the display delay time to calculate the time A3.

[0193] The time A3 in the present embodiment may be an example of the second time information. The second time information may include the time obtained by correcting the time A2 based on the time (display delay) from when the moving body information is received at the terminal 30 until it is displayed.

[0194] The operation signal transmission unit 32 attaches the time A3 to the operation signal (S94).

[0195] Next, the operator response delay correction unit 34c estimates the operator response delay and corrects the time A3 (S95). The operator response delay correction unit 34c calculates the time A4 by correcting the time A3 to a time in the past by the operator response delay time.

[0196] Next, the operation signal acquisition delay correction unit 34b estimates the operation signal acquisition delay and corrects the time A4 (S96). The operation signal acquisition delay correction unit 34b calculates the time A5 by correcting the time A4 to a time in the past by the operation signal acquisition delay time.

[0197] The operation signal receiving unit 11c of the moving body 10 receives the operation signal from the terminal 30 and extracts the time A5 from the received operation signal (S97).

[0198] Next, the operation signal execution delay correction unit 17b corrects and holds the time A5 in consideration of the operation signal execution delay (S98). The operation signal execution delay correction unit 17b estimates the operation signal execution delay and corrects the time A5. The operation signal execution delay correction unit 17b calculates the time A6 by correcting the time A5 to a time in the past by the operation signal execution delay time.

[0199] In the present embodiment, the time A5 may be an example of the second time information, and the time A6 may be an example of the corrected second time information. The corrected second time information is calculated by correcting the second time information based on the time from receiving the operation signal until the control corresponding to the operation signal in the moving body 10 is completed.

[0200] Next, the delay determination unit 12 determines whether the operation signal is delayed (whether an operation signal delay has occurred) based on the time A6 (S99).

[0201] FIG. 15A is a flowchart showing an example of the operation (delay detection method) of the remote control system 1 according to the present embodiment.

[0202] As shown in FIG. 15A, the moving body 10 acquires the moving body information acquisition time (S101). For example, the moving body information acquisition unit 11a acquires, as the moving body information acquisition time, the time when the moving body information is acquired from a camera or the like.

[0203] Next, the moving body 10 acquires the estimated delay time (S102).

[0204] FIG. 15B is a diagram showing the estimated delay times of various delays according to the present embodiment. Delays such as the moving body information acquisition delay and the display delay (terminal screen drawing delay) may not be strictly measurable by the system. Therefore, in the present embodiment, the time required for acquiring the moving body information (moving body information acquisition delay), the time until the terminal 30 acquires the information and draws it on the screen (display delay), the time required for the moving body to execute the operation signal (operation signal execution delay), etc. are estimated based on the table shown in FIG. 15B.

[0205] As shown in FIG. 15B, a delay time is associated in advance for each type of delay. The delay time is, for example, a fixed value. Thereby, it is possible to estimate a rough delay time for each type of delay.

[0206] The moving body information acquisition delay correction unit 17a, the display delay correction unit 34a, the operator reaction delay correction unit 34c, the operation signal acquisition delay correction unit 34b, and the operation signal execution delay correction unit 17b estimate the delay times of the corresponding delays based on, for example, the table shown in FIG. 15B.

[0207] Referring to FIG. 15A again, the moving body 10 rewinds the moving body information acquisition time (time A) by the estimated delay time (S103). Each of the moving body information acquisition delay correction unit 17a, the display delay correction unit 34a, the operator reaction delay correction unit 34c, the operation signal acquisition delay correction unit 34b, and the operation signal execution delay correction unit 17b executes a process of rewinding the received moving body information acquisition time or the corrected moving body information acquisition time by the estimated delay time acquired in step S102.

[0208] FIG. 16 is a flowchart showing another example of the operation (delay detection method) of the remote control system 1 according to the present embodiment. In FIG. 16, the acquisition time of the moving body information is not corrected according to the delay, and the delay is considered by delaying the update timing of the time stamp on the memory.

[0209] As shown in FIG. 16, when the moving body 10 acquires the estimated delay time (S102), it waits for the estimated delay time (S104), and updates the acquisition time of the moving body information on the memory (S105).

[0210] Steps S104 and S105 shown in FIG. 16 will be described with reference to FIGS. 17 and 18. FIG. 17 is a diagram for explaining details of another example of the operation (delay detection method) of the remote control system 1 according to the present embodiment. FIG. 17 shows the flow at the time of replacing the time stamp at the terminal 30 when considering the display delay t1 of the moving body information display unit 33a.

[0211] As shown in FIG. 17, when there is a display delay t1, the drawing of the frame is performed after a delay of the display delay t1 from the time when the frame is received. That is, the operator views the video of the previous frame during the display delay t1 from the time when a new frame is received. Therefore, the time stamp of the previous frame is assigned to the operation performed until the display delay t1 elapses from the time when a new frame is received.

[0212] Taking frame B as an example, after receiving frame B, the moving body information receiving unit 31a waits for the display delay t1 acquired in FIG. 15B, and when the display delay t1 has elapsed, updates the time stamp on the memory from the time stamp of frame A to the time stamp of frame B. In this way, the moving body information receiving unit 31a delays the process of updating the time stamp on the memory to the time stamp of the frame until the start of the screen drawing of the frame. The moving body information receiving unit 31a executes, for example, a process of synchronizing the start of the drawing of frame B and the update of the time stamp of frame B.

[0213] In this case, for example, if an operation is performed by the operator between the time when frame B is received and the time when the display delay t1 has elapsed, the operation signal is given the time stamp of frame A.

[0214] FIG. 18 is a diagram for explaining in further detail still another example of the operation (delay detection method) of the remote control system 1 according to the present embodiment. In FIG. 18, in addition to the display delay t1 of the moving body information display unit 33a, the flow at the time of replacing the time stamp at the terminal 30 is shown when considering the reaction delay t2 of the operator and the delay of the input device 40 (signal acquisition delay t3).

[0215] As shown in FIG. 18, when there are a display delay t1, a reaction delay t2, and a signal acquisition delay t3, after a delay of the display delay t1 from the time when the frame is received, the frame is drawn. Also, the timing at which the operation signal receiving unit 31b receives the operation signal is a timing that is further delayed by the reaction delay t2 and the signal acquisition delay t3 from the display timing of the video. Therefore, for an operation performed between the time when a new frame is received and the display delay t1, the reaction delay t2, and the signal acquisition delay t3, the time stamp of the previous frame is given.

[0216] Taking frame C as an example, after receiving frame C, the moving body information receiving unit 31a waits for the display delay t1, the reaction delay t2, and the signal acquisition delay t3 acquired in FIG. 15B, and when the display delay t1, the reaction delay t2, and the signal acquisition delay t3 have elapsed, updates the time stamp on the memory from the time stamp of frame B to the time stamp of frame C. In this way, the moving body information receiving unit 31a delays the process of updating the time stamp to that of frame C until the display delay t1, the reaction delay t2, and the signal acquisition delay t3 have elapsed. For example, for an operation signal received between the start time of the screen drawing of frame C and the time when the reaction delay t2 and the signal acquisition delay t3 have elapsed, the time stamp of frame B is given.

[0217] (Modification Example 1 of Embodiment 4) Next, the delay detection method and the like according to this modification example will be described with reference to FIGS. 19A to 20. FIG. 19A is a flowchart showing an example of the operation (delay detection method) of the remote control system 1 according to this modification example. In this modification example, the update of the time stamp when the frame rate is low will be described.

[0218] Note that in this modification example, when the frame rate is lower than a certain level, since a person can predict the movement (inter-frame prediction) in their head, control is performed based on the premise that accurate operations can be performed without considering the delay.

[0219] As shown in FIG. 19A, the mobile body information receiving unit 31a determines whether or not a certain time has elapsed since the reception of the previous mobile body information (including the virtual frame) (S111). If it is determined that a certain time has elapsed (yes in S111), the mobile body information receiving unit 31a executes a process of advancing the delay time by a specified time for the previous mobile body information acquisition time stored in the memory (S112). If it is determined that a certain time has not elapsed (no in S111), the mobile body information receiving unit 31a ends the process.

[0220] FIG. 19B is a flowchart showing another example of the operation (delay detection method) of the remote control system 1 according to this modification example.

[0221] As shown in FIG. 19B, when it is determined by the mobile body information receiving unit 31a that a certain time has elapsed (yes in S111), the terminal 30 acquires the time obtained by advancing the previous mobile body information acquisition time stored in the memory by a specified time (S113).

[0222] Next, the terminal 30 acquires an estimated delay time based on, for example, the table shown in FIG. 15B (S114), waits for the estimated delay time (S115), and updates the mobile body information acquisition time on the memory (S116). In step S116, the terminal 30 updates the mobile body information acquisition time on the memory to the time obtained by advancing the previous mobile body information acquisition time by a specified time.

[0223] FIG. 20 is a diagram for explaining another example of the operation (delay detection method) of the remote control system 1 according to this modified example. In FIG. 20, as the delay time, there is a display delay t1, and a timing chart is shown when it is determined that a certain period of time has elapsed at time T after receiving frame B.

[0224] As shown in FIG. 20, since a person can predict the movement between frames, when the movement information receiving unit 31a determines that a certain period of time has elapsed at time T, at time T, the time stamp is updated as if the virtual frame B1 has been received. Note that the drawing of the video is not updated before and after time T.

[0225] In this case, for the operation signals received between the time when the display delay t1 has elapsed from time T and the time when the display delay t1 has elapsed after the next frame C is received, the time stamp of frame B1 is given. Thereby, it is possible to correct the increase in the delay. For example, it is possible to suppress the misjudgment that there is an abnormality (for example, the delay is less than the specified value) but it is misjudged as being delayed due to accidental frame drops or the like.

[0226] (Modification Example 2 of Embodiment 4) Hereinafter, the delay detection method and the like according to this modified example will be described with reference to FIGS. 21A and 21B. FIG. 21A is a flowchart showing an example of the operation (delay detection method) of the remote control system 1 according to this modified example. In this modified example, the update of the time stamp when the frame rate is low will be described.

[0227] Note that in this modified example, when the frame rate is lower than a certain level, since a person can predict the movement (inter-frame prediction) in his / her head, control is performed based on the premise that accurate operations can be performed without considering the delay.

[0228] As shown in FIG. 21A, the mobile body information receiving unit 31a determines whether a frame drop has occurred (S121). If it is determined that a frame drop has occurred (yes in S121), the mobile body information receiving unit 31a executes a process of advancing the delay time by a specified time for the previous mobile body information acquisition time stored in the memory (S112). Further, when it is determined that no frame drop has occurred (no in S121), the mobile body information receiving unit 31a ends the process.

[0229] FIG. 21B is a flowchart showing another example of the operation (delay detection method) of the remote control system 1 according to this modified example.

[0230] As shown in FIG. 21B, when it is determined that a frame drop has occurred (yes in S121), the mobile body information receiving unit 31a executes the processes after step S113 shown in FIG. 19B (S113 to S116).

[0231] The timing chart in this modified example is the same as the timing chart shown in FIG. 20. When a frame drop occurs at time T, assuming that a virtual frame B1 is received at time T, the timestamp is updated.

[0232] (Embodiment 5) Hereinafter, a delay detection method and the like according to this embodiment will be described with reference to FIG. 22. FIG. 22 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example will be described in which an operator determines whether to remotely control the mobile body 10 while referring to the mobile body information transmitted by the mobile body 10.

[0233] As shown in FIG. 22, the attaching unit 15 of the mobile body 10 attaches electronic signature information a to the mobile body information (S131). The electronic signature information a attached to the mobile body information is an example of the first signature information.

[0234] The mobile information receiving unit 31a of the terminal 30 receives mobile information, and obtains and holds the time A and the electronic signature information a from the received mobile information (S132). Further, the operation signal transmitting unit 32 attaches the time A and the electronic signature information a held in step S132 to the operation signal (S133), and transmits the operation signal to which the time A and the electronic signature information a are attached to the mobile body 10 (S17).

[0235] Next, the operation signal receiving unit 11c of the mobile body 10 receives the operation signal from the terminal 30, and extracts the time A and the electronic signature information a from the received operation signal (S134). The electronic signature information a extracted from the operation signal is an example of the second signature information.

[0236] Next, the verification unit 16 verifies the electronic signature information a extracted in step S134 based on the electronic signature information a attached to the mobile information (S135). This is a verification for determining whether the operator is remotely controlling the mobile body 10 while viewing the video transmitted by a mobile body different from the mobile body 10. The verification unit 16 determines, for example, whether the first signature information and the second signature information match. The MRM execution unit 14 executes the MRM (S20) when the verification of the electronic signature information a fails in addition to the case where the operation signal is delayed.

[0237] Thereby, when the operation signal transmitted to the mobile body 10 is an operation signal transmitted to the mobile body 10 after viewing the video transmitted by a mobile body different from the mobile body 10 by the operator, it can be detected by verifying the electronic signature information a. When viewing a video other than the mobile body 10, the operation content may not be suitable for the mobile body 10, so the safety of the mobile body 10 can be ensured by executing the MRM.

[0238] Note that the second signature information may be transmitted from the terminal 30, for example, at the start of remote control.

[0239] (Modification Example 1 of Embodiment 5) Hereinafter, the delay detection method and the like according to this modification example will be described with reference to FIG. 23. FIG. 23 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to this modification example. In this modification example, an example will be described in which it is determined whether or not the terminal 30 that has sent a switching signal (for example, the operation start signal shown in FIG. 9) for switching from automatic driving to remote operation driving is performing remote operation. Note that hereinafter, the description will mainly focus on the differences from FIG. 22.

[0240] As shown in FIG. 23, at the time of remote operation switching, the operator inputs to the input device 40 an input indicating switching to remote operation. The remote operation switching unit 35 of the terminal 30 acquires (for example, receives) a switching signal indicating switching to remote operation from the input device 40 (S141).

[0241] Next, when the remote operation switching unit 35 receives the switching signal, the encryption unit 36 generates an encryption key and a decryption key (S142). The encryption unit 36 generates a pair of an encryption key and a decryption key.

[0242] Next, the remote operation switching unit 35 transmits the switching signal and the decryption key to the moving body 10 (S143).

[0243] Next, when the decryption unit 18a of the moving body 10 receives the decryption key, it holds the received decryption key (S144).

[0244] Next, when the operation signal receiving unit 11c receives the switching signal, it starts receiving the operation signal from the terminal 30 (S145). That is, the moving body 10 is controlled by remote operation.

[0245] The encryption unit 36 of the terminal 30 encrypts the operation signal to which the time A and the electronic signature information a are attached using the encryption key generated in step S142 (S146). The operation signal transmitting unit 32 transmits the encrypted operation signal (S17).

[0246] When the decryption unit 18a of the mobile body 10 receives the encrypted operation signal, it decrypts the operation signal with the latest decryption key it holds (S147). When the decryption is successful, the time A and the electronic signature information a can be extracted from the operation signal.

[0247] In this way, among the pair of the encryption key and the decryption key generated by the terminal 30 that transmitted the switching signal, the terminal 30 holds the encryption key, and the mobile body 10 holds the decryption key. Since the operation signal encrypted and transmitted from the terminal 30 can be decrypted with the decryption key of the mobile body 10, it becomes possible to confirm whether the terminal 30 that sent the switching signal is remotely operating the mobile body 10, that is, whether the received operation signal was transmitted from the terminal 30 that sent the switching signal.

[0248] (Modification Example 2 of Embodiment 5) Hereinafter, the delay detection method and the like according to this modification example will be described with reference to FIG. 24. FIG. 24 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to this modification example. In this modification example, an example of ensuring the security in the remote control system 1 in the mobile body 10 will be described. Specifically, an example of confirming that the operation signal from the terminal 30 has not been tampered with in the mobile body 10 will be described.

[0249] As shown in FIG. 24, the vehicle encryption unit 18b generates two pairs of an encryption key and a decryption key (S151) during the remote operation by the terminal 30, and holds the decryption keys I and II (S152). In the example of FIG. 24, the vehicle encryption unit 18b generates the encryption keys I and II as the encryption keys, and generates the decryption keys I and II as the decryption keys. The encryption key I and the decryption key I are a pair, and the encryption key II and the decryption key II are a pair.

[0250] Next, the vehicle encryption unit 18b encrypts the time (time A) when the vehicle information is acquired with the encryption key I (S153).

[0251] Next, the mobile body information acquisition unit 11a assigns the encrypted time (time A) to the mobile body information (S154), and further assigns the encryption key II to the mobile body information (S155).

[0252] When the mobile body information receiving unit 31a of the terminal 30 receives the mobile body information, it acquires and holds the encrypted time A and the encryption key II (S156).

[0253] The operation signal transmission unit 32 assigns the encrypted time A acquired in step S156 to the operation signal (S157).

[0254] Next, the encryption unit 36 encrypts the operation signal with the encryption key II (S158). The operation signal transmission unit 32 transmits the encrypted operation signal (S17).

[0255] Next, the decryption unit 18a of the mobile body 10 decrypts the operation signal with the held decryption key II (S159).

[0256] Next, when the operation signal reception unit 11c succeeds in decryption, it extracts the time A from the operation signal (S160). At this point, the time A is encrypted.

[0257] Next, the verification unit 16 decrypts the encrypted time A with the decryption key I (S161). As a result, when the decryption is successful, the time A can be acquired. The MRM execution unit 14 executes the MRM when the decryption of the time A fails in addition to the case where the operation signal is delayed (S20).

[0258] Thus, in this modified example, the time information to be assigned to the mobile body information is encrypted with the encryption key I generated by the mobile body 10, and further the operation signal from the terminal 30 is encrypted with the encryption key II generated by the mobile body 10. Thereby, the mobile body 10 can confirm that the operation signal transmitted from the terminal 30 is by the operation of the operator who has viewed the video transmitted by the mobile body 10 and that it has not been tampered with.

[0259] (Embodiment 6) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 25. FIG. 25 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example in which the moving body 10 determines whether data can be transmitted and received in order will be described.

[0260] As shown in FIG. 25, the assigning unit 15 of the moving body 10 assigns a counter to the moving body information (S171) and increments the counter (S172). In step S172, the assigning unit 15 increments the counter by +1. The incremented counter is assigned to the next acquired moving body information. The counter assigned to the moving body information is an example of the first counter information.

[0261] The moving body information receiving unit 31a of the terminal 30 receives the moving body information, acquires and holds the time A and the counter from the received moving body information (S173). Further, the operation signal transmitting unit 32 assigns the time A and the counter held in step S173 to the operation signal (S174).

[0262] When the operation signal receiving unit 11c receives the operation signal, it extracts the time A and the counter (S175). The counter included in the operation signal is an example of the second counter information.

[0263] Next, the verification unit 16 verifies the counter included in the operation signal (S176). This is a verification for determining whether a plurality of operation signals are received in order. When the counter extracted in step 175 is smaller than the held counter, the verification unit 16 determines that the order (reception order) of the operation signals is abnormal, that is, a delay has occurred in the operation signal. When the order of the operation signals is swapped, the MRM is executed. The held counter may be, for example, the counter assigned to the previous operation signal.

[0264] (Embodiment 7) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 26. FIG. 26 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example in which the MRM is released when the moving body 10 recovers from the delayed state will be described.

[0265] The MRM execution unit 14 determines whether or not the delay has become equal to or less than the specified value during the execution of the MRM in the moving body 10. If the delay has become equal to or less than the specified value (recovered), the MRM execution state is released (S181). It can also be said that the MRM execution unit 14 cancels the execution of the MRM when the delay is eliminated during the execution of the MRM.

[0266] In this way, the moving body 10 determines whether or not to release the MRM execution state. If it is determined to release the MRM execution state, it automatically resumes the automatic driving.

[0267] (Embodiment 8) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 27. FIG. 27 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example in which the control server 20 relays the moving body information and the operation signal will be described.

[0268] As shown in FIG. 27, the moving body information transmission unit 11b transmits the moving body information to the control server 20 (S12a), and the moving body information transmission unit 22 of the control server 20 transmits the moving body information to the terminal 30 (S12b). Further, the operation signal transmission unit 32 transmits the operation signal to the control server 20 (S17a), and the operation signal transmission unit 23 of the control server 20 transmits the operation signal to the moving body 10 (S17b).

[0269] When a delay occurs in the operation signal, the MRM execution unit 14 executes the MRM (S20) and transmits MRM information indicating that the MRM has been executed to the control server 20 (S191).

[0270] Next, when the control state management unit 21 of the control server 20 receives the MRM information, it cuts off the operation signal transmission system (S192). If a delay occurs in the operation signal, the control state management unit 21 prohibits the relay of the operation signal from the terminal 30 to the moving body 10.

[0271] In this way, when the control server 20 relays and manages the information exchange between the moving body 10 and the terminal 30, when the MRM is executed on the moving body 10, the operation signal from the terminal 30 is blocked.

[0272] (Embodiment 9) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIGS. 28 to 29B. FIG. 28 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example in which the delay determination process of the operation signal is statistically processed will be described.

[0273] As shown in FIG. 28, in the delay determination of the operation signal, the delay determination unit 12 uses statistical processing (S201). The delay determination unit 12 statistically processes, for example, the sum of the delay from when the moving body information is acquired from the moving body 10 until the moving body information is presented to the operator and the delay from when the operator operates until it is reflected in the moving body 10. When the statistically processed value does not satisfy a predetermined condition, the MRM execution unit 14 executes the MRM (S20).

[0274] Here, a specific processing example of step S201 will be described with reference to FIGS. 29A and 29B. FIGS. 29A and 29B are flowcharts showing examples of the operation (delay detection method) of the remote control system 1 according to the present embodiment.

[0275] As shown in FIG. 29A, the delay determination unit 12 acquires the moving body information acquisition time from the received operation signal and calculates the delay (S211).

[0276] Next, the delay determination unit 12 counts the number of times that exceeded 1.5 seconds among the delay times for the past N times (S212). 1.5 seconds is an example of a predetermined time. Note that the predetermined time is not limited to 1.5 seconds and may be other values.

[0277] Next, the delay determination unit 12 determines whether the number of times is N times or more (S213). If it is N times or more (yes in S213), it determines that a delay has occurred (S214). If it is less than N times (no in S213), since no delay has occurred, the process ends.

[0278] In this way, the statistical process may be to calculate the number of times a delay of a predetermined time or more has been calculated.

[0279] As shown in FIG. 29B, after step S211, the delay determination unit 12 averages the delay times for the past 5 times (S215). That is, the delay determination unit 12 calculates the average value (average delay) of the delay times. 5 times is an example of a predetermined number of times. Note that the predetermined number of times is not limited to 5 times and may be other values.

[0280] Next, the delay determination unit 12 determines whether the average delay is greater than or equal to a specified value (S216). If the average delay is greater than or equal to the specified value (yes in S216), it determines that a delay has occurred (S214). If the average delay is less than the specified value (no in S216), since no delay has occurred, the process ends.

[0281] In this way, the statistical process may be to calculate the average value of the delay times over a predetermined period. Note that it is not limited to calculating the average value, and for example, the mode, median, etc. may be calculated.

[0282] Thereby, it is possible to suppress the determination that a delay has occurred when an irregular delay suddenly occurs. Also, the processing amount of delay determination in the moving body 10 can be reduced.

[0283] Note that the statistical process may include calculating the variance of the delay time within a predetermined period. In this case, when the variance of the delay time is equal to or greater than a specified value, the delay determination unit 12 may determine that it is abnormal because a certain operation cannot be performed and the operability of the moving body 10 may deteriorate. When it is determined to be abnormal, for example, MRM is executed. Instead of the determination in step S216, the delay determination unit 12 may perform a variance determination, or for example, when it is determined to be "no" in step S216, a variance determination may be further performed.

[0284] (Embodiment 10) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 30. FIG. 30 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, an example considering the execution delay of the moving body 10 will be described.

[0285] As shown in FIG. 30, when the operation signal execution unit 13 receives an operation signal, it executes the operation signal (S221). Further, the moving body information acquisition unit 11a acquires moving body information (S222). The delay determination unit 12 calculates the time until the result of the operation signal is reflected based on the moving body information acquired in step S222, and performs a delay determination based on the calculated time. For example, when the operation signal is to increase the rotation speed of the motor by a predetermined number, the time until the instructed rotation speed is reached is calculated as the delay.

[0286] Note that the current time shown in step S19 may be, for example, the time when it is observed that the operation by the operation signal executed in step S221 is reflected in the moving body 10. Further, when a delay occurs, the MRM execution unit 14 executes MRM (S20).

[0287] Thereby, when the time from when the operator operates until it is observed that the operation is reflected in the moving body 10 is equal to or longer than a certain time, MRM can be executed.

[0288] (Embodiment 11) Hereinafter, the delay detection method and the like according to the present embodiment will be described with reference to FIG. 31. FIG. 31 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to the present embodiment. In the present embodiment, the case where the operator performs remote assistance will be described. In remote assistance, the control signal is a single shot (not transmitted at regular intervals).

[0289] Steps S11 to S234 shown in FIG. 31 show the processing in a state where the operator (remote operator) is watching the video in order to operate a button or the like.

[0290] In this case, since the operation signal transmission unit 32 does not receive an operation signal from the input device 40, it assigns the time A to the time signal (S231) and transmits it to the moving body 10 (S232). The time signal is a signal including the time (here, time A, which is an example of the second time information) used to determine whether a delay has occurred during the remote assistance by the operator. The time signal does not include information (for example, a control command) for remotely controlling the moving body 10.

[0291] The delay determination unit 12 determines the time signal delay based on the received time signal (S233). The delay determination unit 12 determines whether there is a delay in the time signal. The delay determination unit 12 makes the determination in step S233 based on whether the value obtained by subtracting time A from the current time when the time signal is received is greater than a specified value. If the subtracted value is greater than the specified value, it is determined that a delay has occurred, and the MRM is executed (S234).

[0292] In this way, when a delay occurs at the stage of checking the video for remote assistance, the MRM is executed.

[0293] Also, when the operation signal transmission unit 32 of the terminal 30 acquires a single-shot operation signal by the operation signal reception unit 31b (S15), it assigns the time A to the operation signal (S16) and transmits the operation signal to the moving body 10 (S17). It can also be said that the acquisition time of the video is transmitted to the moving body 10.

[0294] The operation signal execution unit 13 executes the operation signal (S235).

[0295] Next, the delay determination unit 12 determines the time signal delay (S236). The delay determination unit 12 makes the determination in step S236, for example, based on whether the value obtained by subtracting time A from the current time is greater than a specified value. If the subtracted value is greater than the specified value, it is determined that a delay has occurred, and the MRM is executed (S20).

[0296] (Embodiment 12) Hereinafter, the delay detection method and the like according to this embodiment will be described with reference to FIG. 32. FIG. 32 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, the case where the internal time in the moving body 10 is rolled back will be described. In remote assistance, the control signal is single-shot (not transmitted at regular intervals).

[0297] As shown in FIG. 32, the time management unit 19 checks whether the internal time of the moving body 10 has been rolled back (S241).

[0298] Next, when it is determined that the internal time of the moving body 10 has been rolled back (that is, when there is a difference in the internal time with respect to the reference time (for example, GPS time)), the MRM execution unit 14 executes the MRM (S242). The MRM execution unit 14 determines, for example, whether the internal time of the moving body 10 has been rolled back by a predetermined time or more. The predetermined time is set in advance.

[0299] (Modification of Embodiment 12) Hereinafter, the delay detection method and the like according to this embodiment will be described with reference to FIG. 33. FIG. 33 is a sequence diagram showing the operation (delay detection method) of the remote control system 1 according to this modification. In this modification, the operation when the time management unit 19 is independent will be described. In remote assistance, the control signal is single-shot (not transmitted at regular intervals).

[0300] As shown in FIG. 33, when the moving body 10 acquires moving body information (S251), the time management unit 19 assigns the acquisition time (time A) to the moving body information (S252). Also, after step S18, the time management unit 19 assigns the acquisition time (time A) to the moving body 10 (S253).

[0301] Also, the time management unit 19 checks whether the internal time has been rolled back or the like (S255). The time management unit 19 determines, for example, whether the internal time of the moving body 10 has been rolled back by a predetermined time or more. The predetermined time is set in advance.

[0302] (Other embodiments) As described above, the delay detection method and the like according to one or more aspects have been described based on the embodiments and the like. However, the present disclosure is not limited to these embodiments and the like. As long as the gist of the present disclosure is not deviated from, various modifications conceived by those skilled in the art applied to this embodiment or forms constructed by combining components in different embodiments may also be included in the present disclosure.

[0303] For example, in the above embodiments and the like, the correction of each delay may be executed in either the moving body 10 or the terminal 30. For example, the moving body 10 may perform the correction of each delay collectively, or the terminal 30 may perform the correction collectively. Also, for example, the control server 20 may perform the correction of each delay.

[0304] Also, in the above embodiments and the like, the remote control system 1 has been described by taking the example of including the control server 20, but the present disclosure is not limited thereto. For example, the moving body 10 and the terminal 30 may be able to communicate directly without going through the control server 20, or the moving body 10 and the terminal 30 may have the functions of the control server 20.

[0305] In the above-described embodiments and the like, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0306] Also, the order in which each step in the flowchart is executed is for illustrative purposes in order to specifically describe the present disclosure, and may be an order other than the above. Also, a part of the above steps may be executed simultaneously (in parallel) with other steps, or a part of the above steps may not be executed.

[0307] Also, the division of the functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or a part of the functions may be transferred to other functional blocks. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or time-divisionally.

[0308] Also, the delay detection system or the terminal 30 included in the mobile body 10 according to the above-described embodiments and the like may be realized as a single device or may be realized by a plurality of devices. When the delay detection system or the terminal 30 is realized by a plurality of devices, each component included in the delay detection system or the terminal 30 may be distributed among the plurality of devices in any manner. When the delay detection system or the terminal 30 is realized by a plurality of devices, the communication method between the plurality of devices is not particularly limited, and may be wireless communication or may be wired communication. Also, wireless communication and wired communication may be combined between the devices.

[0309] In addition, each component described in the above embodiments may be implemented as software, or typically, may be implemented as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include some or all of them. Here, an LSI is used as an example, but depending on the degree of integration, it may also be referred to as an IC, a system LSI, a super LSI, or an ultra LSI. Also, the method of integrating into a circuit is not limited to an LSI, and it may be implemented by a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After manufacturing the LSI, an FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that can reconfigure the connection or setting of circuit cells inside the LSI may be used. Furthermore, if a technology for integrating into a circuit that replaces an LSI appears due to the progress of semiconductor technology or other derived technologies, of course, the components may be integrated using that technology.

[0310] A system LSI is a super multi-functional LSI manufactured by integrating a plurality of processing units on one chip. Specifically, it is a computer system including a microprocessor, a ROM, a RAM, etc. A computer program is stored in the ROM. When the microprocessor operates according to the computer program, the system LSI achieves its function.

[0311] Also, one aspect of the present disclosure may be a computer program that causes a computer to execute each characteristic step included in the delay detection method shown in any of FIGS. 4 to 6E, FIGS. 12 to 15A, FIG. 16, FIGS. 19A, 19B, FIGS. 21A to 33.

[0312] Also, for example, the program may be a program for causing a computer to execute. Further, one aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and by causing the processor to execute the program, it becomes possible to cause the device to perform each of the above processes.

Industrial Applicability

[0313] The present disclosure is useful for a remote control system or the like for remotely controlling a moving body.

Explanation of Signs

[0314] 1 Remote control system 10 Moving body 11a Moving body information acquisition unit 11b Moving body information transmission unit (transmission unit) 11c Operation signal reception unit (reception unit) 12 Delay determination unit (determination unit) 13 Operation signal execution unit 14 MRM execution unit 15 Granting unit 16 Verification unit 17a Moving body information acquisition delay correction unit 17b Operation signal execution delay correction unit 18a Decryption unit 18b Vehicle encryption unit 19 Time management unit 20 Control server 21 Control state management unit 22 Moving body information transmission unit 23 Operation signal transmission unit 30 Terminal 31a Moving body information reception unit 31b Operation signal reception unit 32 Operation signal transmission unit 33a Moving body information display unit 33b Delay time display unit 34a represents a delay correction unit 34b is an operation signal acquisition delay correction unit 34c is an operator response delay correction unit 35 is a remote operation switching unit 36 is an encryption unit 40 is an input device a is electronic signature information A and T times N is a mobile body information transmission system t1 is a display delay t2 is a reaction delay t3 is a signal acquisition delay

Claims

1. A delay detection method performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; The first time information includes a time obtained by correcting a time at which predetermined information including a video included in the moving body information was acquired based on a time from when an imaging device mounted on the moving body starts capturing the video to when a device executing the delay detection method acquires the predetermined information. Delay detection method.

2. A delay detection method implemented by a mobile object that can be remotely controlled by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; adding first counter information to the mobile unit information and transmitting the information to the terminal of the operator; receiving the operation signal further including second counter information; The second counter information is verified. Delay detection method.

3. In detecting the delay, a difference between the second time information included in the most recently received operation signal and the current time is periodically calculated. The method for detecting delay according to claim 1 or 2.

4. When the delay is detected, causing the moving object to perform a predetermined operation; The predetermined operation includes executing a Minimal Risk Maneuver (MRM) or ignoring a control command that is included in the received operation signal and is based on the operation by the operator. The method for detecting delay according to claim 1 or 2.

5. When the internal time of the moving body differs from the reference time by a predetermined amount or more, the moving body executes a predetermined operation; The predetermined operation includes executing a Minimal Risk Maneuver (MRM) or ignoring a control command that is included in the received operation signal and is based on the operation by the operator. The method for detecting delay according to claim 1 or 2.

6. The first time information includes a time obtained by correcting a time at which predetermined information including a video included in the moving body information was acquired based on a time from when an imaging device mounted on the moving body starts capturing the video to when a device executing the delay detection method acquires the predetermined information. The method of detecting delay according to claim 2 .

7. The second time information includes a time obtained by correcting the first time information based on a time from when the mobile object information is received to when it is displayed at the terminal. The method for detecting delay according to claim 1 or 2.

8. The second time information is corrected based on a time from when the operation signal is received until the control in the moving body according to the operation signal is completed, and a delay in the remote control system is detected based on the corrected second time information and the predetermined time. The method for detecting delay according to claim 1 or 2.

9. adding first signature information to the mobile unit information and transmitting the mobile unit information to the terminal of the operator; receiving the operation signal further including second signature information; The second signature information is verified based on the first signature information. The method for detecting delay according to claim 1 or 2.

10. The second signature information is information transmitted from the terminal at the start of remote control. The method of detecting delay according to claim 9.

11. adding first counter information to the mobile unit information and transmitting the information to the terminal of the operator; receiving the operation signal further including second counter information; The second counter information is verified. The method of detecting delay according to claim 1 .

12. If the delay is eliminated while the MRM is being executed, the execution of the MRM is cancelled. The method of detecting delay according to claim 4.

13. When the operator does not remotely control the moving body, a time signal including the second time information is received from the terminal. The method for detecting delay according to claim 1 or 2.

14. encrypting the first time information using a first encryption key, adding the encrypted first time information to the mobile unit information and transmitting the information to the terminal; The second time information included in the received operation signal is decrypted using a first decryption key corresponding to the first encryption key. The method for detecting delay according to claim 1 or 2.

15. A second encryption key is further added to the mobile unit information and the mobile unit information is transmitted to the terminal; receiving the operation signal encrypted using the second encryption key; Decrypting the received operation signal using a second decryption key corresponding to the second encryption key. The method for detecting delay according to claim 1 or 2.

16. The time indicated by the first time information and the time indicated by the second time information are the same time. The method for detecting delay according to claim 1 or 2.

17. A delay detection device provided in a mobile object that can be remotely controlled by an operator, a transmitting unit that adds first time information to mobile object information indicating a state of the mobile object and transmits the mobile object information to the operator terminal; a receiving unit that receives an operation signal based on an operation of the operator on the moving object in response to the moving object information, the operation signal including second time information based on the first time information included in the moving object information; a determination unit that detects a delay in a remote control system including the moving object and the terminal based on the second time information and a predetermined time, the predetermined time is the current time, The determination unit calculates a difference between the second time information and the current time, and detects the delay based on the calculated difference; The first time information includes a time obtained by correcting a time at which predetermined information including a video included in the moving body information was acquired based on a time from when an imaging device mounted on the moving body starts capturing the video to when the delay detection device acquires the predetermined information. Delay detection device.

18. A delay detection device provided on a mobile body that can be remotely controlled by an operator, comprising: a transmitting unit that adds first time information to mobile object information indicating a state of the mobile object and transmits the mobile object information to the operator terminal; a receiving unit that receives an operation signal based on an operation of the operator on the moving object in response to the moving object information, the operation signal including second time information based on the first time information included in the moving object information; a determination unit that detects a delay in a remote control system including the moving object and the terminal based on the second time information and a predetermined time, the predetermined time is the current time, The determination unit calculates a difference between the second time information and the current time, and detects the delay based on the calculated difference; The transmission unit further adds first counter information to the mobile object information and transmits the mobile object information to the terminal of the operator; The receiving unit receives the operation signal further including second counter information, The determination unit verifies the second counter information. Delay detection device.

19. A program for causing a computer to execute the delay detection method according to claim 1 or 2.

20. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; The detection of the delay includes periodically calculating a difference between the second time information included in the most recently received operation signal and the current time; The delay is detected after remote control is executed on the moving object based on the received operation signal. Delay detection method.

21. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; When the delay is detected, the mobile unit is caused to execute MRM, and MRM information indicating that the MRM has been executed is transmitted to a server that relays communication between the mobile unit and the terminal, thereby prohibiting the server from transmitting an operation signal from the terminal to the mobile unit. Delay detection method.

22. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; After the delay is detected, it is determined whether the internal time of the mobile unit differs from the reference time by a predetermined amount or more, and if the difference is greater than the predetermined amount, the mobile unit is caused to execute a predetermined operation. Delay detection method.

23. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; When the delay is detected, the mobile unit executes MRM, and when the delay is eliminated during the execution of the MRM, the execution of the MRM is cancelled and automatic driving is restored. Delay detection method.

24. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; When the delay is detected in the moving body, the moving body executes MRM based on the judgment of the moving body. Delay detection method.

25. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; the operation signal includes a control command based on the operation by the operator, When the delay is detected, the MRM is not executed, and the control command included in the operation signal received by the moving body is ignored. Delay detection method.

26. A method for detecting delays performed by a mobile object remotely controllable by an operator, comprising: adding first time information to mobile object information indicating a state of the mobile object and transmitting the mobile object information to the operator's terminal; receiving an operation signal based on an operation of the operator on the moving body in response to the moving body information, the operation signal including second time information based on the first time information included in the moving body information; Detecting a delay in a remote control system including the mobile object and the terminal based on the second time information and a predetermined time; the predetermined time is the current time, The detection of the delay includes calculating a difference between the second time information and the current time, and detecting the delay based on the calculated difference; the time indicated by the first time information and the time indicated by the second time information are the same time, The second time information included in the operation signal is the first time information included in the latest mobile object information of the mobile object information received by the terminal at the time of transmitting the operation signal. Delay detection method.

Images