Delay detection method, delay detection device, and program

Abstract

The delay detection method is executed by a mobile body that can be remotely controlled by an operator, and includes: adding first time information to mobile body information indicating a state of the mobile body (10) and transmitting the same to a terminal (30) of the operator (S2); receiving an operation signal including remote control content of the operator with respect to the mobile body information and including second time information based on the first time information included in the mobile body information (S6); on the basis of the second time information and the predetermined time, a delay in a remote control system including the mobile body (10) and the terminal (30) is detected (S7).

Description

Technical Field

[0001] This disclosure relates to delay detection methods, delay detection devices, and procedures. Background Technology

[0002] In recent years, unmanned material handling solutions enabled by autonomous vehicles have attracted attention. Furthermore, even in non-autonomous driving scenarios, transportation services utilizing vehicles that can be fully remotely controlled are gaining attention from the perspective of saving manpower.

[0003] Achieving Level 4 autonomous driving without human intervention in such vehicles remains challenging. Therefore, we will first explore remote operation with human intervention when necessary. Signal interaction during remote operation requires unstable networks such as wireless, necessitating consideration of safety measures in case of network outages or delays.

[0004] Patent Document 1 discloses a system comprising an industrial vehicle having a vehicle communication unit for wireless communication and a remote communication unit for wireless communication with the vehicle communication unit, and determining the communication delay between the system and a remote operation device used for remote operation of the industrial vehicle.

[0005] Prior technology documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent No. 6943143 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, the technology in Patent Document 1 uses the delay time corresponding to the difference between the reception period required for the vehicle communication unit to receive multiple remote operation signals and the generation period required to generate multiple remote operation signals, thus making it impossible to monitor the delay of the entire system. When performing remote operation, it is sometimes desirable to monitor the delay of the entire system.

[0010] Therefore, this disclosure provides a delay detection method, delay detection device, and program that can more accurately monitor the delay of the entire system.

[0011] Methods for solving problems

[0012] One aspect of this disclosure relates to a delay detection method performed by a mobile body that can be remotely controlled by an operator, comprising: attaching first time information to mobile body information representing the state of the mobile body and sending it to the operator's terminal; receiving an operation signal, the operation signal being based on an operation by the operator to the mobile body in response to the mobile body information, and including second time information based on the first time information contained in the mobile body information; and detecting a delay in a remote control system comprising the mobile body and the terminal based on the second time information and a predetermined time.

[0013] One aspect of this disclosure relates to a delay detection device for a mobile body that can be remotely controlled by an operator. The device comprises: a transmitting unit that appends first time information to mobile body information representing the state of the mobile body and transmits it to the operator's terminal; a receiving unit that receives an operation signal based on an operation performed by the operator on the mobile body information and including second time information based on the first time information contained in the mobile body information; and a determining unit that detects a delay in a remote control system comprising the mobile body and the terminal based on the second time information and a predetermined time.

[0014] One aspect of this disclosure relates to a program that causes a computer to perform the aforementioned delay detection method.

[0015] Invention Effects

[0016] According to one aspect of this disclosure, a delay detection method can be implemented to more accurately monitor the delay of the entire system. Attached Figure Description

[0017] Figure 1 This is a diagram used to illustrate the types of delays.

[0018] Figure 2 This is a block diagram illustrating the functional configuration of the remote control system according to Embodiment 1.

[0019] Figure 3 This is a diagram illustrating the outline of the operation of the remote control system according to Embodiment 1.

[0020] Figure 4 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 1.

[0021] Figure 5A This is a flowchart illustrating the first action in the moving body according to Embodiment 1.

[0022] Figure 5B This is a flowchart illustrating the second action in the terminal according to Embodiment 1.

[0023] Figure 5C This is a flowchart illustrating the third action in the terminal according to Embodiment 1.

[0024] Figure 5D This is a flowchart illustrating the fourth action in the moving body according to Embodiment 1.

[0025] Figure 5E This is a flowchart illustrating the fifth action in the moving body according to Embodiment 1.

[0026] Figure 6A This is a flowchart illustrating a specific example of the first action in a moving body according to Embodiment 1.

[0027] Figure 6B This is a flowchart illustrating a specific example of the second action in the terminal according to Embodiment 1.

[0028] Figure 6C This is a flowchart illustrating a specific example of a third action in a terminal according to Embodiment 1.

[0029] Figure 6D This is a flowchart illustrating a specific example of the fourth action in a moving body according to Embodiment 1.

[0030] Figure 6E This is a flowchart illustrating a specific example of the fifth action in a moving body according to Embodiment 1.

[0031] Figure 7 This is a diagram used to illustrate the details of the operation of the remote control system involved in Embodiment 1.

[0032] Figure 8 This is a diagram illustrating a message example of an operation signal related to Embodiment 1.

[0033] Figure 9 This is a diagram illustrating an example of an operation start signal according to Embodiment 1.

[0034] Figure 10 This is a diagram illustrating an example of an operation end signal according to Embodiment 1.

[0035] Figure 11 This is a diagram illustrating an example of an operator notification signal according to Embodiment 1.

[0036] Figure 12 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 2.

[0037] Figure 13 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 3.

[0038] Figure 14This is a timing diagram illustrating the operation of the remote control system according to Embodiment 4.

[0039] Figure 15A This is a flowchart illustrating an example of the operation of the remote control system according to Embodiment 4.

[0040] Figure 15B This is a graph showing the estimated delay times for various delays involved in Implementation 4.

[0041] Figure 16 This is a flowchart illustrating another example of the operation of the remote control system involved in Embodiment 4.

[0042] Figure 17 This is a diagram illustrating details of another example of the operation of the remote control system involved in Embodiment 4.

[0043] Figure 18 This is a diagram illustrating details of another example of the operation of the remote control system involved in Embodiment 4.

[0044] Figure 19A This is a flowchart illustrating an example of the operation of the remote control system involved in a variation of Embodiment 4, Example 1.

[0045] Figure 19B This is a flowchart illustrating another example of the operation of the remote control system involved in Variation 1 of Embodiment 4.

[0046] Figure 20 This is a diagram illustrating details of another example of the operation of the remote control system involved in Variation 1 of Embodiment 4.

[0047] Figure 21A This is a flowchart illustrating an example of the operation of the remote control system involved in Variation 2 of Embodiment 4.

[0048] Figure 21B This is a flowchart illustrating another example of the operation of the remote control system involved in Variation 2 of Embodiment 4.

[0049] Figure 22 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 5.

[0050] Figure 23 This is a timing diagram showing the operation of the remote control system involved in a variation of embodiment 5, example 1.

[0051] Figure 24 This is a timing diagram showing the operation of the remote control system involved in Variation 2 of Embodiment 5.

[0052] Figure 25 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 6.

[0053] Figure 26 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 7.

[0054] Figure 27 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 8.

[0055] Figure 28 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 9.

[0056] Figure 29A This is a flowchart illustrating an example of the operation of the remote control system according to Embodiment 9.

[0057] Figure 29B This is a flowchart illustrating another example of the operation of the remote control system according to Embodiment 9.

[0058] Figure 30 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 10.

[0059] Figure 31 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 11.

[0060] Figure 32 This is a timing diagram illustrating the operation of the remote control system according to Embodiment 12.

[0061] Figure 33 This is a timing diagram illustrating the operation of the remote control system involved in a variation of embodiment 12. Detailed Implementation

[0062] (The process of reaching this agreement is made public)

[0063] Before describing the embodiments of this disclosure, etc., refer to Figure 1 Explain the process by which this was made public.

[0064] As described in the "Background Art," the remote control of autonomous vehicles by an operator when necessary is being explored. When remotely controlling an autonomous vehicle, the operator needs to identify the surrounding conditions in real time and reflect the operator's actions based on that identification in the behavior of the moving body with a sufficiently short delay.

[0065] Therefore, it is sometimes desirable to monitor the transmission latency of the entire system, including the transmission of operating signals (control signals) used for remote control and vehicle information such as images required to determine operation.

[0066] Furthermore, remote control includes remote operation and remote monitoring. Remote operation refers to using input devices such as a steering wheel to control the movement of a moving body. Additionally, remote assistance refers to allowing the movement of a moving body without using input devices such as a steering wheel. In the case of remote assistance, the moving body drives automatically. An example of remote assistance is outputting a driving start instruction when a stationary moving body begins to move.

[0067] Here, refer to Figure 1 This describes the latency in a system that includes terminals for both the moving body and the operator. Figure 1 This is a diagram used to illustrate the types of delays.

[0068] like Figure 1 As shown, examples of delays include camera delay, transmission delay in an image transmission system, display delay, response delay, signal acquisition delay, transmission delay in a remote operation signal transmission system, and execution delay.

[0069] Video recording delay is the delay required for a camera or other imaging device mounted on a mobile device to record video. For example, video recording delay is the time from when the imaging device starts recording until the captured image (an example of mobile device information) is received by the terminal.

[0070] Transmission delay in an image transmission system refers to the delay in communication when transmitting mobile information, including images, from a mobile body such as a vehicle to an operator's terminal. For example, transmission delay in an image transmission system is the time from when the mobile body sends mobile information to when the terminal receives it.

[0071] Display latency is the processing delay from when the image contained in the mobile information received by the terminal is displayed on the display unit. Display latency can be, for example, the time from when the terminal receives the mobile information to when that mobile information is displayed on the display unit.

[0072] Response delay is the time between the operator's confirmation of the movement information displayed on the display and the operator's subsequent operation for remote control. For example, response delay is the time from when the movement information is displayed on the display to when the operator operates the input device.

[0073] Signal acquisition delay is the delay before a terminal receives the operation signal corresponding to the operator's input. For example, signal acquisition delay is the time from when the operator inputs an operation into the input device to when the terminal receives the operation signal.

[0074] Operation signals are signals based on operator actions on a mobile device, used by an operator to remotely control the device via a terminal. These signals include remote control content (e.g., control commands) of the operator's actions on input devices. Operation signals may also include control values ​​such as motor speed, velocity (throttle or brake), and steering angle.

[0075] In a remote operation signal transmission system, the transmission delay is the communication delay in transmitting the operation signal from the terminal to the mobile unit. For example, the transmission delay in a remote operation signal transmission system is the time from when the terminal sends the operation signal to when the mobile unit receives the operation signal.

[0076] Execution delay is the delay from when the mobile body receives the operation signal until it actually completes the control corresponding to the operation. Execution delay is the time from when the mobile body receives the operation signal to when the control corresponding to the operation signal is completed (e.g., the speed of the mobile body's motor reaches the speed contained in the operation signal).

[0077] Therefore, various delays may exist in the system. Patent Document 1 mainly considers transmission delays in remote operation signal transmission systems, but does not consider other delays.

[0078] Furthermore, if other systems such as ping / pong are used for latency measurement, there is concern that the detection of image / operation latency may be delayed in the event of sudden delays.

[0079] Therefore, the inventors of this application have conducted in-depth research on delay detection methods that can more accurately monitor the delay of the entire system, and have proposed the delay detection methods shown below.

[0080] The first aspect of this disclosure relates to a delay detection method performed by a mobile body that can be remotely controlled by an operator, comprising: attaching first time information to mobile body information representing the state of the mobile body and sending it to the operator's terminal; receiving an operation signal, the operation signal being based on an operation performed by the operator on the mobile body information and including second time information based on the first time information contained in the mobile body information; and detecting a delay in a remote control system including the mobile body and the terminal based on the second time information and a predetermined time.

[0081] Therefore, the mobile unit can detect delays by considering at least the round-trip transmission delays in the remote control system. Consequently, according to the delay detection method, the overall delay of the remote control system can be monitored more accurately compared to using only the transmission delay of one side of the round trip. Furthermore, since the mobile unit can detect delays, safety countermeasures can be taken based on the delay at the mobile unit's side.

[0082] Alternatively, for example, in the delay detection method of the second approach, the predetermined time is the current time, and in the delay detection, the difference between the second time information and the current time is calculated, and the delay is detected based on the calculated difference.

[0083] Therefore, delays can be detected simply by calculating the difference between the information at the second moment and the current moment, without the need for complex calculations.

[0084] Alternatively, for example, the delay detection method involved in the third approach may be in the delay detection method involved in the first or second approach, wherein, in the detection of the delay, the difference between the second time information contained in the most recently received operation signal and the current time is periodically calculated.

[0085] Therefore, by periodically detecting delays, it is possible to monitor delays more accurately, even when delays vary over time.

[0086] Alternatively, for example, the delay detection method involved in the fourth approach may, in any of the delay detection methods involved in the first to third approaches, cause the moving body to perform a predetermined action when the delay is detected.

[0087] Therefore, by performing predetermined actions, safety measures can be taken on the moving body side based on the delay.

[0088] Alternatively, for example, in any of the delay detection methods in the first to third methods, the delay detection method in the fifth method may cause the mobile body to perform a predetermined action if the internal time of the mobile body is regressed by a predetermined amount or more.

[0089] Therefore, even when the internal time of the mobile body is regressed by a predetermined amount, i.e., when the delay cannot be accurately detected, the mobile body can still perform a predetermined action. For example, when the internal time of the mobile body is regressed by a predetermined amount, the mobile body can be placed in a safe state.

[0090] Alternatively, for example, the delay detection method involved in the sixth approach may, in the delay detection method involved in the fourth or fifth approach, include performing the predetermined action by performing MRM (Minimal Risk Maneuver).

[0091] Therefore, the safety of moving objects can be ensured when delays are detected.

[0092] Alternatively, for example, the delay detection method involved in the seventh method may be in the delay detection method involved in the fourth or fifth method, wherein the operation signal contains a control command based on the operation of the operator, and the predetermined action includes ignoring the control command contained in the received operation signal.

[0093] Therefore, when a delay is detected, it is possible to prevent the mobile body from being controlled due to an operating signal deemed inappropriate for the state of the mobile body at that moment. This helps to ensure the safety of the mobile body.

[0094] Alternatively, for example, in the delay detection method involved in the eighth method, in any of the delay detection methods involved in the first to seventh methods, the first time information includes a time obtained by correcting the time for the time up to the time until the predetermined information is obtained for the time when the predetermined information is obtained.

[0095] Therefore, it is possible to determine the delay by taking into account the time until the predetermined information is obtained (e.g., camera delay). By including this time, the delay can be calculated more accurately, thus enabling more precise monitoring of the overall delay of the remote control system.

[0096] Alternatively, for example, in the delay detection method involved in the ninth method, in any of the delay detection methods involved in the first to eighth methods, the second time information includes a time obtained by correcting the first time information based on the time from receiving the mobile body information to displaying it in the terminal.

[0097] Therefore, it is possible to determine the delay by taking into account the time from receiving the moving object information to its display (e.g., display delay). By including this time, the delay can be calculated more accurately, thus enabling more precise monitoring of the overall delay of the remote control system.

[0098] Alternatively, for example, the delay detection method involved in the tenth method may correct the second time information based on the time from receiving the operation signal to the completion of the control corresponding to the operation signal in the mobile body, and detect the delay in the remote control system based on the corrected second time information and the predetermined time.

[0099] Therefore, it is possible to determine the delay by taking into account the time until the control corresponding to the operation signal is completed (e.g., execution delay). By including this time, the delay can be calculated more accurately, and thus the overall delay of the remote control system can be monitored more precisely.

[0100] Alternatively, for example, in the delay detection method involved in the eleventh method, in any of the delay detection methods involved in the first to tenth methods, a first signature information is attached to the mobile body information and sent to the terminal of the operator, the operation signal containing a second signature information is received, and the second signature information is verified based on the first signature information.

[0101] Therefore, electronic signatures can be used to confirm that the operator can remotely control the device after confirming the mobile body information sent by the device.

[0102] Alternatively, for example, the delay detection method involved in the twelfth method may be the delay detection method involved in the eleventh method, wherein the second signature information is information sent from the terminal at the start of remote control.

[0103] Therefore, it is possible to confirm that the operator sends an operation signal based on the confirmation of the moving body information sent by this device at the start of remote operation.

[0104] Alternatively, for example, in the delay detection method involved in the thirteenth method, in any of the delay detection methods involved in the first to twelfth methods, a first counter information is added to the mobile body information and sent to the terminal of the operator, the operation signal containing a second counter information is received, and the second counter information is verified.

[0105] Therefore, a counter can be used to confirm that the moving body information and operation signals are sent in sequence.

[0106] Alternatively, for example, the delay detection method of the fourteenth method may, in the case of delay elimination during the execution of the MRM, release the execution of the MRM in the delay detection method of the sixth method.

[0107] Therefore, the moving body can automatically recover from the MRM if the delay is eliminated.

[0108] Alternatively, for example, in the delay detection method involved in the fifteenth method, in any of the delay detection methods involved in the first to fourteenth methods, a time signal containing the second time information is received from the terminal when the operator does not remotely control the moving body.

[0109] Therefore, delays can be detected even when the operator is providing remote assistance.

[0110] Alternatively, for example, in the delay detection method involved in the sixteenth method, in any of the delay detection methods involved in the first to fifteenth methods, the first encryption key is used to encrypt the first time information, the encrypted first time information is appended to the mobile body information and sent to the terminal, and the second time information contained in the received operation signal is decrypted using the first decryption key corresponding to the first encryption key.

[0111] Therefore, encryption technology can be used to confirm that the operator can remotely control the device based on the confirmation of the mobile body information sent by the device.

[0112] Alternatively, for example, in the delay detection method involved in the seventeenth method, in any of the delay detection methods involved in the first to the sixteenth methods, a second encryption key is attached to the mobile body information and sent to the terminal, the operation signal encrypted using the second encryption key is received, and the received operation signal is decrypted using a second decryption key corresponding to the second encryption key.

[0113] This confirms that the operating signals have not been tampered with. It also prevents the moving body from being controlled by unauthorized operating signals.

[0114] Alternatively, for example, in the delay detection method involved in the eighteenth method, the time represented by the first time information and the time represented by the second time information are the same time.

[0115] Therefore, delays can be detected by taking into account the round-trip transmission delay in the remote control system. Furthermore, there is no need to correct the first-moment information at the terminal, thus reducing the terminal's processing load.

[0116] Furthermore, for example, the delay detection device involved in the nineteenth method is a delay detection device provided by a mobile body that can be remotely controlled by an operator, comprising: a transmitting unit that adds first time information to mobile body information indicating the state of the mobile body and transmits it to the operator's terminal; a receiving unit that receives an operation signal based on the operator's operation on the mobile body in response to the mobile body information, and includes second time information based on the first time information contained in the mobile body information; and a determining unit that detects a delay in a remote control system including the mobile body and the terminal based on the second time information and a predetermined time. Furthermore, for example, the program involved in the twentieth method is a program that causes a computer to execute the delay detection method involved in any of the first to eighteenth methods.

[0117] Therefore, the same effect as the aforementioned delay detection method can be achieved.

[0118] Furthermore, these general or specific methods can be implemented by non-transitory recording media such as systems, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, or recording media. The program can be pre-stored on the recording medium or provided to the recording medium via wide-area communication networks, including the Internet.

[0119] The following describes the implementation methods in detail with reference to the accompanying drawings.

[0120] Furthermore, the embodiments described below are general or specific examples. The numerical values, shapes, constituent elements, the arrangement and connection methods of constituent elements, steps, and the order of steps shown in the following embodiments are all examples and are not intended to limit this disclosure. In addition, constituent elements in the following embodiments that are not described in the independent claims are described as arbitrary constituent elements.

[0121] Furthermore, these figures are schematic diagrams and not necessarily strictly representational. Therefore, for example, the scales and other parameters may not be consistent across figures. Additionally, substantially identical components in each figure will be marked with the same labels, and repetitive descriptions will be omitted or simplified.

[0122] Furthermore, in this specification, terms such as "consistent" indicating relationships between elements, as well as numerical values ​​and numerical ranges, do not merely have a strict meaning, but rather indicate substantially equivalent ranges, for example, including differences of a few percent (or about 10%).

[0123] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not indicate the quantity or order of constituent elements unless otherwise specified, but are used for the purpose of differentiation to avoid confusion of the same constituent elements.

[0124] (Implementation Method 1)

[0125] The following is for reference Figures 2 to 11 This embodiment explains the delay detection method, etc.

[0126] [1-1. Composition of a Remote Control System]

[0127] First, refer to Figure 2 Explain the structure of a remote control system that implements a remote delay method. Figure 2 This is a block diagram illustrating the functional configuration of the remote control system 1 according to this embodiment.

[0128] like Figure 2 As shown, the remote control system 1 includes a mobile body 10, a control server 20, and a terminal 30. The remote control system 1 is a system for remotely operating the automatically driven mobile body 10 when necessary.

[0129] Mobile body 10 is a vehicle capable of switching between autonomous driving and remote-operated driving. The vehicle can be, for example, a car, but could also be a truck, bus, or railway vehicle. Furthermore, mobile body 10 could also be an aerial vehicle such as a drone, or a ship.

[0130] The mobile unit 10 is configured to include various functional components implemented by a CPU (Central Processing Unit) or MPU (Microprocessor Unit) for performing processing, and a memory unit such as ROM (Read-Only Memory) or RAM (Random Access Memory) storing programs for enabling each functional component to perform various processes. Furthermore, the mobile unit 10 includes, as functional components, a mobile unit information acquisition unit 11a, a mobile unit information transmission unit 11b, an operation signal receiving unit 11c, a delay determination unit 12, an operation signal execution unit 13, an MRM (Minimal Risk Maneuver) execution unit 14, an auxiliary unit 15, a verification unit 16, a mobile unit information acquisition delay correction unit 17a, an operation signal execution delay correction unit 17b, a decryption unit 18a, a vehicle encryption unit 18b, and a time management unit 19.

[0131] The mobile body information acquisition unit 11a acquires sensor information such as camera images and radar information, or vehicle speed and other information related to the mobile body, from sensors mounted on the mobile body 10, as mobile body information. Furthermore, the mobile body information acquisition unit 11a acquires the acquisition time of the mobile body information acquired from the sensors, etc., as the mobile body information acquisition time. In addition, the mobile body information may include results obtained by sensing the surroundings of the mobile body 10 or information indicating the driving state of the mobile body 10, such as at least one of camera images, LiDAR information (LiDAR-based measurement results), radar information (radar-based measurement results), sonar information (sonar-based measurement results), the speed of the mobile body, and the angular velocity of the mobile body.

[0132] The mobile information transmission unit 11b appends the mobile information acquisition time to the mobile information and sends it to the terminal 30. The mobile information transmission unit 11b is an example of a transmission unit.

[0133] The operation signal receiving unit 11c receives an operation signal from the terminal 30 and acquires the mobile body information acquisition time contained in the operation signal. The operation signal receiving unit 11c is an example of a receiving unit.

[0134] The delay determination unit 12 calculates the round-trip delay (delay time) based on the time the mobile information was acquired and the current time. The round-trip delay includes at least... Figure 1 The image transmission system and the remote operation signal transmission system shown both exhibit transmission delay. Furthermore, the delay determination unit 12 determines whether the delay is below a threshold. That is, the delay determination unit 12 determines whether a delay exists. The existence of a delay means that the delay is greater than a predetermined value (threshold). The current time is an example of a predetermined time, and the delay determination unit 12 is an example of a determination unit.

[0135] The operation signal execution unit 13 causes the moving body 10 to perform a specific action corresponding to the operation signal based on the operation signal.

[0136] The MRM execution unit 14 causes the moving body 10 to perform MRM when a delay is detected, etc. MRM is an example of a predetermined action.

[0137] The attachment unit 15 attaches predetermined information to the mobile information. For example, the attachment unit 15 may attach an electronic signature as predetermined information to the mobile information; this electronic signature is used for verification using the decryption key held by the mobile 10. The electronic signature may also be a hash value calculated using a hash function. Furthermore, the attachment unit 15 may attach a counter (counter value) as predetermined information to the mobile information. The predetermined information attached by the attachment unit 15 is then attached (reattached) to the operation signal by the terminal 30 and sent back to the mobile 10.

[0138] The verification unit 16 performs verification using predetermined information contained in the received operation signal. If the predetermined information is an electronic signature, the verification unit 16 verifies whether the electronic signature contained in the operation signal is an electronic signature issued by this device (e.g., whether two electronic signatures match). If the predetermined information is a counter, the verification unit 16 verifies whether the receiving order of the received operation signals matches the order of the counter.

[0139] The mobile information acquisition delay correction unit 17a estimates the time required to acquire mobile information and corrects the mobile information acquisition time. The mobile information acquisition delay correction unit 17a estimates... Figure 1 The camera delay is shown, and the acquisition time of the moving body information contained in the moving body information is corrected based on the estimated camera delay.

[0140] The operation signal execution delay correction unit 17b estimates the time required to execute the operation signal and corrects the timing of acquiring moving body information. The operation signal execution delay correction unit 17b estimates... Figure 1 The execution delay is shown, and the acquisition time of the moving body information contained in the operation signal is corrected based on the execution delay.

[0141] The decryption unit 18a decrypts the operation signal based on the decryption key.

[0142] The vehicle encryption unit 18b generates a pair of encryption and decryption keys, encrypts the acquisition time of the moving body information attached to the image, and sends the encryption key to the terminal 30.

[0143] The time management unit 19 synchronizes with GPS (Global Positioning System) and continuously outputs a high-precision time. Furthermore, the time management unit 19 detects the time deviation between the high-precision time and the internal time of the moving body 10 and issues a notification.

[0144] Furthermore, the mobile body 10 may at least include a mobile body information transmitting unit 11b, an operation signal receiving unit 11c, and a delay determination unit 12. For example, the mobile body 10 may also include a mobile body information acquiring unit 11a, a mobile body information transmitting unit 11b, an operation signal receiving unit 11c, a delay determination unit 12, an operation signal execution unit 13, and an MRM execution unit 14. Furthermore, the delay detection device is implemented using the various components included in the mobile body 10. The delay detection device may at least include a mobile body information transmitting unit 11b, an operation signal receiving unit 11c, and a delay determination unit 12.

[0145] The control server 20 is an information processing device that is communicatively connected to both the mobile body 10 and the terminal 30. The control server 20 can be a cloud server or a local server.

[0146] The control server 20 is configured to include functional units implemented by a CPU or MPU for executing processing, and a memory unit such as ROM or RAM storing programs for causing each functional unit to perform various processes. Furthermore, the control server 20 includes a control status management unit 21, a mobile body information transmission unit 22, and an operation signal transmission unit 23 as functional components.

[0147] The control status management unit 21 determines between which terminal 30 and mobile body 10 the mobile body information and operation signals are exchanged. For example, the control status management unit 21 determines the terminal 30 to send to the destination based on the destination information contained in the mobile body information, and determines the mobile body 10 to send to the destination based on the destination information contained in the operation signals.

[0148] Based on the decision of the control status management unit 21, the mobile information transmission unit 22 transmits the mobile information to the designated terminal 30.

[0149] Based on the decision of the control status management unit 21, the operation signal transmission unit 23 transmits the operation signal to the designated mobile body 10.

[0150] Terminal 30 is an input device 40 for remote operation via operator input (see below). Figure 12 This is a remote operation terminal device that connects to and receives information required for remote operation from the mobile device 10 and the control server 20. For example, the terminal 30 performs processing to display images contained in mobile device information received from the mobile device 10 to the operator. Furthermore, the terminal 30 performs processing to send operation signals received from the input device 40 to the mobile device 10. In this embodiment, the terminal 30 is characterized in that it appends the mobile device information acquisition time contained in the mobile device information to the operation signal representing the operator's input to the input device 40 regarding the mobile device information received from the mobile device 10, and then sends it to the mobile device 10.

[0151] The terminal 30 is configured to include various functional units implemented by a CPU or MPU for executing processing, and a memory unit such as ROM or RAM storing programs for enabling various functional units to perform various processes. Additionally, the terminal 30 includes, as functional units, a mobile information receiving unit 31a, an operation signal receiving unit 31b, an operation signal transmitting unit 32, a mobile information display unit 33a, a delay time display unit 33b, a display delay correction unit 34a, an operation signal acquisition delay correction unit 34b, an operator response delay correction unit 34c, a remote operation switching unit 35, and an encryption unit 36.

[0152] The mobile information receiving unit 31a receives mobile information and extracts the mobile information acquisition time contained in the mobile information.

[0153] The operation signal receiving unit 31b is connected to the input device 40 in a communicable manner and receives operation signals from the input device 40.

[0154] The operation signal transmitting unit 32 adds the mobile body information acquisition time extracted by the mobile body information receiving unit 31a to the operation signal and sends it to the mobile body 10.

[0155] The mobile information display unit 33a displays mobile information on the screen. In this embodiment, the mobile information display unit 33a displays an image. The mobile information display unit 33a is a monitor, such as a liquid crystal display device, but is not limited thereto. Furthermore, display is an example of a prompt. The prompt can be a display, or it can be sound or light.

[0156] The delay time display unit 33b displays the delay time (e.g., round-trip delay time) on the screen. The delay time display unit 33b is a monitor, such as a liquid crystal display device, but is not limited thereto. Furthermore, the delay time display unit 33b and the moving body information display unit 33a may also be implemented by a single display device.

[0157] The display delay correction unit 34a estimates the time delay from receiving mobile information to displaying it on the screen of the mobile information display unit 33a, and corrects the acquisition time of the mobile information extracted by the mobile information receiving unit 31a. This correction is performed to take into account... Figure 1 The displayed delay is used to determine the delay.

[0158] The operation signal acquisition delay correction unit 34b estimates the operation signal transmission and processing delay of the input device 40 and corrects the acquisition time of the mobile information extracted by the mobile information receiving unit 31a. This correction is performed to take into account... Figure 1 The signal processing delay shown is used to determine the delay.

[0159] The operator reaction delay correction unit 34c estimates the operator's reaction delay and corrects the acquisition time of the mobile information extracted by the mobile information receiving unit 31a. This correction is performed to take into account... Figure 1 The reaction delay shown is used to determine the delay.

[0160] The remote operation switching unit 35 receives a switching button signal from the controller (not shown) and sends a remote control switching signal to the mobile body 10. The controller may also be included in the input device 40, for example. The remote control switching signal indicates a switch from automatic driving to remote control-based (e.g., remote operation) driving.

[0161] The encryption unit 36 ​​generates encryption and decryption keys and encrypts the operation signals.

[0162] In addition, the remote control system 1 may also include an input device 40 for the operator to make inputs corresponding to remote operations. The input device 40 is connected to the terminal 30 in a communicative manner, and may include at least one of a steering wheel, brake, accelerator, etc.

[0163] [1-2. Operation of the remote control system]

[0164] Next, refer to Figures 3 to 11 Explain the operation of the remote control system 1 constructed as described above. First, refer to... Figure 3 This section provides a summary of the actions performed in the remote control system 1. Figure 3 This is a diagram used to illustrate the outline of the operation (delay detection method) of the remote control system 1 according to this embodiment.

[0165] like Figure 3 As shown, the mobile body 10 appends the mobile body information acquisition time to the mobile body information (S1), and sends the mobile body information with the mobile body information acquisition time appended to it to the terminal 30 (S2). The mobile body information acquisition time is an example of the first moment information.

[0166] Terminal 30 receives mobile information transmitted in the mobile information transmission system N (S3), reallocates the mobile information acquisition time to an operation signal from the input device 40 corresponding to the received mobile information (S4), and sends the operation signal (S5). Control server 20 sends the operation signal from terminal 30 to mobile body 10 (S6). Mobile body 10 verifies the mobile information acquisition time contained in the operation signal (S7). The mobile information acquisition time contained in the operation signal is an example of second time information.

[0167] Thus, by associating motion information (e.g., individual frames of an image) with the operation signal and measuring the round-trip delay, the total transmission delay of the operation signal based on the image frames actually seen by the operator can be measured. For example, it can be verified that the operation signal received in step S6 corresponds to an image captured by motion 10 a few seconds ago. For example, motion 10 calculates the delay time (total transmission delay) in steps S2 to S6 based on the motion information acquisition time contained in the operation signal and the current time, and detects the delay based on whether the delay time is below a threshold. Motion 10 can be said to be performing delay determination. Furthermore, motion 10 uses the motion information acquisition time attached by motion 10 to perform delay determination, so there is no need for time synchronization between motion 10 and terminal 30. The current time is an example of a predetermined time.

[0168] The following is for reference Figure 4 Further explanation of the actions in remote control system 1 is needed. Figure 4 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 4 The diagram illustrates the actions taken when MRM is executed if the total time from acquiring information from mobile device 10 and displaying the mobile device information to the operator, plus the time from the operator's operation to the response being reflected on mobile device 10, is not less than a certain time. Furthermore, Figure 4 The timing diagram shown illustrates the scenario of remote operation. In subsequent timing diagrams, unless otherwise specified, all other timing diagrams represent the scenario of remote operation.

[0169] like Figure 4 As shown, the mobile body information acquisition unit 11a of the mobile body 10 acquires mobile body information from a camera or the like, and appends the time of acquiring the mobile body information, i.e., the acquisition time of the mobile body (mobile body acquisition time) (time A), to the mobile body information (S11).

[0170] Next, the mobile information transmission unit 11b transmits the mobile information with time A appended to it to the terminal 30 (S12). The mobile information transmission unit 11b transmits the mobile information to the terminal 30 via wireless communication.

[0171] Next, the mobile information receiving unit 31a of the terminal 30 receives the mobile information sent from the mobile information sending unit 11b, and obtains and holds the time A from the received mobile information (S13).

[0172] Next, the mobile information display unit 33a displays mobile information (S14). The mobile information display unit 33a may display an image as mobile information, for example. The operator confirms the image displayed on the mobile information display unit 33a and performs operations for remote operation on the input device 40 as needed.

[0173] Next, the operation signal receiving unit 31b obtains from the input device 40 the operation signal sent to the moving body 10 in response to the image displayed in step S14 (S15).

[0174] Next, the operation signal transmitting unit 32 appends the time A obtained in step S13 to the operation signal (S16) and transmits the operation signal (S17). The operation signal transmitting unit 32 transmits the operation signal to the mobile body 10 via wireless communication.

[0175] Next, the operation signal receiving unit 11c receives the operation signal from the terminal 30 and extracts time A from the received operation signal (S18).

[0176] Next, the delay determination unit 12 determines whether the operation signal is delayed (whether operation signal delay has occurred) based on the time A extracted from the operation signal (S19). The delay determination unit 12 determines whether delay exists by subtracting time A from the current time from a predetermined value (threshold). If the delay determination unit 12 determines that delay has occurred when the difference time is greater than the predetermined value, and determines that delay has not occurred when the difference time is less than the predetermined value.

[0177] Furthermore, the delay determination unit 12 repeats the processing of step S19 until the remote operation ends. For example, the delay determination unit 12 periodically calculates the difference between the time of acquisition of the moving body contained in the most recently received operation signal and the current time.

[0178] Next, when the delay determination unit 12 determines that a delay has occurred, the MRM execution unit 14 executes MRM (S20) for safety reasons, since the received operation signal may be an operation signal corresponding to the operation performed by the operator when observing an image captured at an earlier time.

[0179] Furthermore, if the delay determination unit 12 determines that the delay of the operation signal is below a predetermined value, the operation signal execution unit 13 executes the operation signal (S21). The operation signal execution unit 13 causes the moving body 10 to perform the action indicated by the operation signal. The action may be, for example, a change in speed, steering angle, etc., but is not limited to these.

[0180] Furthermore, when the mobile body 10 can be remotely controlled from multiple terminals 30, it can also be... Figure 4 At any timing between steps S11 and S21 shown, the mobile body 10 sends operator notification signals to multiple terminals 30 (see below). Figure 11 ).

[0181] Next, refer to Figures 5A to 5E illustrate Figure 4 A summary (concept) of each process shown. Figure 5AThis is a flowchart illustrating the first action (delay detection method) in the moving body 10 according to this embodiment. Figure 5A The actions of mobile body 10 in acquiring and sending mobile body information are shown. Figure 5A Corresponding to Figure 4 Steps S11 and S12 are shown.

[0182] like Figure 5A As shown, the mobile body information acquisition unit 11a acquires mobile body information from a camera or the like (S31a), and appends the mobile body information acquisition time (time A) to the mobile body information (S32a). Then, the mobile body information transmission unit 11b transmits the mobile body information with time A appended to it to the terminal 30 (S33a).

[0183] Figure 5B This is a flowchart illustrating the second action (delay detection method) in the terminal 30 according to this embodiment. Figure 5B The actions of terminal 30 in receiving mobile body information are shown. Figure 5B Corresponding to Figure 4 Steps S13 and S14 are shown.

[0184] like Figure 5B As shown, the mobile body information receiving unit 31a receives mobile body information from the mobile body 10 (S41a), extracts the mobile body information acquisition time from the received mobile body information and holds it (S42a).

[0185] Next, the mobile body information display unit 33a displays the mobile body information on the screen (S43a).

[0186] Figure 5C This is a flowchart illustrating the third action (delay detection method) in the terminal 30 according to this embodiment. Figure 5C The actions of terminal 30 in sending operation signals are shown. Figure 5C Corresponding to Figure 4 Steps S15 to S17 are shown.

[0187] like Figure 5C As shown, the operation signal receiving unit 31b receives the operation signal from the input device 40 (S51a) and appends the latest mobile body information acquisition time to the operation signal (S52a). Then, the operation signal transmitting unit 32 transmits the operation signal to the mobile body 10 (S33a).

[0188] Figure 5D This is a flowchart illustrating the fourth action (delay detection method) in the moving body 10 according to this embodiment. Figure 5D The action of the mobile body 10 in receiving the operation signal is shown. Figure 5D Corresponding to Figure 4Step S18 is shown.

[0189] like Figure 5D As shown, the operation signal receiving unit 11c receives the operation signal from the terminal 30 (S61a), extracts the moving body information from the operation signal to obtain the time and holds it (S62a).

[0190] Figure 5E This is a flowchart illustrating the fifth action (delay detection method) in the moving body 10 according to this embodiment. Figure 5E The action of delay determination in moving body 10 is shown. Figure 5E Corresponding to Figure 4 Steps S20 and S21 are shown.

[0191] like Figure 5E As shown, the delay determination unit 12 compares the latest time of acquiring the maintained moving body information with the current time (S71a).

[0192] If the delay determination unit 12 determines that the current time minus the mobile information acquisition time is greater than a predetermined value (S72a is yes), the MRM execution unit 14 executes MRM (S73a). Conversely, if the delay determination unit 12 determines that the current time minus the mobile information acquisition time is not greater than the predetermined value (S72a is no), the delay determination unit 12 returns to step S71a and continues processing until the remote operation ends.

[0193] Next, refer to Figures 6A to 6E illustrate Figure 4 A specific example of each of the processes shown. Figure 6A This is a flowchart illustrating a specific example of the first action (delay detection method) in the moving body 10 according to this embodiment. Figures 6A to 6E The illustration shows a specific example of an action where the moving body information is a camera image, WebRTC (Web Real-Time Communication) is used as the image transmission protocol for the camera image, and JSON (JavaScript Object Notation) or protocolBuffer is used in the serialization of the operation signals.

[0194] Figure 6A The actions of the moving body 10 in the acquisition and transmission of the image stream are shown. Figure 6A Corresponding to Figure 4 Steps S11 and S12 are shown.

[0195] like Figure 6AAs shown, the mobile body information acquisition unit 11a receives an image stream from a camera or the like as mobile body information (S31b), and updates the image acquisition timestamp in the memory to the reception time of step S31b (S32b). Updating the image acquisition timestamp in the memory means updating the image acquisition timestamp appended to the mobile body information. The image acquisition timestamp is an example of the mobile body acquisition time.

[0196] Next, the mobile information transmission unit 11b encodes the image stream (S33b) and appends the image acquisition timestamp to the frame data (S34b). Appending the image acquisition timestamp to the frame data is an example of appending the mobile acquisition time to the mobile information.

[0197] Next, the mobile information transmission unit 11b divides the frame data into RTP (Real-time Transport Protocol) packets (S35b) and sends the image packets to the terminal 30 (S36b).

[0198] Figure 6B This is a flowchart illustrating a specific example of the second action in the terminal 30 according to this embodiment. Figure 6B Corresponding to Figure 4 Steps S13 and S14 are shown.

[0199] like Figure 6B As shown, the mobile information receiving unit 31a receives an image stream from the mobile unit 10 (S41b), constructs frame data (data of one frame) based on the received RTP packets (S42b), extracts a timestamp from the frame data (S43b), and updates the timestamp in the memory (S44b). Updating the timestamp in the memory means updating the timestamp attached to the operation signal. Furthermore, the mobile information receiving unit 31a decodes the frame data (S45b).

[0200] Next, the mobile body information display unit 33a displays the image based on the decoded frame data (S46b). This allows the operator to be displayed the image, i.e., the status of the mobile body 10.

[0201] Figure 6C This is a flowchart illustrating a specific example of the third action (delay detection method) in the terminal 30 according to this embodiment. Figure 6C Corresponding to Figure 4 Steps S15 to S17 are shown.

[0202] like Figure 6CAs shown, the operation signal receiving unit 31b receives an operation packet as an operation signal from the input device 40 (S51b), performs deserialization of the operation packet (S52b), and appends a timestamp from the memory to the operation information (operation signal) (S53b). Then, the operation signal sending unit 32 performs serialization of the operation information (S54b), divides the operation information into packets (S55b), and sends the operation packets to the mobile body 10 (S56b).

[0203] Figure 6D This is a flowchart illustrating a specific example of the fourth action (delay detection method) in the moving body 10 according to this embodiment. Figure 6D Corresponding to Figure 4 Step S18 is shown.

[0204] like Figure 6D As shown, the operation signal receiving unit 11c receives the operation packet from the terminal 30 as an operation signal (S61b), performs deserialization of the operation packet (S62b), extracts the timestamp from the operation signal (S63b), and updates the timestamp in the memory (S64b).

[0205] Figure 6E This is a flowchart illustrating a specific example of the fifth action (delay detection method) in the moving body 10 according to this embodiment. Figure 6E Corresponding to Figure 4 Steps S20 and S21 are shown.

[0206] like Figure 6E As shown, the delay determination unit 12 compares the timestamp in the memory with the current time (S71b).

[0207] If the delay determination unit 12 determines that the current time - timestamp (time of acquisition of mobile information) > a predetermined value (S72b is yes), the MRM execution unit 14 executes MRM (S73b). Furthermore, if the delay determination unit 12 determines that the current time - timestamp > a predetermined value is not true (S72b is no), the operation signal execution unit 13 executes an operation signal (S74b) and returns to step S71b to continue processing until the remote operation ends.

[0208] Figure 7 This is a diagram used to illustrate the details of the operation (delay detection method) of the remote control system 1 involved in this embodiment. Figure 7 The process of re-attaching the timestamp in terminal 30 is illustrated. Specifically, Figure 7 A timing diagram is shown for image frame reception timing, image rendering, timestamps in memory, operation signal transmission, and operation signal reception timing, without display delays and signal acquisition delays. Furthermore, Figure 7 An example is shown where frames are received in the order of frame A, frame B, frame C, and frame D.

[0209] If frame A is received, frame A is drawn, and the timestamp in memory is updated to the timestamp of frame A, and this timestamp is maintained until frame B is received. If an operation signal is received from input device 40 during the period from receiving frame A to receiving frame B, it is considered that the operation signal is based on the operator's confirmation of the image of frame A, and therefore the operation signal is appended with the timestamp of frame A. Then, the operation signal with the timestamp of frame A appended is sent to the mobile body 10.

[0210] If frame B is received, the image is switched to be drawn for frame B, and the timestamp in memory is updated to the timestamp of frame B.

[0211] In addition, if multiple operation signals are received from the input device 40 during the period from receiving frame C to receiving frame D, it is assumed that each of these multiple operation signals is an operation signal based on the operator confirming the image of frame C, and therefore each of these multiple operation signals is appended with the timestamp of frame C.

[0212] Here, refer to Figures 8 to 11 Explain the message examples for each signal. Figure 8 This is a diagram illustrating an example of an operation signal involved in this embodiment. Figures 8 to 11 The example shown uses RTCP (Real-time Transport Control Protocol) as the protocol.

[0213] Figure 8 The V=2, P, IC, PT=APP=204 and length L shown are common components. Furthermore, Figure 8 For convenience, “PT=APP=204” will be represented as “PT=APP”.

[0214] The “SSRC of RTP packet sender” stores the identifier of terminal 30 (image receiver). This identifier is used to reject operations from any terminal 30 other than those with its own operating permissions.

[0215] “RCMD(ASCII)” stores the message name, which contains information indicating that it is an operation signal.

[0216] The "Received frame border RTP timestamp" stores the timestamp (Unix time) of the received mobile information (e.g., the mobile acquisition time appended to the mobile information in mobile 10). For example, it can store the timestamp embedded in the RTP packet through an insertable stream.

[0217] The "SSRC of RTP packet sender" stores the identifier of the mobile unit 10 (image sending end). The identifier is used to determine whether the operation signal comes from the terminal 30 that has viewed the image sent by this device.

[0218] The "sequence number" stores the sequence number. The sequence number is used to confirm whether messages were sent in order.

[0219] The “Control message” stores control values ​​such as acceleration and steering wheel angle.

[0220] Furthermore, in RTCP, the items included in the message can be arbitrarily set. In this disclosure, the "Received frame border RTP timestamp" item is included, and the timestamp contained in the mobile body information (the time of mobile body acquisition) is stored in the "Received frame border RTP timestamp", that is, the timestamp attached to the mobile body 10.

[0221] Figure 9 This diagram illustrates an example of an operation start signal according to this embodiment. The operation start signal is a signal sent from the terminal 30 to the mobile body 10 at the moment when remote operation begins.

[0222] The “SSRC of packet sender” stores the identifier of terminal 30 (image receiver). Here, the identifier of terminal 30 that requests operation permission from mobile body 10 is stored.

[0223] “RREQ(ASCII)” stores the message name, which contains information indicating the start of remote operation.

[0224] Figure 10 This diagram illustrates an example of an operation end signal according to this embodiment. The operation end signal is a signal sent from the terminal 30 to the mobile body 10 at the moment when remote operation ends.

[0225] The “SSRC of packet sender” stores the identifier of terminal 30 (image receiver). Here, the identifier of terminal 30 that has relinquished the operation rights of mobile body 10 is stored.

[0226] "REND(ASCII)" stores the message name, which contains information indicating the end of the remote operation.

[0227] Figure 11This diagram illustrates an example of an operator notification signal according to this embodiment. It is a signal sent from the mobile body 10 to multiple terminals 30 that have the authority to remotely operate the mobile body 10, and remote operation is being performed by one of the multiple terminals 30. Since the mobile body 10 can only be remotely operated from one terminal 30, an operator notification signal is sent to notify other terminals 30 that remote operation has been accepted while the mobile body 10 is being remotely operated from one terminal 30.

[0228] The “SSRC of packet sender” stores the identifier of the mobile body 10 (image sending end).

[0229] The “ROPR(ASCII)” field stores the message name, which indicates which terminal 30 is performing the remote operation.

[0230] The "SSRC of control RTCP packet sender" field stores the operator's identifier. Here, it stores information indicating which image receiver is remotely operating the mobile unit 10. The image receiver refers to the operator corresponding to the terminal 30 that receives mobile unit information from the mobile unit 10. Alternatively, it can be understood as storing information indicating which operator's operation packet the mobile unit 10 is receiving.

[0231] (Implementation Method 2)

[0232] The following is for reference Figure 12 This embodiment explains the delay detection method, etc. Furthermore, the following mainly describes the differences from Embodiment 1; content that is the same as or similar to Embodiment 1 is omitted or simplified. In addition, in Embodiments 2 and later, the configuration of each component of the remote control system may be the same as that of the remote control system 1 according to Embodiment 1, and the notation used in Embodiment 1 will be used for explanation.

[0233] Figure 12 This is a timing diagram showing the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 12 The example illustrates how to execute MRM if no operation signal is received within a certain period after receiving the operation signal.

[0234] like Figure 12 As shown, the operation signal receiving unit 11c of the mobile body 10 holds the time when the operation signal is received, i.e., the operation signal receiving time (time B) (S81).

[0235] Then, the delay determination unit 12 determines the operation signal delay based on time B until the next control signal (operation signal) arrives or the MRM is executed (S82). The delay determination unit 12 performs this determination based on whether the current time - time B > a predetermined value is met. If the current time - time B > the predetermined value is met, the delay determination unit 12 determines that the operation signal has been delayed; if the current time - time B > the predetermined value is not met, it determines that the operation signal has not been delayed.

[0236] MRM execution unit 14 executes MRM (S83) if the delay determination unit 12 determines that a delay in the operation signal has occurred.

[0237] Thus, in this embodiment, if the total time from obtaining mobile body information from mobile body 10 and prompting the operator with mobile body information, and the time from the operator's operation to the response to mobile body 10, cannot be less than a certain time (predetermined value), control is performed to execute MRM.

[0238] (Implementation Method 3)

[0239] The following is for reference Figure 13 This embodiment explains the delay detection method, etc. Figure 13 This is a timing diagram showing the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 13 The example illustrates how, in the event of a delay, the MRM is not executed and operation signals with a delay of a certain value are ignored on the mobile body 10 side.

[0240] like Figure 13 As shown, when the delay determination unit 12 determines that the delay of the operation signal is below a predetermined value, that is, when it determines that the operation signal is not delayed, the operation signal execution unit 13 executes the operation signal (S21). The operation signal execution unit 13 causes the moving body 10 to perform the action represented by the operation signal.

[0241] Furthermore, the operation signal execution unit 13 ignores the operation signal when the delay determination unit 12 determines that the delay of the operation signal is greater than a predetermined value, i.e., when the operation signal is determined to be delayed. For example, when the operation signal is determined to be delayed, the operation signal execution unit 13 ignores the control command contained in the received operation signal. That is, when the operation signal is determined to be delayed, the operation signal execution unit 13 does not perform control on the moving body 10 based on the operation signal. Ignoring the operation signal is an example of a predetermined action.

[0242] Furthermore, the mobile body 10 in this embodiment may not include the MRM execution unit 14.

[0243] (Implementation Method 4)

[0244] The following is for reference Figures 14 to 18 This embodiment explains the delay detection method, etc. Figure 14 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example considering the acquisition delay (camera delay) of moving body information and the display delay of the monitor is described.

[0245] like Figure 14 As shown, the mobile information acquisition delay correction unit 17a of the mobile body 10 estimates the mobile information acquisition delay and corrects time A1 (S91). The mobile information acquisition delay correction unit 17a corrects the mobile information acquisition delay time by shifting time A1 back to a past time. For example, if time A1 is 10:00 and the mobile information acquisition delay is 1 minute, the mobile information acquisition delay correction unit 17a corrects time A1 to 9:59, thereby calculating time A2. Furthermore, a delay of 1 minute is unlikely to occur in reality, but 1 minute is illustrated for illustrative purposes.

[0246] In this embodiment, time A2 is an example of first time information. The first time information may also include time A2, which is obtained by correcting time A1, which is the time until the predetermined information (e.g., camera image) is acquired by the mobile body 10, based on the time (e.g., camera delay) until the predetermined information is acquired.

[0247] The mobile body information receiving unit 31a of the terminal 30 receives mobile body information, obtains time A2 from the received mobile body information and holds it (S92).

[0248] If the moving body information is displayed (S14), the display delay correction unit 34a estimates the display delay and corrects time A2 (S93). The display delay correction unit 34a calculates time A3 by correcting the display delay time of time A2 to a past time.

[0249] In this embodiment, time A3 can also be an example of second time information. The second time information can also include a time obtained by correcting time A2 based on the time (display delay) from the time the mobile body information is received to the time it is displayed in the terminal 30.

[0250] The operation signal transmitting unit 32 appends time A3 to the operation signal (S94).

[0251] Next, the operator reaction delay correction unit 34c estimates the operator reaction delay and corrects time A3 (S95). The operator reaction delay correction unit 34c calculates time A4 by correcting the operator reaction delay time to a past time from time A3.

[0252] Next, the operation signal acquisition delay correction unit 34b estimates the operation signal acquisition delay and corrects time A4 (S96). The operation signal acquisition delay correction unit 34b calculates time A5 by correcting the operation signal acquisition delay time from time A4 to a past time.

[0253] The operation signal receiving unit 11c of the mobile body 10 receives the operation signal from the terminal 30 and extracts the time A5 (S97) from the received operation signal.

[0254] Next, the operation signal execution delay correction unit 17b considers the operation signal execution delay, corrects time A5, and holds it (S98). The operation signal execution delay correction unit 17b estimates the operation signal execution delay and corrects time A5. The operation signal execution delay correction unit 17b calculates time A6 by correcting the operation signal execution delay time from time A5 to a past time.

[0255] In this embodiment, time A5 can also be an example of second time information, and time A6 can also be an example of corrected second time information. The corrected second time information is calculated by correcting the second time information based on the time from receiving the operation signal to the completion of the control corresponding to the operation signal in the moving body 10.

[0256] Next, the delay determination unit 12 determines whether the operation signal is delayed (whether an operation signal delay has occurred) based on time A6 (S99).

[0257] Figure 15A This is a flowchart illustrating an example of the operation (delay detection method) of the remote control system 1 according to this embodiment.

[0258] like Figure 15A As shown, the mobile body 10 acquires the mobile body information acquisition time (S101). For example, the mobile body information acquisition unit 11a acquires the time when it acquires mobile body information from a camera or the like as the mobile body information acquisition time.

[0259] Next, the moving body 10 obtains the estimated delay time (S102).

[0260] Figure 15B This is a graph illustrating the estimated delay times for various delays involved in this embodiment. Delays such as mobile information acquisition delay and display delay (terminal screen rendering delay) are sometimes impossible to measure strictly within the system. Therefore, in this embodiment, based on... Figure 15B The table shown estimates the time required to acquire information about the moving body (moving body information acquisition delay), the time until the terminal 30 acquires the information and displays it on the screen (display delay), and the time required for the moving body to execute operation signals (operation signal execution delay), etc.

[0261] like Figure 15B As shown, for each type of delay, a correspondence with a delay time is pre-established. The delay time is, for example, a fixed value. Therefore, the approximate delay time for each type of delay can be estimated.

[0262] The moving body information acquisition delay correction unit 17a, display delay correction unit 34a, operator response delay correction unit 34c, operation signal acquisition delay correction unit 34b, and operation signal execution delay correction unit 17b are, for example, based on... Figure 15B The table shown estimates the corresponding delay time.

[0263] Refer again Figure 15A The mobile body 10 rewinds the estimated delay time from the mobile body information acquisition time (time A) (S103). The mobile body information acquisition delay correction unit 17a, the display delay correction unit 34a, the operator response delay correction unit 34c, the operation signal acquisition delay correction unit 34b, and the operation signal execution delay correction unit 17b each perform the processing of the estimated delay time obtained in the rewind step S102 on the received mobile body information acquisition time or the corrected mobile body information acquisition time.

[0264] Figure 16 This is a flowchart illustrating another example of the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 16 Instead of correcting the acquisition time of mobile body information based on the delay, the delay is taken into account by postponing the update timing of the timestamp in the memory.

[0265] like Figure 16 As shown, if the mobile body 10 obtains the estimated delay time (S102), it waits for the estimated delay time (S104) and updates the mobile body information acquisition time in the memory (S105).

[0266] Reference Figure 17 and Figure 18 Explanation Figure 16 Steps S104 and S105 are shown. Figure 17 This is a diagram illustrating details of another example of the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 17 The text describes the process of reapplying the timestamp in the terminal 30, taking into account the display delay t1 of the mobile information display unit 33a.

[0267] like Figure 17As shown, in the presence of a display delay t1, the frame is drawn after a delay of t1 from the moment it is received. That is, the operator visually recognizes the image of the previous frame during the period from the moment the new frame is received until the display delay t1 has elapsed. Therefore, for operations performed during the period from the moment the new frame is received until the display delay t1 has elapsed, a timestamp of the previous frame is appended.

[0268] Taking frame B as an example, after receiving frame B, the mobile information receiving unit 31a waits... Figure 15B During the display delay t1 obtained from the memory, at the moment after the display delay t1 has elapsed, the timestamp in the memory is updated from the timestamp of frame A to the timestamp of frame B. Thus, the motion information receiving unit 31a postpones the process of updating the timestamp in the memory to the timestamp of that frame until the frame's drawing begins. For example, the motion information receiving unit 31a performs a process that synchronizes the start of drawing frame B with the update of frame B's timestamp.

[0269] In this case, for example, if the operator performs an operation during the period from the moment frame B is received until the display delay t1, the operation signal is appended with the timestamp of frame A.

[0270] Figure 18 This is a diagram illustrating details of another example of the operation (delay detection method) of the remote control system 1 according to this embodiment. Figure 18 The text describes the process of re-attaching the timestamp in the terminal 30 when, in addition to the display delay t1 of the mobile information display unit 33a, the operator's reaction delay t2 and the delay of the input device 40 (signal acquisition delay t3) are also taken into account.

[0271] like Figure 18 As shown, in the presence of display delay t1, response delay t2, and signal acquisition delay t3, the frame is drawn after a delay of display delay t1 from the moment the frame is received. Furthermore, the timing for the operation signal receiving unit 31b to receive the operation signal becomes a timing that further delays the response delay t2 and signal acquisition delay t3 from the image display timing. Therefore, for operations performed during the period of display delay t1, response delay t2, and signal acquisition delay t3 from the moment a new frame is received, a timestamp from the previous frame is appended.

[0272] Taking frame C as an example, after receiving frame C, the mobile information receiving unit 31a waits... Figure 15BDuring the period of display delay t1, response delay t2, and signal acquisition delay t3, after the display delay t1, response delay t2, and signal acquisition delay t3 have elapsed, the timestamp in the memory is updated from the timestamp of frame B to the timestamp of frame C. Thus, the mobile unit information receiving unit 31a postpones the processing of updating the timestamp to frame C until the display delay t1, response delay t2, and signal acquisition delay t3 have elapsed. For example, for operation signals received from the start of screen rendering of frame C up to the time after the response delay t2 and signal acquisition delay t3, the timestamp of frame B is appended.

[0273] (Modification 1 of Implementation Method 4)

[0274] The following is for reference Figures 19A to 20 This section explains the delay detection method involved in this variation. Figure 19A This is a flowchart illustrating an example of the operation (delay detection method) of the remote control system 1 involved in this variation. In this variation, the updating of the timestamp in the case of a low frame rate is explained.

[0275] Furthermore, in this variant, control is based on the premise that, when the frame rate is low but above a certain level, a person can predict motion in their brain (inter-frame prediction), and therefore can perform accurate operations even without considering latency.

[0276] like Figure 19A As shown, the mobile information receiving unit 31a determines whether a certain amount of time has elapsed since the last reception of mobile information (including virtual frames) (S111). If it determines that a certain amount of time has elapsed (S111 is yes), it performs a process that advances the last mobile information acquisition time stored in the memory by a predetermined time and delays it by a predetermined time (S112). Conversely, if it determines that a certain amount of time has not elapsed (S111 is no), the mobile information receiving unit 31a terminates the process.

[0277] Figure 19B This is a flowchart illustrating another example of the operation (delay detection method) of the remote control system 1 involved in this variation.

[0278] like Figure 19B As shown, when the mobile information receiving unit 31a determines that a certain amount of time has elapsed (S111 is yes), the terminal 30 acquires the time when the last mobile information acquisition time stored in the memory is ahead of a predetermined time (S113).

[0279] Next, terminal 30, for example, is based on Figure 15BThe terminal 30 obtains the estimated delay time (S114) from the table shown, waits for the estimated delay time (S115), and updates the mobile information acquisition time in the memory (S116). In step S116, the terminal 30 updates the mobile information acquisition time in the memory to a time that makes the previous mobile information acquisition time advance by a predetermined time.

[0280] Figure 20 This is a diagram illustrating details of another example of the operation (delay detection method) of the remote control system 1 involved in this variation. Figure 20 In the diagram, t1 represents a timing diagram where a display delay t1 exists as a delay time, and a certain amount of time is determined to have elapsed at time T after receiving frame B.

[0281] like Figure 20 As shown, since humans can predict motion between frames, the moving body information receiving unit 31a considers that a certain amount of time has elapsed at time T as having received the virtual frame B1 at time T and updates the timestamp. Furthermore, the image rendering is not updated before and after time T.

[0282] In this case, for the operation signals received from time T after a display delay t1 until the next frame C is received after a display delay t1, a timestamp of frame B1 is appended. This allows for correction of excessively long delays. For example, it can suppress situations where delays are mistakenly perceived as delays due to accidental frame drops, even when there are no abnormalities (e.g., delays less than a specified value).

[0283] (Modification 2 of Implementation Method 4)

[0284] The following is for reference Figure 21A and Figure 21B This section explains the delay detection method involved in this variation. Figure 21A This is a flowchart illustrating an example of the operation (delay detection method) of the remote control system 1 involved in this variation. In this variation, the updating of the timestamp in the case of a low frame rate is explained.

[0285] Furthermore, in this variant, control is based on the premise that, when the frame rate is low but above a certain level, a person can predict motion in their brain (inter-frame prediction), and therefore can perform accurate operations even without considering latency.

[0286] like Figure 21A As shown, the mobile information receiving unit 31a determines whether frame loss has occurred (S121). If frame loss is determined to have occurred (S121 is yes), it performs a process that advances the last mobile information acquisition time stored in the memory by a predetermined time and delays the time by a predetermined time (S112). Furthermore, if the mobile information receiving unit 31a determines that no frame loss has occurred (S121 is no), it terminates the process.

[0287] Figure 21B This is a flowchart illustrating another example of the operation (delay detection method) of the remote control system 1 involved in this variation.

[0288] like Figure 21B As shown, when the mobile information receiving unit 31a determines that a frame loss has occurred (S121 is yes), it executes... Figure 19B The processing after step S113 (S113~S116) is shown.

[0289] The timing diagram in this variation and Figure 20 Similarly, in the timing diagram shown, if a frame is lost at time T, it is considered that virtual frame B1 is received at time T, and the timestamp is updated.

[0290] (Implementation Method 5)

[0291] The following is for reference Figure 22 This embodiment explains the delay detection method, etc. Figure 22 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example is described of determining whether the operator remotely controls the mobile body 10 while observing the mobile body information sent by the mobile body 10.

[0292] like Figure 22 As shown, the attachment part 15 of the mobile body 10 attaches the electronic signature information a to the mobile body information (S131). The electronic signature information a attached to the mobile body information is an example of the first signature information.

[0293] The mobile body information receiving unit 31a of the terminal 30 receives mobile body information, obtains and stores time A and electronic signature information a from the received mobile body information (S132). In addition, the operation signal transmitting unit 32 appends the time A and electronic signature information a stored in step S132 to the operation signal (S133), and transmits the operation signal with the appended time A and electronic signature information a to the mobile body 10 (S17).

[0294] Next, the operation signal receiving unit 11c of the mobile body 10 receives the operation signal from the terminal 30 and extracts the time A and electronic signature information a from the received operation signal (S134). The electronic signature information a extracted from the operation signal is an example of second signature information.

[0295] Next, the verification unit 16 verifies the electronic signature information a extracted in step S134 based on the electronic signature information a attached to the mobile information (S135). This verification is to determine whether the operator is remotely controlling the mobile information sent by the mobile 10 while observing it. For example, the verification unit 16 determines whether the first signature information and the second signature information are consistent. In addition to the case of operation signal delay, the MRM execution unit 14 also executes MRM if the verification of the electronic signature information a fails (S20).

[0296] Therefore, when the operation signal sent to mobile body 10 is an operation signal sent by the operator after observing an image sent by a mobile body different from mobile body 10, it can be detected by verifying the electronic signature information a. When observing an image other than mobile body 10, the operation content may not be suitable for mobile body 10, so the security of mobile body 10 can be ensured by executing MRM.

[0297] In addition, the second signature information may be sent from terminal 30 at the start of remote control, for example.

[0298] (Modification 1 of Implementation Method 5)

[0299] The following is for reference Figure 23 This section explains the delay detection method involved in this variation. Figure 23 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this variation. In this variation, it is explained that a switching signal for switching from automatic driving to remote-controlled driving has been sent (e.g., Figure 9 This example illustrates whether terminal 30 (shown as an operation start signal) is performing a remote operation. Furthermore, the following primarily explains... Figure 22 The differences.

[0300] like Figure 23 As shown, when switching to remote operation, the operator performs an input indicating a switch to remote operation on the input device 40. The remote operation switching unit 35 of the terminal 30 obtains (e.g., receives) a switching signal indicating a switch to remote operation from the input device 40 (S141).

[0301] Next, if the remote operation switching unit 35 receives a switching signal, the encryption unit 36 ​​generates an encryption key and a decryption key (S142). The encryption unit 36 ​​generates a pair of encryption and decryption keys.

[0302] Next, the remote operation switching unit 35 transmits the switching signal and decryption key to the mobile body 10 (S143).

[0303] Next, if the decryption unit 18a of the mobile body 10 receives the decryption key, it retains the received decryption key (S144).

[0304] Next, if the operation signal receiving unit 11c receives a switching signal, it begins to receive operation signals from the terminal 30 (S145). That is, the mobile body 10 is remotely controlled.

[0305] The encryption unit 36 ​​of terminal 30 encrypts the operation signal with time A and electronic signature information a attached using the encryption key generated in step S142 (S146). The operation signal transmission unit 32 transmits the encrypted operation signal (S17).

[0306] If the decryption unit 18a of the mobile unit 10 receives the encrypted operation signal, it decrypts the operation signal using the latest decryption key it holds (S147). If the decryption is successful, it is possible to extract the time A and the electronic signature information a from the operation signal.

[0307] Thus, the encryption key generated by the terminal 30 that sent the switching signal is kept by the terminal 30, and the decryption key is kept by the mobile body 10. By being able to decrypt the operation signal encrypted and sent from the terminal 30 using the decryption key of the mobile body 10, it is possible to confirm whether the terminal 30 that sent the switching signal is remotely operating the mobile body 10, that is, to confirm whether the received operation signal was sent from the terminal 30 that sent the switching signal.

[0308] (Modification 2 of Implementation Method 5)

[0309] The following is for reference Figure 24 This section explains the delay detection method involved in this variation. Figure 24 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this variation. In this variation, an example of ensuring security in the remote control system 1 within the mobile body 10 is described. Specifically, an example of confirming that the operation signal from the terminal 30 has not been tampered with is described within the mobile body 10.

[0310] like Figure 24 As shown, during remote operation of the terminal 30, the vehicle encryption unit 18b generates two pairs of encryption and decryption keys (S151) and retains decryption keys I and II (S152). Figure 24 In the example, the vehicle encryption unit 18b generates encryption keys I and II as encryption keys, and generates decryption keys I and II as decryption keys. Encryption key I and decryption key I are a pair, and encryption key II and decryption key II are a pair.

[0311] Next, the vehicle encryption unit 18b uses encryption key I to encrypt the time (time A) when the moving body information is obtained (S153).

[0312] Next, the mobile body information acquisition unit 11a appends the encrypted time (time A) to the mobile body information (S154), and further appends the encryption key II to the mobile body information (S155).

[0313] If the mobile information receiving unit 31a of terminal 30 receives mobile information, it acquires the encrypted time A and the encryption key II and stores them (S156).

[0314] The operation signal transmitting unit 32 appends the encrypted time A obtained in step S156 to the operation signal (S157).

[0315] Next, the encryption unit 36 ​​encrypts the operation signal using encryption key II (S158). The operation signal transmission unit 32 transmits the encrypted operation signal (S17).

[0316] Next, the decryption unit 18a of the mobile body 10 decrypts the operation signal using the held decryption key II (S159).

[0317] Next, if the operation signal receiving unit 11c successfully decrypts the signal, it extracts time A from the operation signal (S160). At this time, time A is encrypted.

[0318] Next, the verification unit 16 uses the decryption key I to decrypt the encrypted time A (S161). Thus, if the decryption is successful, time A can be obtained. The MRM execution unit 14 also performs MRM in the case of operation signal delay, except in the case of decryption failure of time A (S20).

[0319] Thus, in this modified example, the encryption key I generated by the mobile body 10 is used to encrypt the time information attached to the mobile body information, and the encryption key II generated by the mobile body 10 is further used to encrypt the operation signal from the terminal 30. Therefore, the mobile body 10 can confirm that the operation signal sent from the terminal 30 is based on the operation of an operator who has observed the image sent by the mobile body 10 and that it has not been tampered with.

[0320] (Implementation Method 6)

[0321] The following is for reference Figure 25 This embodiment explains the delay detection method, etc. Figure 25 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example is described of the mobile body 10 determining whether data is transmitted and received in sequence.

[0322] like Figure 25As shown, the attachment unit 15 of the mobile body 10 attaches a counter to the mobile body information (S171) and increments the counter (S172). In step S172, the attachment unit 15 increments the counter by 1. The incremented counter is then attached to the next piece of mobile body information acquired. The counter attached to the mobile body information is an example of the first counter information.

[0323] The mobile information receiving unit 31a of the terminal 30 receives mobile information, obtains time A and counter from the received mobile information, and holds them (S173). In addition, the operation signal transmitting unit 32 appends the time A and counter held in step S173 to the operation signal (S174).

[0324] If the operation signal receiving unit 11c receives an operation signal, it extracts time A and the counter (S175). The counter included in the operation signal is an example of second counter information.

[0325] Next, the verification unit 16 verifies the counters included in the operation signals (S176). This verification is performed to determine whether multiple operation signals are received in sequence. If the counter extracted by the verification unit 16 in step 175 is less than the held counter, it is determined that the order (reception order) of the operation signals is abnormal, i.e., the operation signals are delayed. In the case of reversed order of the operation signals, MRM is executed. The held counter may, for example, be a counter added to the previous operation signal.

[0326] (Implementation Method 7)

[0327] The following is for reference Figure 26 This embodiment explains the delay detection method, etc. Figure 26 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example is described where the moving body 10 releases the MRM upon recovery from a delayed state.

[0328] The MRM execution unit 14 determines whether the delay during the execution of MRM in the mobile body 10 has fallen below a predetermined value. If the delay has fallen below the predetermined value (recovered), the MRM execution state is terminated (S181). The MRM execution unit 14 can be said to terminate the execution of MRM if the delay is eliminated during the execution of MRM.

[0329] Thus, the mobile body 10 determines whether to deactivate the MRM execution state, and if it determines that the MRM execution state has been deactivated, it automatically resumes autonomous driving.

[0330] (Implementation Method 8)

[0331] The following is for reference Figure 27 This embodiment explains the delay detection method, etc. Figure 27 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example of the control server 20 relaying moving body information and operation signals is described.

[0332] like Figure 27 As shown, the mobile information transmitting unit 11b transmits mobile information to the control server 20 (S12a), and the mobile information transmitting unit 22 of the control server 20 transmits the mobile information to the terminal 30 (S12b). Furthermore, the operation signal transmitting unit 32 sends an operation signal to the control server 20 (S17a), and the operation signal transmitting unit 23 of the control server 20 transmits the operation signal to the mobile body 10 (S17b).

[0333] MRM execution unit 14 performs MRM (S20) when the operation signal is delayed, and sends MRM information indicating that MRM has been performed to control server 20 (S191).

[0334] Next, if the control status management unit 21 of the control server 20 receives the MRM information, it cuts off the operation signal transmission system (S192). If the operation signal is delayed, the control status management unit 21 prohibits the relay of operation signals from the terminal 30 to the mobile body 10.

[0335] Thus, when the information exchange between the mobile body 10 and the terminal 30 is relayed / managed by the control server 20, if MRM is performed in the mobile body 10, the operation signal from the terminal 30 is cut off.

[0336] (Implementation Method 9)

[0337] The following is for reference Figures 28 to 29B This embodiment explains the delay detection method, etc. Figure 28 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example of performing statistical processing on the delay determination process of the operation signal is described.

[0338] like Figure 28 As shown, the delay determination unit 12 uses statistical processing (S201) to determine the delay of the operation signal. For example, the delay determination unit 12 performs statistical processing on the sum of the delay from acquiring mobile information from the mobile body 10 and displaying the mobile information to the operator, and the delay from the operator's operation to the response to the mobile body 10. The MRM execution unit 14 executes MRM (S20) if the value after statistical processing does not meet the predetermined conditions.

[0339] Here, refer to Figure 29A and Figure 29BThis section provides a specific example of the processing steps in step S201. Figure 29A and Figure 29B This is a flowchart illustrating various examples of the operation (delay detection method) of the remote control system 1 according to this embodiment.

[0340] like Figure 29A As shown, the delay determination unit 12 obtains the moving body information acquisition time from the received operation signal and calculates the delay (S211).

[0341] Next, the delay determination unit 12 calculates the number of times the delay time exceeds 1.5 seconds out of the past N delay times (S212). 1.5 seconds is an example of a predetermined time. In addition, the predetermined time is not limited to 1.5 seconds and can be other values.

[0342] Next, the delay determination unit 12 determines whether the number of delays is more than N (S213). If it is more than N (S213 is yes), it determines that a delay has occurred (S214). If it is less than N (S213 is no), the process ends because no delay has occurred.

[0343] Thus, statistical processing can also involve calculating the number of times a delay exceeding a predetermined time is recorded.

[0344] like Figure 29B As shown, after step S211, the delay determination unit 12 averages the delay times of the past 5 delays (S215). That is, the delay determination unit 12 calculates the average delay time (average delay). 5 times is an example of a predetermined number of times. Furthermore, the predetermined number of times is not limited to 5 times and can be other values.

[0345] Next, the delay determination unit 12 determines whether the average delay is above a specified value (S216). If the average delay is above the specified value (S216 is yes), it determines that a delay has occurred (S214). If the average delay is less than the specified value (S216 is no), the process ends because no delay has occurred.

[0346] Thus, statistical processing can also involve calculating the average of delay times over a predetermined period. Furthermore, it is not limited to calculating the average; for example, it can also calculate the mode, median, etc.

[0347] Therefore, it is possible to suppress situations where an abnormal delay is suddenly detected as a delay. Furthermore, the processing load for delay determination in the moving body 10 can be reduced.

[0348] Furthermore, statistical processing may also include calculating the variance of the delay time within a predetermined period. In this case, if the variance of the delay time is above a predetermined value, the delay determination unit 12 may determine it as abnormal because there is a possibility that the operability of the mobile body 10 will decrease due to the inability to perform certain operations. In the case of determining an abnormality, for example, MRM is executed. The delay determination unit 12 may also perform variance determination instead of the determination in step S216, for example, it may further determine variance if the determination in step S216 is negative.

[0349] (Implementation Method 10)

[0350] The following is for reference Figure 30 This embodiment explains the delay detection method, etc. Figure 30 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, an example considering the execution delay of the moving body 10 is described.

[0351] like Figure 30 As shown, if the operation signal execution unit 13 receives an operation signal, it executes the operation signal (S221). Furthermore, the moving body information acquisition unit 11a acquires moving body information (S222). The delay determination unit 12 calculates the time until the result of the operation signal is reflected based on the moving body information acquired in step S222, and performs a delay determination based on the calculated time. For example, if the operation signal is to increase the motor speed by a predetermined value, the time until the indicated speed is reached is calculated as the delay.

[0352] Furthermore, the current time shown in step S19 could also be the time when the action of the operation signal executed in step S221 is reflected to the moving body 10. Additionally, the MRM execution unit 14 executes MRM (S20) in the event of a delay.

[0353] Therefore, if the time from the operator's operation to the observation of the situation reflected in the moving body 10 is a certain period of time or more, MRM can be executed.

[0354] (Implementation Method 11)

[0355] The following is for reference Figure 31 This embodiment explains the delay detection method, etc. Figure 31 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, the case where the operator provides remote assistance is described. In remote assistance, the control signal is sent individually (not at regular intervals).

[0356] Figure 31Steps S11 to S234 show the processing of the operator (remote operator) while observing the image in order to operate buttons, etc.

[0357] In this case, the operation signal transmitting unit 32 does not receive an operation signal from the input device 40, therefore it appends time A to the time signal (S231) and transmits it to the mobile body 10 (S232). The time signal is a signal that includes the time (in this case, time A, an example of second time information) used to determine whether a delay has occurred during remote assistance by the operator. The time signal does not contain information for remotely controlling the mobile body 10 (e.g., control commands).

[0358] The delay determination unit 12 determines a delay in the time signal based on the received time signal (S233). The delay determination unit 12 determines whether a delay exists in the time signal. The delay determination unit 12 performs the determination in step S233 based on whether the value obtained by subtracting time A from the current time of the received time signal is greater than a predetermined value. If the value obtained by subtraction is greater than the predetermined value, it is determined that a delay has occurred, and MRM is executed (S234).

[0359] Thus, if a delay occurs during the stage of confirming the images used for remote assistance, MRM is executed.

[0360] Furthermore, if the operation signal transmitting unit 32 of the terminal 30 receives a single operation signal from the operation signal receiving unit 31b (S15), it appends time A to the operation signal (S16) and transmits the operation signal to the mobile body 10 (S17). This is equivalent to sending the image acquisition time to the mobile body 10.

[0361] The operation signal execution unit 13 executes the operation signal (S235).

[0362] Next, the delay determination unit 12 determines the timing signal delay (S236). The delay determination unit 12 performs the determination in step S236, for example, based on whether the value obtained by subtracting time A from the current time is greater than a predetermined value. If the value obtained by subtraction is greater than the predetermined value, it is determined that a delay has occurred, and MRM is executed (S20).

[0363] (Implementation Method 12)

[0364] The following is for reference Figure 32 This embodiment explains the delay detection method, etc. Figure 32 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 according to this embodiment. In this embodiment, the case of time rollback within the moving body 10 is described. In remote assistance, the control signal is sent individually (not at regular intervals).

[0365] like Figure 32 As shown, the time management unit 19 confirms whether the internal time of the moving body 10 has been rolled back (S241).

[0366] Next, if the MRM execution unit 14 determines that the internal time of the mobile body 10 has rolled back (i.e., there is a difference between the internal time and the reference time (e.g., GPS time),), it executes MRM (S242). The MRM execution unit 14 determines, for example, whether the internal time of the mobile body 10 has rolled back more than a predetermined time. The predetermined time is preset.

[0367] (A variation of Implementation Method 12)

[0368] The following is for reference Figure 33 This embodiment explains the delay detection method, etc. Figure 33 This is a timing diagram illustrating the operation (delay detection method) of the remote control system 1 involved in this modification. In this modification, the operation of the timing management unit 19 is explained when it is independent. In remote assistance, the control signal is sent individually (not at regular intervals).

[0369] like Figure 33 As shown, if the mobile body 10 acquires mobile body information (S251), the time management unit 19 appends the acquired time (time A) to the mobile body information (S252). Furthermore, after step S18, the time management unit 19 appends the acquired time (time A) to the mobile body 10 (S253).

[0370] In addition, the time management unit 19 checks whether the internal time has been rolled back (S255). For example, the time management unit 19 determines whether the internal time of the mobile body 10 has rolled back beyond a predetermined time. The predetermined time is set in advance.

[0371] (Other implementation methods)

[0372] The above description, based on embodiments and other means, describes delay detection methods and the like in one or more ways, but this disclosure is not limited to these embodiments. Various modifications to these embodiments that would be conceived by those skilled in the art, or combinations of constituent elements from different embodiments, may also be included in this disclosure as long as they do not depart from the spirit of the disclosure.

[0373] For example, in the above-described embodiments, the correction of each delay can be performed in either the mobile body 10 or the terminal 30. For example, the correction of each delay can be performed uniformly by the mobile body 10 or uniformly by the terminal 30. Furthermore, for example, the correction of each delay can also be performed by the control server 20.

[0374] Furthermore, in the above embodiments, an example of the remote control system 1 having a control server 20 has been described, but it is not limited to this. For example, the mobile body 10 and the terminal 30 may communicate directly without going through the control server 20, and the mobile body 10 and the terminal 30 may also have the functions of the control server 20.

[0375] Furthermore, in the above-described embodiments, each component can be constructed using dedicated hardware or implemented by executing software programs suitable for each component. Each component can be implemented by a program execution unit such as a CPU or processor reading and executing software programs stored on recording media such as a hard disk or semiconductor memory.

[0376] Furthermore, the execution order of the steps in the flowchart is illustrative for the purpose of illustrating this disclosure, and other orders are possible. Additionally, some of the above steps may be executed simultaneously (in parallel) with other steps, or some of the above steps may not be executed.

[0377] Furthermore, the division of functional blocks in the block diagram is one example. Multiple functional blocks can also be implemented as a single functional block, or a single functional block can be divided into multiple functional blocks, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar capabilities can also be processed in parallel or time-division by a single piece of hardware or software.

[0378] Furthermore, the delay detection system or terminal 30 included in the mobile body 10 described in the above embodiments can be implemented as a single device or by multiple devices. When the delay detection system or terminal 30 is implemented by multiple devices, the various components of the delay detection system or terminal 30 can be distributed among the multiple devices in any way. When the delay detection system or terminal 30 is implemented by multiple devices, the communication method between these multiple devices is not particularly limited; it can be either wireless communication or wired communication. Moreover, wireless communication and wired communication can be used in combination between the devices.

[0379] Furthermore, the constituent elements described in the above embodiments can also be implemented as software, or typically as an LSI (Liquid Crystal Interchange) for integrated circuits. They can be individually chip-based, or chip-based in a manner that includes some or all of them. Here, it is referred to as an LSI, but depending on the level of integration, it is sometimes also called an IC, system LSI, very large-scale LSI, or ultra-large-scale LSI. In addition, the method of integrated circuit implementation is not limited to LSI; it can also be implemented through dedicated circuits (general-purpose circuits that execute dedicated programs) or general-purpose processors. FPGAs (Field-Programmable Gate Arrays) that are programmable after LSI manufacturing or reconfigurable processors that can reconfigure the connection or configuration of circuit cells within the LSI can also be used. Furthermore, with the advancement of semiconductor technology or other derived technologies, if an integrated circuit implementation technology that replaces LSI emerges, then that technology can of course be used for the integration of constituent elements.

[0380] A system LSI is a multifunctional LSI that integrates multiple processing units onto a single chip. Specifically, it is configured as a computer system containing a microprocessor, ROM, RAM, etc. The computer program is stored in the ROM. The microprocessor executes the computer program, thereby enabling the system LSI to perform its functions.

[0381] Furthermore, another aspect of this disclosure is that the computer executes the code contained in [the present disclosure]. Figures 4 to 6E , Figures 12 to 15A , Figure 16 , Figure 19A , Figure 19B , Figures 21A to 33 The computer program for each characteristic step in any of the delay detection methods shown.

[0382] Furthermore, for example, the program can also be a program that enables a computer to execute. Additionally, one aspect of this disclosure can be a computer-readable, non-transitory recording medium on which such a program is recorded. For example, such a program can be recorded on a recording medium for distribution or circulation. For example, the distributed program can be installed into other devices having a processor, and the processor can execute the program, thereby causing the device to perform the aforementioned processes.

[0383] Industrial applicability

[0384] This disclosure is useful as a remote control system for remotely controlling a mobile body, etc.

[0385] Marker description

[0386] 1. Remote Control System

[0387] 10 moving bodies

[0388] 11a Mobile Entity Information Acquisition Department

[0389] 11b Mobile Information Transmission Unit (Transmission Unit)

[0390] 11c Operation Signal Receiver (Receiver)

[0391] 12. Delayed Decision-Making Unit (Decision-Making Unit)

[0392] 13 Operation signal execution unit

[0393] 14 MRM Execution Department

[0394] 15. Additional Section

[0395] 16 Verification Department

[0396] 17a Moving Body Information Acquisition Delay Correction Unit

[0397] 17b Operation Signal Execution Delay Correction Unit

[0398] 18a Decryption Department

[0399] 18b Vehicle Encryption Department

[0400] 19 Time Management Department

[0401] 20 Control Server

[0402] 21 Control Status Management Department

[0403] 22. Mobile Information Transmission Department

[0404] 23 Operation Signal Transmission Unit

[0405] 30 terminals

[0406] 31a Mobile Information Receiving Unit

[0407] 31b Operation signal receiving unit

[0408] 32 Operation signal transmitting unit

[0409] 33a Mobile Information Display Unit

[0410] 33b Delay time display unit

[0411] 34a Display Delay Correction Unit

[0412] 34b Operation signal acquisition delay correction unit

[0413] 34c Operator Response Delay Correction Unit

[0414] 35 Remote Operation Switching Unit

[0415] 36. Encryption Department

[0416] 40 Input Devices

[0417] a electronic signature information

[0418] At times A and T

[0419] N Mobile Information Transmission System

[0420] t1 Display Delay

[0421] t2 reaction delay

[0422] t3 signal acquisition delay

Claims

1. A delay detection method, wherein the delay detection method is performed by a mobile body that can be remotely controlled by an operator. First-time information is appended to the mobile body information representing the state of the mobile body and sent to the operator's terminal; Receive an operation signal, the operation signal being based on the operator's operation to the mobile body in response to the mobile body information, and including second time information based on the first time information contained in the mobile body information; Based on the second time information and the predetermined time, a delay in the remote control system containing the mobile body and the terminal is detected.

2. The delay detection method as described in claim 1, The predetermined time is the current time. In the detection of the delay, the difference between the second time information and the current time is calculated, and the delay is detected based on the calculated difference.

3. The delay detection method as described in claim 2, In the detection of the delay, the difference between the second time information contained in the most recently received operation signal and the current time is calculated periodically.

4. The delay detection method as described in claim 1, If the delay is detected, the moving body is made to perform a predetermined action.

5. The delay detection method as described in claim 1, If the internal time of the moving body is regressed by a predetermined amount, the moving body is made to perform a predetermined action.

6. The delay detection method as described in claim 4 or 5, Performing the predetermined action includes performing the Minimum Risk Management (MRM) operation.

7. The delay detection method as described in claim 4 or 5, The operation signal includes control commands based on the operator's operation. The predetermined action includes ignoring the control command contained in the received operation signal.

8. The delay detection method as described in claim 1, The first time information includes a time that is corrected based on the time up to the time before the predetermined information is acquired for the time when the predetermined information is acquired.

9. The delay detection method as described in claim 1 or 8, The second time information includes a time obtained by correcting the first time information based on the time from receiving the mobile body information to displaying it in the terminal.

10. The delay detection method as described in claim 1 or 8, The second time information is corrected based on the time from receiving the operation signal to the completion of the control corresponding to the operation signal in the mobile body, and the delay in the remote control system is detected based on the corrected second time information and the predetermined time.

11. The delay detection method as described in any one of claims 1 to 5, The mobile entity information is further appended with first signature information and sent to the operator's terminal. Receive the operation signal that also contains the second signature information. The second signature information is verified based on the first signature information.

12. The delay detection method as described in claim 11, The second signature information is the information sent from the terminal when remote control begins.

13. The delay detection method as described in any one of claims 1 to 5, The mobile body information is further appended with first counter information and sent to the operator's terminal. Receive the operation signal that also includes information about the second counter. Verify the information of the second counter.

14. The delay detection method as described in claim 6, If a delay is eliminated during the execution of the MRM, the execution of the MRM is terminated.

15. The delay detection method as described in any one of claims 1 to 5, When the operator does not remotely control the moving body, a time signal containing the second time information is received from the terminal.

16. The delay detection method as described in claim 1, The first time information is encrypted using the first encryption key, the encrypted first time information is appended to the mobile body information, and then sent to the terminal. The second time information contained in the received operation signal is decrypted using a first decryption key corresponding to the first encryption key.

17. The delay detection method as described in claim 1 or 16, A second encryption key is also attached to the mobile body information and sent to the terminal. Receive the operation signal encrypted using the second encryption key. The received operation signal is decrypted using a second decryption key corresponding to the second encryption key.

18. The delay detection method as described in any one of claims 1 to 5, The time represented by the first time information and the time represented by the second time information are the same time.

19. A delay detection device, which is a delay detection device for a mobile body that can be remotely controlled by an operator, comprising: The transmitting unit appends first-time information to the mobile body information indicating the state of the mobile body and transmits it to the operator's terminal; The receiving unit receives an operation signal, the operation signal being based on an operation performed by the operator on the mobile body information, and including second time information based on the first time information contained in the mobile body information; and The determination unit detects a delay in the remote control system, which includes the mobile body and the terminal, based on the second time information and the predetermined time.

20. A program that causes a computer to execute the delay detection method as described in any one of claims 1 to 5, 8, and 16.

Images