Remote control system, control method and control device

Abstract

To enable an operator who performs a remote operation to recognize a state of a moving object when a communication state between the moving object and a control device deteriorates.SOLUTION: A remote control system S has a moving object 2 that is remotely controlled and a control device 1 that remotely controls the moving object. The moving object 2 has a transmission unit 262 that transmits traveling state information to the control device 1, the traveling state information indicating a traveling state of the moving object 2 and indicating whether or not it is being remotely controlled from the control device 1. The control device 1 has: an acquisition unit 131 that acquires the traveling state information transmitted from the moving object 2; and a display control unit 133 that, when the moving object 2 releases the remote control and is unable to acquire the traveling state information until a stop state detection period, which is a predetermined period, has elapsed, in order to detect a stopping control state in which the moving object is stopped, displays on a display unit that the moving object is in a state which is estimated to be the stopping control state.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a control system, a control method, and a control device.

Background Art

[0002] There is known a system in which a plurality of cameras are mounted on a vehicle and remote operation is performed while viewing the images of the cameras (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional remote driving technology, when the communication status between a moving body such as a vehicle and a control device deteriorates, there may be a situation where the state of the vehicle managed on the control device side does not match the actual state on the moving body side, resulting in safety problems.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to enable an operator who performs remote operation to recognize the state of a moving body when the communication status between the moving body and the control device deteriorates.

Means for Solving the Problems

[0006] In the remote control system according to the first aspect of the present invention, there is provided a remote control system including a moving body that is remotely controlled and a control device that remotely controls the moving body. The moving body includes a moving body transmission unit that transmits, to the control device, running state information indicating a running state of the moving body and also indicating whether the moving body is remotely controlled by the control device. The control device includes a device acquisition unit that acquires the running state information transmitted from the moving body, and a display control unit that causes a display unit to display that the moving body is in a stop control state when the running state information cannot be acquired until a stop state detection period, which is a predetermined period set in advance for detecting a stop control state in which the moving body releases remote control and stops, has elapsed.

[0007] When the display control unit acquires the running state information indicating that the moving body is not in a state of being remotely controlled from the moving body, the display control unit may cause the display unit to display that the moving body is in the stop control state.

[0008] (1) When the display control unit acquires the running state information indicating that the moving body is not in a state of being remotely controlled from the moving body, the display control unit causes the display unit to display that the moving body is in the stop control state; (2) when the running state information cannot be acquired until the stop state detection period has elapsed, the display control unit may cause the display unit to display that the moving body is in a state where it is presumed to be in the stop control state.

[0009] When the display control unit causes the display unit to display that the moving body is in the stop control state, and when the device acquisition unit acquires the running state information, the display control unit may cause the display unit to display that the moving body is in the remote control state.

[0010] When the running state information cannot be acquired until the stop state detection period has elapsed, the display control unit may cause the display unit to display, in a display mode different according to the length of the period during which the running state information cannot be acquired, that the moving body is in a state where it is presumed to be in the stop control state.

[0011] If the display control unit is unable to obtain the travel state information until an unstable detection period, which is shorter than the stop state detection period, elapses, the display unit may be caused to display that the communication with the moving body is unstable.

[0012] The control device further includes a device transmission unit that transmits operation instruction information for instructing the operation content of the remotely controlled moving body to the moving body. The moving body includes a moving body receiving unit that receives the operation instruction information transmitted from the control device, and a state control unit that shifts from the remote control state to the stop control state and stops the moving body when the operation instruction information is not received until a release determination period, which is a predetermined period for the moving body receiving unit to determine the cancellation of remote control, elapses. The moving body may further include a moving body output unit that outputs information for alerting the outside of the moving body when the state control unit stops the moving body.

[0013] In the remote control system according to the second aspect of the present invention, there is provided a remote control system including a moving body remotely controlled and a control device for remotely controlling the moving body. The moving body includes a moving body receiving unit that obtains operation instruction information for instructing the operation content of the remotely controlled moving body from the control device, and a moving body transmission unit that transmits operation reflection information indicating the operation content reflected on the moving body among the operation content indicated by the obtained operation instruction information to the control device. The control device includes a device acquisition unit that acquires the operation reflection information transmitted from the moving body, and a display control unit that causes the display unit to display that the moving body is in a state presumed to be the stop control state when the operation reflection information cannot be acquired until a stop state detection period, which is a predetermined period for detecting a stop control state in which the moving body releases remote control and stops, elapses.

[0014] The mobile body transmission unit further transmits the video imaged by the imaging means mounted on the mobile body to the control device, the device acquisition unit further acquires the video, and the display control unit: (1) superimposes the position of the mobile body calculated based on the operation reflection information on the video and displays it on the display unit, and (2) when the operation reflection information cannot be acquired until the stop state detection period defined to ensure the accuracy of the position of the mobile body calculated based on the operation reflection information has elapsed, the display unit may display that the mobile body is in the stop control state.

[0015] When the operation reflection information cannot be acquired until an unstable detection period, which is a period shorter than the stop state detection period, has elapsed, the display control unit may display on the display unit that the communication with the mobile body is unstable.

[0016] If the operation instruction information is not acquired until a release determination period, which is a predetermined period for determining the cancellation of remote control, has elapsed, the mobile body may further include a state control unit that shifts from the remote control state to the stop control state and stops the mobile body.

[0017] When the state control unit stops the mobile body, the mobile body may further include a mobile body output unit that outputs information for alerting the outside of the mobile body.

[0018] In the control method according to the third aspect of the present invention, a step of periodically acquiring running state information, which is the running state of a remotely controlled mobile body and indicates whether the mobile body is remotely controlled or not, executed by a computer, and when the running state information cannot be acquired until a predetermined period defined for detecting a stop control state in which the mobile body releases remote control and stops has elapsed, a step of displaying on a display unit that the mobile body is in a state estimated to be the stop control state.

[0019] In the control method according to the fourth aspect of the present invention, which is a control method for a control device to remotely control a moving body, the moving body includes a step of acquiring operation instruction information for instructing the operation content of the remotely controlled moving body from a control device that controls the moving body, and a step of transmitting operation reflection information indicating the operation content reflected on the moving body among the operation content indicated by the acquired operation instruction information to the control device. The control device includes a step of acquiring the operation reflection information transmitted from the moving body, and a step of causing a display unit to display that the moving body is in a state presumed to be a stop control state when the operation reflection information cannot be acquired until a predetermined period elapses for detecting a stop control state in which the moving body releases remote control and stops.

[0020] In the control device according to the fifth aspect of the present invention, which is a running state of a moving body remotely controlled, it includes an acquisition unit that periodically acquires running state information indicating whether or not it is remotely controlled, and a display control unit that causes a display unit to display that the moving body is in a stop control state when the running state information cannot be acquired until a predetermined period elapses for detecting a stop control state in which the moving body releases remote control and stops.

[0021] An apparatus acquisition unit that acquires operation reflection information transmitted from a moving body remotely controlled, and a display control unit that causes a display unit to display that the moving body is in a state presumed to be a stop control state when the operation reflection information cannot be acquired until a predetermined period elapses for detecting a stop control state in which the moving body releases remote control and stops.

Advantages of the Invention

[0022] According to the present invention, an operator performing remote operation can recognize the state of the moving body when the communication situation between the moving body and the control device deteriorates.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0024] [Overview of Remote Control System S] FIG. 1 is a diagram for explaining the overview of the remote control system S. The remote control system S includes a control device 1 and a moving body 2. The remote control system S may further include an information terminal used by an operator U who operates the moving body 2.

[0025] The control device 1 is a device for remotely operating and controlling the moving body 2. The control device 1 receives an operation from an operator U who remotely operates the moving body 2, and controls the moving body 2 based on the received operation. Further, the control device 1 acquires information regarding the position, state, or external environment of the moving body 2, and displays the information of the moving body 2 so that the operator U can recognize the state of the moving body 2.

[0026] The moving body 2 moves under the control from the control device 1. The moving body 2 is, as an example, a vehicle traveling on a road. The moving body 2 may be a flying drone, or may be a drone moving in water or on water. In the following description in this specification, the case where the moving body 2 is a vehicle will be described. However, if it is a flying drone, it may perform an emergency landing instead of stopping. If it is a drone moving in water, it may be configured to be able to select an emergency or emergency surfacing instead of stopping.

[0027] When the remote control of the moving body 2 is started by a signal from the control device 1, the moving body 2 transitions to a state of being remotely controlled by the control device 1 (hereinafter referred to as the "remote control state"). When the moving body 2 falls into a state where it is difficult to be remotely controlled, the remote control state is released and the moving body 2 stops (hereinafter referred to as the "stop control state"). A state where it is difficult to be remotely controlled is, for example, a case where a signal from the control device 1 does not reach for a predetermined period or more. The moving body 2 may transition to the stop control state when it receives a signal indicating that the remote control state is released from the control device 1.

[0028] The processing of the remote control system S will be described with reference to FIG. 1(a). The operator U operates the moving body 2 ((1) in FIG. 1). The operator U may operate the moving body 2 via the user interface of the information terminal used by the operator U, or may operate the moving body 2 via the user interface of the control device 1.

[0029] The control device 1 transmits operation instruction information to the moving body 2 ((2) in FIG. 1). The operation instruction information is information for instructing the remotely controlled moving body 2 to perform an operation corresponding to the operation content input by the operator U. When the moving body 2 is a vehicle, the operation instruction information includes, for example, the operation amount of the steering, the operation amount of the accelerator, and the operation amount of the brake.

[0030] The mobile body 2 periodically transmits driving state information to the control device 1 ((3) in FIG. 1). The driving state information indicates whether the mobile body 2 is remotely controlled. The mobile body 2 periodically transmits operation reflection information to the control device 1 ((4) in FIG. 1). The operation reflection information is information corresponding to the operation instruction information, and indicates that the operation content indicated by the operation instruction information received by the mobile body 2 has been reflected in the control of the mobile body 2. As an example, in the operation reflection information, among the operation contents indicated by the operation instruction information, the reflected operation content is included. In the operation reflection information, a time stamp indicating the time when the operation instruction information was reflected may be included. In the operation reflection information, information such as the speed, acceleration, and position of the mobile body 2 may be included.

[0031] The mobile body 2 periodically transmits image data indicating the external situation imaged by the imaging means mounted on the mobile body 2 to the control device 1 ((5) in FIG. 1). As an example, the mobile body 2 periodically transmits images obtained by imaging the front, rear, and sides of the mobile body 2 to the control device 1. In the image data, a time stamp indicating the time when the image was taken may be associated.

[0032] The control device 1 periodically determines the state of the mobile body 2 ((6) in FIG. 1). As an example, when the control device 1 obtains driving state information indicating that it is in a remote control state, the control device 1 determines that the mobile body 2 is in a remote control state. Also, when it is shown in the obtained driving state information that the mobile body 2 is in a stop control state, the control device 1 determines that the mobile body 2 is in a stop control state.

[0033] The control device 1 outputs the determined state of the mobile body 2 ((7) in FIG. 1). As an example, the control device 1 causes the display unit to display a screen in which the determined state is superimposed on the image data obtained from the mobile body 2. The control device 1 may output the determined state to the display unit mounted on the control device 1, or may output the determined state to the information terminal used by the operator U via the network.

[0034] When communication becomes unstable due to network delays, failures, etc., the operator U can recognize that remote control is difficult, enabling necessary operations such as stopping to be performed. The processing in this case will be described with reference to Fig. 1(b). When the state of not acquiring the traveling state information continues until it exceeds the stop state detection period ((3b) in Fig. 1), the control device 1 determines that the moving body 2 is in a presumed stop state ((6b) in Fig. 1). The presumed stop state is a state where, although it is not certain that it is in the stop control state, the moving body 2 is presumed to be in the stop control state. The control device 1 outputs that it is in the presumed stop state ((7b) in Fig. 1).

[0035] With the remote control system S configured in this way, when the communication situation between the moving body 2 and the control device 1 deteriorates, the operator U can recognize the state of the moving body 2.

[0036] [Configuration of Control Device 1] Fig. 2 is a block diagram showing the configuration of the control device 1. The control device 1 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an acquisition unit 131, a determination unit 132, a display control unit 133, an operation reception unit 134, and a transmission unit 135. Note that the control device 1 may further include an input unit for inputting operations to be performed on the moving body 2 and a display unit for displaying the status of the moving body 2.

[0037] The communication unit 11 is a communication interface for transmitting and receiving data with other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance the programs executed by the control unit 13.

[0038] The control unit 13 is a processor such as a CPU (Central Processing Unit), for example. By executing the program stored in the storage unit 12, the control unit 13 functions as an acquisition unit 131, a determination unit 132, a display control unit 133, an operation reception unit 134, and a transmission unit 135.

[0039] The acquisition unit 131 acquires the traveling state information transmitted from the moving body 2. The traveling state information indicates the traveling state of the moving body 2. As an example, the traveling state information indicates whether the moving body 2 is remotely controlled. FIG. 3 is a diagram showing an example of the data structure of the traveling state information. The traveling state information includes a "traveling state ID", a "moving body ID", a "date and time", and a "traveling state". The "traveling state ID" is an ID (Identification) for identifying the transmitted traveling state information. The "moving body ID" is an ID for identifying the moving body 2 that transmits the traveling state information. The "date and time" indicates the date and time when the traveling state information is generated or transmitted. The "traveling state" is information indicating whether the moving body 2 is in a remote control state or a stop control state. The acquisition unit 131 acquires the traveling state information periodically transmitted from the moving body 2. The acquisition unit 131 acquires the image data transmitted from the moving body 2.

[0040] The operation reception unit 134 receives an operation performed on the moving body 2 by the operator U. The operation reception unit 134 may receive an operation input at the user interface of the information terminal used by the operator U via the communication unit 11, or may receive an operation input at the user interface (not shown) of the control device 1. The transmission unit 135 transmits operation instruction information corresponding to the operation received by the operation reception unit 134 to the moving body 2. The transmission unit 135 transmits the operation instruction information to the moving body 2.

[0041] The display control unit 133 causes the display unit to display that the mobile body 2 is in a presumed stop state when the running state information cannot be acquired before the stop state detection period elapses. The stop state detection period is a predetermined period set in advance to detect a stop control state in which the mobile body 2 is in a state of releasing remote control and stopping. The stop state detection period may be determined based on, for example, the purpose of movement of the mobile body 2 (such as home delivery, transportation, etc.), the presence or absence of a person riding on the mobile body 2, the speed at which the mobile body 2 moves, the traffic volume at the location where the mobile body 2 is located, and other properties. As an example, when a person is riding on the mobile body 2, a shorter stop state detection period may be set compared to the case where a person rides.

[0042] The determination unit 132 measures the time elapsed from the timing when the acquisition unit 131 acquires the running state information (hereinafter referred to as "elapsed time"). When the acquisition unit 131 newly acquires the running state information, the determination unit 132 resets the elapsed time being measured and newly starts measuring the elapsed time. When the elapsed time exceeds the stop state detection period, the determination unit 132 determines that the mobile body 2 is in a presumed stop state. When the determination unit 132 determines that the mobile body 2 is in a presumed stop state, the display control unit 133 causes the display unit to display predetermined information. The predetermined information is information for causing an operator who remotely operates the mobile body 2 to recognize that the mobile body 2 is in a presumed stop state.

[0043] FIG. 4 is a diagram showing an example of a screen to be displayed by the display control unit 133. In the screen shown in FIG. 4, an object (O1) indicating the determined state of the mobile body 2 is superimposed and displayed on the image data acquired by the acquisition unit 131. That is, when the determination unit 132 determines that the mobile body 2 is in a presumed stop state, it is displayed in the object O1 that the mobile body 2 is in a presumed stop state. The display control unit 132 may display the object O1 in different modes depending on whether it is in a remote control state or a presumed stop state. For example, when in a remote control state, the background of the object O1 may be displayed in a color such as white, colorless, or blue, and when in a presumed stop state, the background of the object O1 may be displayed in a color having a warning meaning such as yellow or red.

[0044] With the control device 1 configured in this way, when the communication situation between the moving body 2 and the control device 1 deteriorates, the operator performing the remote operation can recognize the state of the moving body 2.

[0045] When information indicating that the moving body 2 is not under remote control is acquired, the control device 1 may be configured to display information for the operator to recognize the state of the moving body 2.

[0046] When the display control unit 133 acquires the traveling state information indicating that the moving body 2 is not in a state of being remotely controlled, the display control unit 133 causes the display unit to display that the moving body 2 is in the stop control state. That is, when the information indicating the stop control state is included in the traveling state information acquired by the acquisition unit 131, the display control unit 133 causes the display unit to display a screen indicating the stop control state. FIG. 5 is a diagram showing an example of the screen to be displayed by the display control unit 133. In FIG. 5, it is displayed that the object O1 is in the stop control state. In the screen shown in FIG. 5, an object (O2) displayed as "Resume Remote Operation" is displayed. When the operator U operates the object O2, the control device 1 transmits information for requesting to resume the remote control to the moving body 2.

[0047] When the determination unit 132 determines the stop estimation state and the traveling state information indicating the remote control state is acquired, the control device 1 may be configured to cause the display unit to display that it is in the remote control state. When the display control unit 133 causes the display unit to display that the moving body 2 is in the stop control state and the acquisition unit 131 acquires the traveling state information indicating the remote control state, the display control unit 133 causes the display unit to display that the moving body 2 is in the remote control state.

[0048] The stop state detection period may be detected by a plurality of thresholds. As an example, two-stage thresholds are set for the stop state detection period. Hereinafter, the stop state detection period set for a shorter period is referred to as the first stop state detection period, and the stop state detection period set for a longer period is referred to as the second stop state detection period.

[0049] When the control unit 133 cannot obtain the traveling state information before the stop state detection period elapses, the control unit 133 causes the display unit to display that the moving body 2 is in the estimated stop state in different display modes according to the length of the period during which the traveling state information cannot be obtained. As described above, the determination unit 132 measures the time elapsed since the timing when the traveling state information was obtained from the moving body 2, and when the elapsed time exceeds the first stop state detection period, causes the display unit to display that the moving body is in the estimated stop state in the first mode. The first mode is a mode in which it is easy to attract the attention of the operator U that the moving body is in the estimated stop state. The first mode displays that the moving body is in the estimated stop state using a background color such as yellow that implies caution. In the first mode, it may be displayed that the communication state is unstable.

[0050] When the elapsed time exceeds the second stop state detection period, the display control unit 133 causes the display unit to display that the moving body is in the estimated stop state in the second mode. In the second mode, it is displayed that the moving body is in the estimated stop state in a mode that is more likely to attract the attention of the operator U than the first mode. In the second mode, for example, the fact that the moving body is in the estimated stop state is displayed using a background color such as red that implies more caution.

[0051] When it is difficult to obtain the information acquired from the moving body 2 due to communication delay or the like in the network, the control device 1 may be configured to display that the communication state is unstable.

[0052] The display control unit 133 causes the display unit to display that the communication with the mobile body 2 is unstable when it is unable to acquire the traveling state information before the unstable detection period elapses. The display control unit 133 may display that the communication with the mobile body 2 is unstable in a manner (e.g., a different background color) different from the case of displaying the stop estimation state. The unstable detection period is a period set to determine whether to notify the operator that the communication is in an unstable state, and is set shorter than the stop state detection period. As an example, the unstable detection period is set to be longer than the maximum value of the delay time assumed in the normal operation state. As described above, the determination unit 132 measures the elapsed time. When the elapsed time exceeds the unstable detection period, the display control unit 133 causes the display unit to display that the communication is unstable.

[0053] [State of the mobile body 2 managed by the control device 1] FIG. 6 is a state transition diagram showing the state of the mobile body 2 managed by the control device 1. It is assumed that the control device 1 is in the initial state when it is not connected to the mobile body 2.

[0054] When the operation reception unit 134 receives an operation to start remote control ((1) in FIG. 6), the control device 1 establishes a connection with the mobile body 2, and the determination unit 132 determines that the mobile body 2 is in the remote control state.

[0055] In the remote control state, the display control unit 133 causes the display unit to display the image indicated by the image data acquired from the mobile body 2. In the remote control state, the operation reception unit 134 receives an operation to be performed on the mobile body 2 from the operator U, and transmits operation instruction information corresponding to the received operation to the mobile body 2. In the remote control state, the acquisition unit 131 acquires the traveling state information.

[0056] When the acquisition unit 131 acquires the driving state information indicating that the mobile body 2 is in the stop control state in the remote control state (2) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the stop control state. Further, when the operation reception unit 134 receives an operation to cancel the remote operation in the remote control state (3) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the stop control state. When the determination unit 132 determines that the mobile body 2 is in the stop control state, the display control unit 133 causes the display unit to display that it is in the stop control state.

[0057] When the acquisition unit 131 acquires the driving state information indicating that the mobile body 2 is in the remote control state in the stop control state (4) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the remote control state. Further, when the operation reception unit 134 receives an operation to resume the remote operation in the stop control state (5) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the remote control state.

[0058] When the driving state information is not acquired for a predetermined period in the remote control state (6) in FIG. 1, the determination unit 132 determines that the mobile body 2 is in the stop estimation state. When the determination unit 132 determines that the mobile body 2 is in the stop estimation state, the display control unit 133 causes the display unit to display that the mobile body 2 is in the stop estimation state.

[0059] When the acquisition unit 131 acquires the driving state information indicating that the mobile body 2 is in the remote control state in the stop estimation state (7) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the remote control state. When the acquisition unit 131 acquires the driving state information indicating that the mobile body 2 is in the stop control state in the stop estimation state (8) in FIG. 6, the determination unit 132 determines that the mobile body 2 is in the stop control state.

[0060] When an operation to end the remote control is received in the remote control state (9) in FIG. 6, the control device 1 disconnects the connection with the mobile body 2, and the determination unit 132 returns the state to the initial state. [Configuration of Mobile Body 2] FIG. 7 is a block diagram showing the configuration of the moving body 2. The moving body 2 includes an imaging unit 21, an output unit 22, a movement control unit 23, a communication unit 24, a storage unit 25, and a control unit 26. The control unit 26 includes a transmission unit 262, a reception unit 261, a state control unit 263, and an output unit 264.

[0061] The imaging unit 21 is, for example, a camera. The imaging unit 21 generates image data by imaging the internal or external situation of the moving body 2 under remote control. The imaging unit 21 inputs the generated image data to the transmission unit 262. The output unit 22 performs an output based on the control of the output unit 264. As an example, the output unit may be an LED display or a display for transmitting information by light or video, or a speaker for transmitting information by sound.

[0062] The movement control unit 23 controls the devices mounted on the moving body 2 based on the operation instruction information acquired from the control device 1. The movement control unit 23 is, for example, an ECU (Electronic Control Unit). When the movement control unit 23 controls the moving body 2 based on the operation instruction information, the movement control unit 23 inputs operation reflection information corresponding to the operation instruction information reflected in the control to the transmission unit 262.

[0063] The reception unit 261 receives the operation instruction information transmitted from the control device 1. The reception unit 261 inputs the received operation instruction information to the movement control unit 23.

[0064] The transmission unit 262 transmits the traveling state information to the control device 1. As an example, the transmission unit 262 periodically generates the traveling state information based on the state of the moving body 2 managed by the state control unit 263 and transmits it to the control device 1. For example, the transmission unit 262 periodically inquires the state control unit 263 about the state of the moving body 2 and transmits the traveling state information including the result obtained from the state control unit 263 to the control device 1.

[0065] The transmission unit 262 acquires the operation reflection information obtained from the movement control means 23. The transmission unit 262 transmits the acquired operation reflection information to the control device 1. The transmission unit 262 acquires image data from the imaging means 21. The transmission unit 262 further transmits the image data captured by the imaging means 21 mounted on the moving body 2 to the control device 1.

[0066] The state control unit 263 manages the state of the moving body 2. When the state control unit 263 does not receive operation instruction information until the cancellation determination period elapses, it shifts from the remote control state to the stop control state and stops the moving body 2. The cancellation determination period is a predetermined period for the reception unit 261 to determine the cancellation of remote control. The state control unit 263 measures the period elapsed since the timing when the operation instruction information was acquired. When the state control unit 263 acquires the operation instruction information, it resets the period being measured and newly starts the measurement of the period. When the period elapsed since the timing when the operation instruction information was acquired exceeds the cancellation determination period, the state control unit 263 releases the remote control state and shifts to the stop control state. When the state control unit 263 shifts to the stop control state, it controls the movement control means 23 to stop the moving body 2. As an example, the movement control means 23 operates to increase the operation amount of the brake while maintaining the operation amount of the steering indicated by the last received operation instruction information, and stops the moving body 2. The movement control means 23 may control to gradually increase the operation amount of the brake. The movement control means 23 may autonomously control the moving body 2 to safely stop based on the sensors mounted on the moving body 2 or the images captured of the outside. By the moving body 2 performing such an operation, the moving body 2 can be safely stopped when the operation from the remote side becomes impossible.

[0067] When the state control unit 263 stops the moving body 2, the output unit 264 outputs information for alerting the outside of the moving body 2. FIG. 8 is a diagram showing an example of the information output by the output unit 264 in the stop control state. As an example, the output unit 264 causes the output means 22 mounted outside the moving body 2 to display, as characters, information for alerting that the moving body 2 is in a stopped state. Note that the output unit 264 may output, by voice, to the output means that the moving body 2 is in a stopped state, or may control to generate a warning sound.

[0068] With the remote control system S configured in this way, it is possible to prompt attention to the surroundings of the moving body 2.

[0069] [State of the moving body 2 managed by the control device 1] FIG. 9 is a state transition diagram showing the state of the moving body 2 managed by the state control unit 263. Assume that the control device 1 is in the initial state when not connected to the moving body 2.

[0070] When the reception unit 261 receives a signal to start remote control ((1) in FIG. 9), the moving body 2 establishes a connection with the moving body 2, and the state control unit 263 transitions the state of the moving body 2 to the remote control state. In the remote control state, the reception unit 261 acquires operation instruction information, the movement control means 23 controls the moving body 2 based on the operation instruction information acquired by the reception unit 261, and the transmission unit 262 transmits operation reflection information corresponding to the operation instruction information to the control device 1. In the remote control state, the transmission unit 262 periodically transmits traveling state information.

[0071] In the remote control state, if operation instruction information is not acquired beyond the release determination period ((2) in FIG. 9), the state control unit 263 transitions to the stop control state. In the stop control state, the state control unit 263 controls the movement control means 23 to stop the moving body 2, and the output unit 264 outputs information for attracting attention to the outside of the moving body 2. Also, in the remote control state, when the receiving unit 261 acquires a signal indicating that the remote control is released from the control device 1 ((3) in FIG. 9), the state control unit 263 transitions to the stop control state.

[0072] In the stop control state, when the receiving unit 261 acquires a signal indicating that the remote control is resumed from the control device 1 ((4) in FIG. 9), the state control unit 263 transitions to the remote control state. In the remote control state, when the receiving unit 261 acquires a signal to end the remote control from the control device 1 ((5) in FIG. 9), the moving body 2 disconnects the connection with the control device 1 and returns to the initial state.

[0073] With the remote control system S configured in this way, when the signal on the transmission side to the moving body 2 is interrupted during communication instability, the moving body can be stopped by appropriate means so as not to cause the moving body 2 to run wild, and when the signal transmitted from the moving body 2 is interrupted, the state is displayed to prevent excessive operation by the remote operator, and the moving body 2 can be appropriately stopped by stopping while maintaining the operation amount, enabling appropriate control of the moving body according to the communication situation. <Modification Example> In the above description, an example of determining the state of the moving body 2 based on the traveling state information has been described. However, the control device 1 may be configured to determine the stop estimation state based on the period during which the operation reflection information periodically transmitted by the moving body 2 cannot be acquired.

[0074] The acquisition unit 131 acquires the operation reflection information transmitted from the moving body 2. When the display control unit 133 cannot acquire the operation reflection information until the stop control state detection period elapses, the display control unit 133 causes the display unit to display that the moving body 2 is in the stop estimation state. That is, the determination unit 132 measures the period during which the operation reflection information is not acquired in the same manner as the above-described process, and determines the stop estimation state. Note that a threshold value different from the case of determining based on the traveling state information may be set for the stop state detection period in this case.

[0075] The control device 1 may be configured to display that the communication is unstable based on the period during which the operation reflection information periodically transmitted from the moving body 2 cannot be acquired. When the display control unit 133 cannot acquire the operation reflection information until the instability detection period elapses, the display control unit 133 causes the display unit to display that the communication with the moving body 2 is unstable. A threshold value different from the case of determining based on the traveling state information may be set for the instability detection period in this case.

[0076] With the control system S according to the modification configured in this way, it becomes possible to more accurately determine the state of the moving body 2. This is because the control system S according to the modification can also detect problems due to the communication state from the control device 1 to the moving body 2 in addition to the communication state from the moving body 2 to the control device 1.

[0077] Since the image data acquired by the control device 1 from the moving body 2 is data captured in the past compared to the timing of being displayed on the control device 1, a time lag may occur. Therefore, the control device 1 may be configured to superimpose and display the predicted position of the moving body 2 on the image data based on the operation reflection information received from the moving body 2.

[0078] In this case, the display control unit 133 identifies operation instruction information that was not reflected at the time when the image data was generated, based on the time stamp associated with the image data and the time stamp of the time when the operation instruction information was transmitted. The display control unit 133 calculates the position where the moving body 2 is expected to be located in the displayed image, based on the identified unreflected operation reflection information and the state of the moving body 2 (such as speed, acceleration, position, steering operation amount, etc.) at the time when the image was generated. The display control unit 133 causes the display unit to display, by superimposing on the video, the position of the moving body 2 calculated based on the operation reflection information.

[0079] In this case, when the communication delay increases, the deviation between the image data and the predicted position of the moving body 2 increases. Therefore, the stop state detection period may be determined based on the accuracy of the predicted position of the moving body 2. When the display control unit 133 cannot obtain the running state information for the stop state detection period determined to ensure the accuracy of the position of the moving body 2 calculated based on the operation reflection information, the display control unit 133 causes the display unit to display that the moving body 2 is in the stop control state. With the control device 1 configured in this way, it is possible to provide the operator U with information for determining the reliability of the information indicating the position of the moving body 2 superimposed on the image data.

[0080] The display control unit 133 may identify the location where the moving body 2 started the stop control, based on the acquired video, operation reflection information, and operation instruction information, and cause it to be displayed by superimposing on the acquired video. With this configuration, it is possible to provide information that assists in better understanding the state of the vehicle of the operator U.

[0081] The control device 1 may be configured to determine the stop estimation state when the deviation between the position of the moving body 2 at the time of acquiring the image data and the calculated predicted position exceeds a predetermined threshold value.

[0082] Note that the present invention can contribute to Goal 9, "Build the infrastructure for industry and innovation," of the Sustainable Development Goals (SDGs) led by the United Nations.

[0083] As described above, the present invention has been explained using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist. For example, all or part of the device can be configured by functionally or physically dispersing and integrating it in any unit. Also, new embodiments resulting from any combination of a plurality of embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined.

Explanation of Signs

[0084] 1 Control device 2 Mobile body 11 Communication unit 12 Storage unit 13 Control unit 131 Acquisition unit 133 Display control unit 134 Operation reception unit 135 Transmission unit 21 Imaging means 22 Output means 23 Mobile body control means 24 Communication unit 25 Storage unit 26 Control unit 261 Reception unit 262 Transmission unit 263 State control unit 264 Output unit

Claims

1. A remote control system comprising a mobile body controlled remotely and a control device for remotely controlling the mobile body, wherein the mobile body has a mobile body transmission unit that transmits to the control device travel state information indicating the travel state of the mobile body and indicating whether the mobile body is remotely controlled by the control device; the control device includes a device acquisition unit that acquires the travel state information transmitted from the mobile body; a display control unit that causes the display unit to display that the mobile body is in a state where it is presumed to be in a stop control state when the travel state information cannot be acquired until a stop state detection period, which is a predetermined period determined in advance for detecting a stop control state in which the mobile body releases remote control and stops, has elapsed; a remote control system having the above components.

2. When the display control unit acquires travel state information indicating that the mobile body is not in a state of being remotely controlled from the mobile body, the display control unit causes the display unit to display that the mobile body is in a stop control state. The remote control system according to Claim 1.

3. The display control unit 1. When acquiring travel state information indicating that the mobile body is not in a state of being remotely controlled from the mobile body, causes the display unit to display that the mobile body is in a stop control state; 2. When the travel state information cannot be acquired until the stop state detection period has elapsed, causes the display unit to display that the mobile body is in a state where it is presumed to be in a stop control state. The remote control system according to Claim 1.

4. When the display control unit causes the display unit to display that the mobile body is in a stop control state, and when the device acquisition unit acquires the travel state information, the display control unit causes the display unit to display that the mobile body is in a remote control state. The remote control system according to Claim 1.

5. When the travel state information cannot be acquired until the stop state detection period has elapsed, the display control unit causes the display unit to display, in a display mode different according to the length of the period during which the travel state information cannot be acquired, that the mobile body is in a state where it is presumed to be in a stop control state. The remote control system according to Claim 2.

6. When the travel state information cannot be acquired until an unstable detection period, which is a period shorter than the stop state detection period, has elapsed, the display control unit causes the display unit to display that the communication with the mobile body is unstable. The remote control system according to claim 1.

7. The control device further includes a device transmission unit that transmits operation instruction information for instructing the operation content of the moving body being remotely controlled to the moving body, The moving body a moving body receiving unit that receives the operation instruction information transmitted from the control device, a state control unit that, when the operation instruction information is not received until a release determination period, which is a predetermined period for the moving body receiving unit to determine the release of remote control, elapses, shifts from the remote control state to the stop control state and stops the moving body, and further includes a moving body output unit that outputs information for alerting the outside of the moving body when the state control unit stops the moving body. The remote control system according to claim 1.

8. A remote control system having a moving body controlled remotely and a control device for remotely controlling the moving body, The moving body a moving body receiving unit that acquires operation instruction information for instructing the operation content of the moving body being remotely controlled from the control device, a moving body transmission unit that transmits operation reflection information indicating the operation content reflected on the moving body among the operation content indicated by the acquired operation instruction information to the control device, and has The control device a device acquisition unit that acquires the operation reflection information transmitted from the moving body, a display control unit that causes the display unit to display that the moving body is in a state presumed to be the stop control state when the operation reflection information cannot be acquired until a stop state detection period, which is a predetermined period determined in advance to detect the stop control state in which the moving body releases remote control and stops, elapses. A remote control system having

9. The moving body transmission unit further transmits an image captured by an imaging means mounted on the moving body to the control device, The device acquisition unit further acquires the image, The display control unit (1) superimposes the position of the moving body calculated based on the operation reflection information on the image and displays it on the display unit, and (2) causes the display unit to display that the moving body is in the stop control state when the operation reflection information cannot be acquired until the stop state detection period determined to ensure the accuracy of the position of the moving body calculated based on the operation reflection information elapses. The remote control system according to claim 8.

10. When the display control unit cannot obtain the operation reflection information until an unstable detection period, which is shorter than the stop state detection period, elapses, the display unit is caused to display that the communication with the moving body is unstable. The remote control system according to claim 8.

11. The moving body further includes a state control unit that shifts from the remote control state to the stop control state and stops the moving body when the operation instruction information is not obtained until a release determination period, which is a predetermined period for determining the release of remote control, elapses. The remote control system according to claim 9.

12. when the state control unit stops the moving body, further includes a moving body output unit that outputs information for alerting the outside of the moving body. The remote control system according to claim 11.

13. executed by a computer, a step of periodically obtaining running state information indicating whether or not the moving body is remotely controlled, which is the running state of the remotely controlled moving body; a step of causing the display unit to display that the moving body is in a state presumed to be the stop control state when the running state information cannot be obtained until a predetermined period elapses for detecting a stop control state in which the moving body releases remote control and stops; and a control method having the above.

14. A control method for a control device to remotely control a moving body, wherein the moving body obtains operation instruction information for instructing the operation content of the remotely controlled moving body from a control device that controls the moving body; and transmits operation reflection information indicating the operation content reflected on the moving body among the operation content indicated by the obtained operation instruction information to the control device; and executes, wherein the control device obtains the operation reflection information transmitted from the moving body; and causes the display unit to display that the moving body is in a state presumed to be the stop control state when the operation reflection information cannot be obtained until a predetermined period elapses for detecting a stop control state in which the moving body releases remote control and stops; and executes a control method.

15. a acquisition unit that periodically acquires running state information indicating whether or not the moving body is remotely controlled, which is the running state of the remotely controlled moving body; When the running state information cannot be acquired until a predetermined period of time elapses for detecting a stop control state in which the mobile body releases remote control and stops, a display control unit that causes a display unit to display that the mobile body is in the stop control state; A control device having the same.

16. A device acquisition unit that acquires operation reflection information transmitted from a mobile body controlled remotely; When the operation reflection information cannot be acquired until a predetermined period of time elapses for detecting a stop control state in which the mobile body releases remote control and stops, a display control unit that causes a display unit to display that the mobile body is in a state estimated to be the stop control state; A control device having the same.

Images