Remote control system for forklift

Abstract

To provide a remote control system capable of providing useful and abundant information to an operator performing remote control while viewing remote video.SOLUTION: A remote control system 1A comprises an imaging device 20 installed on a forklift F, a head-mounted video display device 10 installed on the head of an operator O, a command receiving device 11 that receives remote control commands from the operator O, and a composite remote video generation device 12A that obtains details of the remote control commands received by the command reception device 11, the operating state of the forklift F that operates according to the remote control commands, and remote video captured by the imaging device 20, and generates a composite remote video by combining the remote video with a command display image showing the contents of the remote operation command and an operation display image showing the operating state of the forklift F. The head-mounted video display device 10 displays the generated composite remote video in front of the operator O's eyes.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote operation system that enables a forklift at a remote location to be remotely operated while viewing a remote video displayed on a head-mounted video display device.

Background Art

[0002] Conventionally, a remote operation system that enables a work machine at a remote location to be remotely operated while viewing a remote video displayed on a head-mounted video display device has been known. For example, Patent Document 1 describes a remote operation system for remotely operating a shovel as a work machine.

[0003] The system described in Patent Document 1 is configured such that the content of the image displayed on the head-mounted video display device is switched according to the movement of the operator's head detected by a wearable sensor. More specifically, when the operator's head pitches down, the displayed image switches from a forward video (remote video) taken by an imaging device attached to the shovel to an instrument panel screen image, and when the operator's head pitches up from that state, the displayed image returns to the remote video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the system described in the above Patent Document 1, a remote video or an instrument panel image is selectively displayed on the head-mounted video display device. That is, in this system, no instrument information of the shovel is provided to the operator who is checking the movement of the arm with the remote video. For this reason, the operator has to perform the remote operation relying on less information than the information obtained when performing the boarding operation.

[0006] The present invention has been made in view of the above circumstances, and an object thereof is to provide a remote operation system capable of providing useful and abundant information to an operator who is performing a remote operation while viewing a remote video.

Means for Solving the Problems

[0007] In order to solve the above problems, a remote operation system according to the present invention includes an imaging device provided on a forklift located at a place away from the operator, a head-mounted video display device worn on the head of the operator, and a command reception device that is within reach of the operator's hand and receives a remote operation command from the operator. A composite remote video generation device that acquires the content of the remote operation command received by the command reception device, the operating state of the forklift that operates according to the remote operation command, and the remote video captured by the imaging device, and generates a composite remote video by synthesizing a command display image indicating the content of the remote operation command and an operation display image indicating the operating state of the forklift on the remote video. The head-mounted video display device has a configuration in which the composite remote video generated by the composite remote video generation device is displayed in front of the operator's eyes.

[0008] The head-mounted video display device of the remote operation system may include a posture detection unit that detects the posture of the operator's head. In this case, it is preferable that the composite remote video generation device of the remote operation system acquires the posture detected by the posture detection unit and generates a composite remote video by synthesizing a command display image and an operation display image on a part of the remote video cut out according to the acquired posture.

[0009] The composite remote video generation device of the remote operation system may be configured to be installed on the operator side and acquire the operating state and remote video of the forklift from the forklift via the Internet line, or may be configured to be installed on the forklift and acquire the content of the remote operation command from the command receiving device via the Internet line, or may be configured to be installed at a location away from the operator and the forklift, acquire the operating state and remote video of the forklift from the forklift via the Internet line, and acquire the content of the remote operation command from the command receiving device via the Internet line.

[0010] The command receiving device of the remote operation system may include four buttons arranged in a cross shape corresponding to the raising and lowering of the forks provided on the forklift and the forward and backward tilting of the mast provided on the forklift. In this case, the command display image preferably includes four cargo handling symbols arranged in a cross shape corresponding to the four buttons, and when any one of the four buttons is operated, the appearance of the cargo handling symbol corresponding to the button changes.

[0011] The command receiving device of the remote operation system may include an analog stick for commanding the traveling direction and traveling speed of the forklift. In this case, the command display image preferably includes a vehicle symbol with the forklift as a motif and a traveling symbol whose relative position from the vehicle symbol changes according to the tilting direction and tilting amount of the analog stick.

Advantages of the Invention

[0012] According to the present invention, it is possible to provide a remote operation system capable of providing useful and abundant information to an operator who is performing remote operation while viewing a remote video.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of a remote operation system according to the present invention will be described with reference to the accompanying drawings.

[0015] [First Embodiment] FIG. 1 shows a remote operation system 1A according to a first embodiment of the present invention. The remote operation system 1A is for an operator O to remotely operate a forklift F at a distant location. As shown in the figure, it includes a head-mounted video display device 10, a command reception device 11, a first arithmetic unit 12A, and a first communication device 13 on the operator O side, and an imaging device 20, a second arithmetic unit 21A, and a second communication device 22 on the forklift F side.

[0016] The head-mounted video display device 10 is what is called a head-mounted display (HMD). The head-mounted video display device 10 is mounted on the head of the operator O and displays a composite remote video described later in front of the operator O.

[0017] The head-mounted video display device 10 has a posture detection unit (not shown). The posture detection unit detects the posture of the head of the operator O and outputs a signal corresponding to the posture (hereinafter referred to as the "posture signal"). The posture signal includes a signal indicating how much the head of the operator O is tilted in the left-right direction and a signal indicating how much the head of the operator O is tilted in the up-down direction.

[0018] The command reception device 11 is something like a controller of a game console. As shown in FIG. 2, it has four buttons 30, 31, 32, 33 arranged in a cross shape and an analog stick 34. The command reception device 11 outputs a signal indicating whether each of the buttons 30, 31, 32, 33 is pressed (hereinafter referred to as the "button operation signal") and a signal indicating the tilting direction and tilting amount of the analog stick 34 (hereinafter referred to as the "stick operation signal"). The buttons 30, 31, 32, 33 are operated by the left thumb of the operator O. Also, the analog stick 34 is operated by the right thumb of the operator O.

[0019] The first arithmetic device 12A is something like a personal computer and is connected to the head-mounted video display device 10 and the command reception device 11 by wire via an external device connection interface such as USB. In this embodiment, this first arithmetic device 12A corresponds to the "synthetic remote video generation device".

[0020] The first communication device 13 is something like a router and is connected to the first arithmetic device 12A by wire via a communication interface such as LAN. The first communication device 13 is also connected to the Internet line NT.

[0021] The imaging device 20 is a 360° camera attached to a remotely operated forklift F. The forklift F is provided with a pole 25 having a length of less than 1 m instead of a seat on which an operator sits, and the imaging device 20 is attached to the upper end of this pole 25. That is, the imaging device 20 is attached at a position where the head of the operator would be located if the operator were sitting on the seat. The imaging device 20 captures remote images for 360° and outputs a signal regarding the remote images (hereinafter referred to as a "remote image signal").

[0022] The second arithmetic device 21A is a vehicle control unit ECU capable of controlling a cargo handling device 23 (which includes a mast, forks, and a hydraulic system for driving them) and a traveling device 24 (which includes a steering device for changing the direction of the steering wheels and a motor for driving the drive wheels). The second arithmetic device 21A also has a function of grasping the state of each part of the forklift F and a function of determining whether an abnormality has occurred in each part of the forklift F. The second arithmetic device 21A generates a signal regarding the state of each part of the forklift F that it has grasped and the presence or absence of an abnormality (hereinafter referred to as an "operation state signal").

[0023] In this embodiment, the second arithmetic device 21A can always grasp at least the weight of the load on the forks, the tilt angle of the mast, the lift height of the forks, the rotation speed of the motor (the traveling speed of the forklift F), and the remaining battery level.

[0024] The second communication device 22 is something like a router installed in the facility where the forklift F performs work, and is wirelessly connected to the second arithmetic device 21A via a communication interface. The second communication device 22 is also connected to the Internet line NT.

[0025] When the operator O presses any one of the buttons 30, 31, 32, 33 of the command receiving device 11, the button operation signal corresponding to the button (for example, the first button 30) is sent to the second arithmetic unit 21A through the path of "command receiving device 11 → first arithmetic unit 12A → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21A". The button operation signal may be converted into an appropriate format midway.

[0026] The second arithmetic unit 21A operates the handling device 23 according to the button operation signal sent from the command receiving device 11. Specifically, the second arithmetic unit 21A raises the fork according to the button operation signal corresponding to the first button 30, lowers the fork according to the button operation signal corresponding to the second button 31, tilts the mast backward according to the button operation signal corresponding to the third button 32, and tilts the mast forward according to the button operation signal corresponding to the fourth button 33.

[0027] When the operator O tilts the analog stick 34 of the command receiving device 11 in any direction, the stick operation signal corresponding to the tilting direction and tilting amount is sent to the second arithmetic unit 21A through the same path as the button operation signal, that is, the path of "command receiving device 11 → first arithmetic unit 12A → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21A". The stick operation signal may be converted into an appropriate format midway.

[0028] The second arithmetic unit 21A operates the traveling device 24 according to the stick operation signal sent from the command receiving device 11. Specifically, the second arithmetic unit 21A operates the steering device so that the traveling direction of the forklift F coincides with the direction corresponding to the tilting direction indicated by the stick operation signal, and operates the motor so that the traveling speed of the forklift F coincides with the speed corresponding to the tilting amount indicated by the stick operation signal.

[0029] The remote video signal related to the 360° remote video output by the imaging device 20 is sent to the first arithmetic device 12A through the path of "imaging device 20 → second arithmetic device 21A → second communication device 22 → Internet line NT → first communication device 13 → first arithmetic device 12A". The remote video signal may be converted into an appropriate format midway.

[0030] The operation state signal regarding the state of each part of the forklift F grasped by the second arithmetic device 21A and the presence or absence of occurrence of an abnormality is sent to the first arithmetic device 12A through the path of "second arithmetic device 21A → second communication device 22 → Internet line NT → first communication device 13 → first arithmetic device 12A".

[0031] The posture signal output by the head-mounted video display device 10 is sent to the first arithmetic device 12A through the path of "head-mounted video display device 10 → first arithmetic device 12A".

[0032] In this way, the first arithmetic device 12A as a composite remote video generation device acquires the button operation signal and the stick operation signal output by the command reception device 11, the posture signal output by the head-mounted video display device 10, the remote video signal output by the imaging device 20, and the operation state signal output by the second arithmetic device 21A.

[0033] The first arithmetic device 12A extracts a part of the remote video based on the acquired posture signal. In other words, as shown in FIG. 3, the first arithmetic device 12A cuts out an image A of a size corresponding to the field of view of the operator O from the 360° remote video according to the posture of the operator O. If the posture of the head of the operator O changes, the position of the cut-out image A also changes. On the other hand, even if the posture of the head of the operator O changes, the size of the cut-out image A does not change.

[0034] The first arithmetic device 12A generates a command display image based on the acquired button operation signal and stick operation signal.

[0035] As shown in FIG. 4, the command display image includes four handling symbols 40, 41, 42, 43 arranged in a cross shape, corresponding to the buttons 30, 31, 32, 33 of the command receiving device 11. The first handling symbol 40 corresponds to the first button 30 for raising the fork, and its appearance (for example, color; the same applies hereinafter) changes depending on whether the first button 30 is pressed or not. The second handling symbol 41 corresponds to the second button 31 for lowering the fork, and its appearance changes depending on whether the second button 31 is pressed or not. The third handling symbol 42 corresponds to the third button 32 for tilting the mast backward, and its appearance changes depending on whether the third button 32 is pressed or not. Also, the fourth handling symbol 43 corresponds to the fourth button 33 for tilting the mast forward, and its appearance changes depending on whether the fourth button 33 is pressed or not.

[0036] As shown in FIG. 4, the command display image further includes a vehicle symbol 44 with a forklift F as a motif and a traveling symbol 45 located in the vicinity thereof. The relative position of the traveling symbol 45 with respect to the vehicle symbol 44 changes according to the tilting direction and tilting amount of the analog stick 34.

[0037] The arrangement of the handling symbols 40, 41, 42, 43, the vehicle symbol 44, and the traveling symbol 45 in the command display image generally corresponds to the arrangement of the buttons 30, 31, 32, 33 and the analog stick 34 in the command receiving device 11. That is, the symbols 40, 41, 42, 43 corresponding to the buttons 30, 31, 32, 33 operated by the left thumb are arranged on the left side of the command display image, and the symbols 44, 45 corresponding to the analog stick 34 operated by the right thumb are arranged on the right side of the command display image.

[0038] It can be said that the command display image is an image showing the content of the remote operation command received by the command receiving device 11, in other words, the content of the remote operation command input by the operator O.

[0039] The first arithmetic unit 12A generates an operation display image based on the acquired operation state signal.

[0040] As shown in FIG. 4, the operation display image includes a load meter 50 indicating the weight of the load on the fork, a tilt meter 51 indicating the tilt angle of the mast, a lift meter 52 indicating the lift height of the fork, a speed meter 53 indicating the traveling speed of the forklift F, a battery meter 54 indicating the amount of power remaining in the battery, and an awameter 56 indicating the current time or the operating integrated time. In addition, the operation display image also includes a plurality of warning symbols 55 that change in appearance such as lighting up when an abnormality occurs.

[0041] As shown in FIG. 4, the first arithmetic unit 12A generates a composite remote video by synthesizing a command display image and an operation display image with a part (image A) of the remote video. Then, the first arithmetic unit 12A sends a signal related to the generated composite remote video (hereinafter referred to as "composite remote video signal") toward the head-mounted video display device 10.

[0042] The head-mounted video display device 10 displays the composite remote video in front of the operator O based on the sent composite remote video signal.

[0043] Thus, in the remote operation system 1A according to the first embodiment, the command display image and the operation display image are presented to the operator O who is performing remote operation while viewing the remote video. Therefore, the operator O can confirm that the operation command has been correctly received by the command display image or grasp the state of the forklift F that cannot be grasped from the remote video by the operation display image without interrupting the remote operation.

[0044] [Second Embodiment] FIG. 5 shows a remote control system 1B according to the second embodiment of the present invention. The remote control system 1B includes a head-mounted video display device 10, a command receiving device 11, a first arithmetic unit 12B, and a first communication device 13 on the operator O side, and an imaging device 20, a second arithmetic unit 21B, and a second communication device 22 on the forklift F side. The remote control system 1B is different from the remote control system 1A according to the first embodiment in that it includes a first arithmetic unit 12B instead of the first arithmetic unit 12A and a second arithmetic unit 21B instead of the second arithmetic unit 21A, but is common with the remote control system 1A in other respects.

[0045] Similar to the first arithmetic unit 12A, the first arithmetic unit 12B is wired-connected to the head-mounted video display device 10 and the command receiving device 11 via an external device connection interface such as USB.

[0046] Similar to the second arithmetic unit 21A, the second arithmetic unit 21B is a vehicle control unit ECU capable of controlling the cargo handling device 23 and the traveling device 24, and has a function of grasping the state of each part of the forklift F and a function of determining whether an abnormality has occurred in each part of the forklift F. Further, similar to the second arithmetic unit 21A, the second arithmetic unit 21B generates an operation state signal regarding the state of each part of the forklift F grasped and the presence or absence of an abnormality. In this embodiment, this second arithmetic unit 21B corresponds to the "synthetic remote video generation device".

[0047] When the operator O presses any one of the buttons 30, 31, 32, 33 of the command receiving device 11, a button operation signal corresponding to the button (for example, the first button 30) is sent to the second arithmetic unit 21B through the path of "command receiving device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21B". The button operation signal may be converted into an appropriate format on the way.

[0048] Similar to the second arithmetic unit 21A, the second arithmetic unit 21B operates the cargo handling device 23 according to the button operation signal sent from the command receiving device 11.

[0049] When the operator O tilts the analog stick 34 of the command receiving device 11 in any direction, a stick operation signal corresponding to the tilting direction and tilting amount is sent to the second arithmetic unit 21B through the same path as the button operation signal, that is, the path of "command receiving device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21B". The stick operation signal may be converted into an appropriate format midway.

[0050] Similar to the second arithmetic unit 21A, the second arithmetic unit 21B operates the traveling device 24 according to the stick operation signal sent from the command receiving device 11.

[0051] The remote video signal regarding the 360° remote video output by the imaging device 20 is sent to the second arithmetic unit 21B through the path of "imaging device 20 → second arithmetic unit 21B".

[0052] The attitude signal output by the head-mounted video display device 10 is sent to the second arithmetic unit 21B through the path of "head-mounted video display device 10 → first arithmetic unit 12B → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21B". The attitude signal may be converted into an appropriate format midway.

[0053] In this way, the second arithmetic unit 21B as a composite remote video generation device acquires the button operation signal and stick operation signal output by the command receiving device 11, the attitude signal output by the head-mounted video display device 10, the remote video signal output by the imaging device 20, and the operation state signal generated by the second arithmetic unit 21B itself.

[0054] Similar to the first arithmetic unit 12A, the second arithmetic unit 21B extracts a part of the remote video based on the acquired attitude signal (see Fig. 3).

[0055] Similar to the first arithmetic unit 12A, the second arithmetic unit 21B generates a command display image and an operation display image based on the acquired button operation signal, stick operation signal, and operation state signal.

[0056] Similar to the first arithmetic unit 12A, the second arithmetic unit 21B generates a composite remote video by synthesizing a command display image and an operation display image with a part of the remote video (image A) (see Fig. 4). Then, the second arithmetic unit 21B sends a composite remote video signal related to the generated composite remote video to the head-mounted video display device 10 through the path of "second arithmetic unit 21B → second communication device 22 → Internet line NT → first communication device 13 → first arithmetic unit 12B → head-mounted video display device 10".

[0057] The head-mounted video display device 10 displays the composite remote video in front of the operator O based on the sent composite remote video signal.

[0058] In this way, also in the remote operation system 1B according to the second embodiment, the command display image and the operation display image are presented to the operator O who is performing the remote operation while viewing the remote video. Therefore, the operator O can confirm that the operation command has been correctly received by the command display image or grasp the state of the forklift F that cannot be grasped from the remote video by the operation display image without interrupting the remote operation.

[0059] [Third Embodiment] Fig. 6 shows a remote control system 1C according to the third embodiment of the present invention. The remote control system 1C includes a head-mounted video display device 10, a command receiving device 11, a first arithmetic unit 12B, and a first communication device 13 on the operator O side, an imaging device 20, a second arithmetic unit 21A, and a second communication device 22 on the forklift F side, and a third arithmetic unit 60. The remote control system 1C is different from the remote control system 1A according to the first embodiment in that it includes the first arithmetic unit 12B instead of the first arithmetic unit 12A and further includes the third arithmetic unit 60, but is common with the remote control system 1A in other respects. The remote control system 1C is different from the remote control system 1B according to the second embodiment in that it includes the second arithmetic unit 21A instead of the second arithmetic unit 21B and further includes the third arithmetic unit 60, but can also be said to be common with the remote control system 1B in other respects.

[0060] The third arithmetic unit 60 is a computer that provides a cloud service set at a location away from both the operator O and the forklift F and is connected to the Internet line NT. In this embodiment, this third arithmetic unit 60 corresponds to the "synthetic remote video generation device".

[0061] When the operator O presses any one of the buttons 30, 31, 32, 33 of the command receiving device 11, the button operation signal corresponding to the button (for example, the first button 30) is sent to the second arithmetic unit 21A through the path of "command receiving device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21A" and is also sent to the third arithmetic unit 60 through the path of "command receiving device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → third arithmetic unit 60". The button operation signal may be converted into an appropriate format on the way.

[0062] When the operator O tilts the analog stick 34 of the command reception device 11 in any direction, a stick operation signal corresponding to the tilting direction and tilting amount is sent to the second arithmetic unit 21A through the same path as the button operation signal, that is, the path of "command reception device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21A", and is also sent to the third arithmetic unit 60 through the path of "command reception device 11 → first arithmetic unit 12B → first communication device 13 → Internet line NT → third arithmetic unit 60". The stick operation signal may be converted into an appropriate format midway.

[0063] The remote video signal regarding the 360° remote video output by the imaging device 20 is sent to the third arithmetic unit 60 through the path of "imaging device 20 → second arithmetic unit 21A → second communication device 22 → Internet line NT → third arithmetic unit 60". The remote video signal may be converted into an appropriate format midway.

[0064] The operation state signal regarding the state of each part of the forklift F grasped by the second arithmetic unit 21A and the presence or absence of occurrence of an abnormality is sent to the third arithmetic unit 60 through the path of "second arithmetic unit 21A → second communication device 22 → Internet line NT → third arithmetic unit 60".

[0065] The posture signal output by the head-mounted video display device 10 is sent to the third arithmetic unit 60 through the path of "head-mounted video display device 10 → first arithmetic unit 12B → first communication device 13 → Internet line NT → third arithmetic unit 60". The posture signal may be converted into an appropriate format midway.

[0066] In this way, the third arithmetic unit 60 as a composite remote video generation device acquires the button operation signal and stick operation signal output by the command reception device 11, the posture signal output by the head-mounted video display device 10, the remote video signal output by the imaging device 20, and the operation state signal output by the second arithmetic unit 21A.

[0067] Similar to the first arithmetic unit 12A and the second arithmetic unit 21B, the third arithmetic unit 60 extracts a part of the remote video based on the acquired posture signal (see FIG. 3).

[0068] Similar to the first arithmetic unit 12A and the second arithmetic unit 21B, the third arithmetic unit 60 generates a command display image and an operation display image based on the acquired button operation signal, stick operation signal, and operation state signal.

[0069] Similar to the first arithmetic unit 12A and the second arithmetic unit 21B, the third arithmetic unit 60 generates a composite remote video by synthesizing the command display image and the operation display image with a part of the remote video (image A) (see FIG. 4). Then, the third arithmetic unit 60 sends a composite remote video signal related to the generated composite remote video to the head-mounted video display device 10 through the path of "third arithmetic unit 60 → Internet line NT → first communication device 13 → first arithmetic unit 12B → head-mounted video display device 10".

[0070] The head-mounted video display device 10 displays the composite remote video in front of the operator O based on the sent composite remote video signal.

[0071] In this way, also in the remote operation system 1C according to the third embodiment, the command display image and the operation display image are presented to the operator O who is performing the remote operation while viewing the remote video. Therefore, the operator O can confirm that the operation command has been correctly received by the command display image or grasp the state of the forklift F that cannot be grasped from the remote video by the operation display image without interrupting the remote operation.

[0072] [Modification Example] The first embodiment, the second embodiment, and the third embodiment of the remote operation system according to the present invention have been described above, but the configuration of the present invention is not limited to these.

[0073] For example, the head-mounted video display device 10 may be wirelessly connected to the first arithmetic units 12A and 12B or the first communication device 13. When the head-mounted video display device 10 is wirelessly connected to the first communication device 13, the attitude signal is sent to the first arithmetic unit 12A through the path of "head-mounted video display device 10 → first communication device 13 → first arithmetic unit 12A", or sent to the second arithmetic unit 21B through the path of "head-mounted video display device 10 → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21B", or sent to the third arithmetic unit 60 through the path of "head-mounted video display device 10 → first communication device 13 → Internet line NT → third arithmetic unit 60". Also, in this case, the composite remote video signal is sent to the head-mounted video display device 10 through the path of "first arithmetic unit 12A → first communication device 13 → head-mounted video display device 10", the path of "second arithmetic unit 21B → second communication device 22 → Internet line NT → first communication device 13 → head-mounted video display device 10", or the path of "third arithmetic unit 60 → Internet line NT → first communication device 13 → head-mounted video display device 10".

[0074] Also, the command receiving device 11 may be wirelessly connected to the first arithmetic units 12A and 12B or the first communication device 13. When the command receiving device 11 is wirelessly connected to the first communication device 13, the button operation signal and the stick operation signal are sent to the first arithmetic unit 12A through the path of "command receiving device 11 → first communication device 13 → first arithmetic unit 12A", or sent to the second arithmetic unit 21B through the path of "command receiving device 11 → first communication device 13 → Internet line NT → second communication device 22 → second arithmetic unit 21B", or sent to the third arithmetic unit 60 through the path of "command receiving device 11 → first communication device 13 → Internet line NT → third arithmetic unit 60".

[0075] Also, the first arithmetic units 12A and 12B may be wirelessly connected to the first communication device 13.

[0076] In addition, the command reception device 11 is not limited to something like a controller of a game console held with both hands. In the present invention, any device that is within reach of the operator O's hand and can receive a remote operation command from the operator O can be used as the command reception device 11.

[0077] In addition, the imaging device 20 is not limited to a 360° camera. Examples of alternatives to the 360° camera include combinations of a plurality of cameras facing different directions. In this case, the composite remote video generation devices 12A, 21B, 60 may select which remote video captured by which camera among the plurality of cameras according to the posture signal, and may composite the command display image and the operation display image on the selected remote video.

[0078] In addition, the imaging device 20 may be attached to the forklift F by means other than the pole 25. For example, the imaging device 20 may be suspended from the head guard.

[0079] In addition, the second arithmetic units 21A, 21B may be units separate from the vehicle control unit ECU that is responsible for controlling the cargo handling device 23 and the traveling device 24. However, in this case, the two units must be able to communicate signals with each other.

[0080] In addition, the command display image and the operation display image shown in FIG. 4 are merely examples. The layout of these images and the information included in these images may be changed as appropriate. Further, it is preferable that the layout of these images and the information included in these images can be customized by the operator O.

[0081] In addition, the remote operation system according to the present invention may be used to remotely operate an engine-type forklift.

Explanation of Reference Numerals

[0082] 1A, 1B, 1C Remote operation system 10 Head-mounted video display device 11 Instruction receiving device 12A, 12B First arithmetic unit 13 First communication device 20 Imaging device 21A, 21B Second arithmetic unit 22 Second communication device 23 Handling device 24 Traveling device 25 Pole 30 First handling symbol 31 Second handling symbol 32 Third handling symbol 33 Fourth handling symbol 34 Analog stick 40 First handling symbol 41 Second handling symbol 42 Third handling symbol 43 Fourth handling symbol 44 Vehicle symbol 45 Traveling symbol 50 Load meter 51 Tilt meter 52 Lift meter 53 Speed meter 54 Battery meter 55 Warning symbol 56 Slope meter 60 Third arithmetic unit F Forklift NT Internet line O Operator

Claims

1. An imaging device provided on the forklift at a location remote from the operator; A head-mounted image display device that is mounted on the head of the operator; a command receiving device that is within reach of the operator and receives remote operation commands from the operator; a synthetic remote image generating device that receives the content of the remote operation command received by the command receiving device, the operation state of the forklift operating in accordance with the remote operation command, and a remote image captured by the imaging device, and generates a synthetic remote image by synthesizing a command display image showing the content of the remote operation command and an operation display image showing the operation state of the forklift with the remote image; Equipped with the head-mounted image display device displays the synthetic remote image generated by the synthetic remote image generation device in front of the operator's eyes; the command receiving device includes four buttons arranged in a cross shape corresponding to raising and lowering a fork provided on the forklift and tilting forward and backward a mast provided on the forklift, the command display image includes four loading / unloading symbols arranged in a cross shape corresponding to the four buttons; When any of the four buttons is operated, the appearance of the loading symbol corresponding to that button changes. A remote control system comprising:

2. An imaging device provided on the forklift at a location remote from the operator; A head-mounted image display device that is mounted on the head of the operator; a command receiving device that is within reach of the operator and receives remote operation commands from the operator; a synthetic remote image generating device that receives the content of the remote operation command received by the command receiving device, the operation state of the forklift operating in accordance with the remote operation command, and a remote image captured by the imaging device, and generates a synthetic remote image by synthesizing a command display image showing the content of the remote operation command and an operation display image showing the operation state of the forklift with the remote image; Equipped with the head-mounted image display device displays the synthetic remote image generated by the synthetic remote image generation device in front of the operator's eyes; the command receiving device includes an analog stick for commanding a travel direction and a travel speed of the forklift, The command display image includes a vehicle symbol having a forklift motif, and a driving symbol whose relative position from the vehicle symbol changes depending on the tilt direction and tilt amount of the analog stick. A remote control system comprising:

3. An imaging device provided on the forklift at a location remote from the operator; A head-mounted image display device that is mounted on the head of the operator; a command receiving device that is within reach of the operator and receives remote operation commands from the operator; a synthetic remote image generating device that receives the content of the remote operation command received by the command receiving device, the operation state of the forklift operating in accordance with the remote operation command, and a remote image captured by the imaging device, and generates a synthetic remote image by synthesizing a command display image showing the content of the remote operation command and an operation display image showing the operation state of the forklift with the remote image; Equipped with the head-mounted image display device displays the synthetic remote image generated by the synthetic remote image generation device in front of the operator's eyes; The operation state of the forklift shown by the operation display image includes the lift height of the forks constituting the forklift. A remote control system comprising:

4. the head-mounted image display device includes a posture detection unit that detects a posture of the head of the operator, The synthetic remote image generating device acquires the posture detected by the posture detection unit, and generates the synthetic remote image by synthesizing the command display image and the action display image with a part of the remote image cut out according to the posture.

4. The remote control system according to claim 1, wherein the remote control system is a remote control system for controlling a remote control device.

5. The synthetic remote image generating device is installed on the operator's side and acquires the operation state of the forklift and the remote image from the forklift via an Internet line.

4. The remote control system according to claim 1, wherein the remote control system is a remote control system for controlling a remote control device.

6. The synthetic remote image generating device is installed on the forklift and receives the contents of the remote operation command from the command receiving device via an Internet line.

4. The remote control system according to claim 1, wherein the remote control system is a remote control system for controlling a remote control device.

7. The synthetic remote image generating device is installed at a location remote from the operator and the forklift, and receives the operation state of the forklift and the remote image from the forklift via an Internet line, and also receives the contents of the remote operation command from the command receiving device via the Internet line.

4. The remote control system according to claim 1, wherein the remote control system is a remote control system for controlling a remote control device.

Images