Remote control system for heavy machinery using a head-mounted display

Abstract

To provide a remote control system for a jumbo drill, which allows a user to perform work from a remote location by operating a working part with a sense of tactile response and reality, and which provides a sense of perspective and distance similar to that of operating a heavy machine seated in an operating room. [Configuration] A remote control system configured to remotely control heavy machinery using a remote control device to perform work, an imaging device that captures images of the working part and the work location of the heavy equipment; an imaging device for capturing an image of the control panel and control lever of the remote control device; a head-mounted display that displays real-time images of the working unit and working location and real-time images of the control panel and operating levers; The present invention is characterized by the above-mentioned configuration.

Description

[Technical Field]

[0001] The present invention relates to a remote control system for heavy machinery, and more particularly to a remote control system for remotely controlling heavy machinery to perform work based on images displayed on a head-mounted display. [Background technology]

[0002] In recent years, when heavy machinery is used for work such as cutting the face of a tunnel using a drill jumbo, the operator does not sit in an operation room installed in the drill jumbo and operate it, but rather operates it remotely from a remote operation room installed at a location away from the face of the tunnel (see, for example, Patent Document 1).

[0003] With remote control, even if a collapse occurs at the face, the operator is safe because he or she is located away from the face. Furthermore, even if there is no risk of a collapse, the operator is located away from the cutting work site, where vibrations, noise, dust, flying stones, etc. caused by cutting by the drill jumbo are constantly generated, so there is an advantage that the operator does not have to be in a bad working environment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-163878 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent document 1 describes a technology in which images captured by an imaging device installed on construction machinery such as a drill jumbo are displayed on a monitor screen in a remote control room, and the drill jumbo is remotely controlled while viewing the images displayed on the monitor screen.

[0006] However, remote control while viewing the image displayed on the monitor requires not only the operation of the drill's control lever, but also the orientation of the imaging device and the zooming in and out of the display, making the operation cumbersome. Furthermore, the image displayed on the monitor screen makes it difficult to grasp the sense of perspective and distance, and the sense of tactile response and realism in operating the drill is often lacking, making it difficult to say that the operation is easy.

[0007] Therefore, the object of the present invention is to provide a heavy machinery remote control system using a head-mounted display that allows a user to perform work in tunnels, etc., from a remote location by operating the work unit with a sense of tactile response and realism, while obtaining a sense of perspective and distance similar to that obtained when operating the heavy machinery while seated in the control room. [Means for solving the problem]

[0008] The present invention, which solves the above problems, has the following configuration.

[0009] 1. A remote control system configured to remotely control heavy machinery using a remote control device to perform work, an imaging device that captures images of the working part and the work location of the heavy equipment; an imaging device for capturing an image of the control panel and control lever of the remote control device; a head-mounted display that displays real-time images of the working unit and working location and real-time images of the control panel and operating levers; A heavy machinery remote control system using a head-mounted display, characterized by having a configuration including:

[0010] 2. A remote control system for heavy machinery using a head-mounted display as described in 1 above, characterized in that it is configured to display real-time images of the work unit and work location and real-time images of the control panel and operating levers in a single field of view.

[0011] 3. The imaging device that captures the work unit and work area is configured to capture images by following the operator's line of sight using a head tracking function, and display the captured images on a head-mounted display. The image of the control panel and the control lever is displayed on the head-mounted display as a pass-through image, with the image fixed in a specific area. 3. A remote control system for heavy machinery using the head-mounted display described in 1 or 2 above.

[0012] 4. A remote control system for heavy machinery using a head-mounted display as described in 1 or 2 above, characterized in that the working unit and work location are photographed using a wide-angle camera.

[0013] 5. A heavy machinery remote control system using a head-mounted display as described in 1 or 2 above, characterized in that the working unit and work location are photographed using a PTZ camera in a zoomable manner.

[0014] 6. A heavy machinery remote control system using a head-mounted display as described in 1 or 2 above, characterized in that the heavy machinery and the remote control device are connected via wired or wireless communication. [Effects of the Invention]

[0015] According to the invention shown in claim 1, it is possible to provide a heavy machinery remote control system using a head-mounted display that can provide a sense of perspective and distance similar to that of operating the heavy machinery while seated in the control room, and allows work to be performed remotely from a remote location by operating the working unit with a sense of tactile response and realism.

[0016] In particular, by displaying real-time images of the heavy equipment's working unit and work location, as well as real-time images of the control panel and operating levers on a head-mounted display, it is possible to perform work while checking the movement of the operating levers and the work status of the working unit. In addition, the images of the work unit and work location are real-time images, and are displayed on a head-mounted display, so that the operator can move their head in the direction they want to see and align their gaze, allowing them to work in a state that is similar to that of actually sitting in the control room of the heavy equipment and operating it.

[0017] According to the invention set forth in claim 2, a real-time image of the working unit and the work location and a real-time image of the control panel and the control lever are compositely displayed in one field of view, so that the operator can work while simultaneously viewing both the situation of operating the control lever and the situation of the working unit operating by the control lever. Therefore, the operator can remotely operate the machine with the same feeling as if he or she were sitting in the control room of the heavy machinery working at the actual site, for example, in the work location inside a tunnel.

[0018] According to the invention described in claim 3, even if the operator is seated in the control room of the heavy machinery and moves his / her eyes in various directions to view the image, the control panel and control levers are always fixed in specific positions within the field of view, so that the operator can always check the status of the operation, control panel and control levers.

[0019] According to the invention described in claim 4, the work area and work section can be photographed over a wide area, and the wide area of ​​the work area can be confirmed from the video. Therefore, even when remotely operating from a position far from the work area, work can be performed without blind spots or oversights.

[0020] According to the invention as set forth in claim 5, the zoom display by the PTZ camera allows for enlarged confirmation of areas that require detailed checking, such as the state of the work area.

[0021] According to the invention set forth in claim 6, wired communication allows for stable, high-speed communication, making it possible to display real-time images without stress, even when displaying high-resolution images, and allows for stable remote operation. Furthermore, wireless communication allows for communication without the need to lay cables at civil engineering sites, which can easily become complicated, making it easy to use. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram showing an embodiment of a remote control system for a heavy machine (drill jumbo) according to the present invention. [Figure 2]A diagram showing an example of an image displayed on a head-mounted display. [Figure 3] A diagram showing an example of real-time video of the working part (drill) and the working location (cutting location). [Figure 4] A diagram showing an example of real-time images of a control panel and control levers. [Figure 5] A diagram showing an example of real-time video display that follows the operator's line of sight using the head tracking function. [Figure 6] FIG. 10 is a diagram showing an example of a configuration for specifying a zoom target area for a working part (drill) and a working location (cutting location); [Figure 7] FIG. 7 is a diagram showing a display example of the zoom target area shown in FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0023] A heavy machinery remote control system using a head-mounted display according to the present invention (hereinafter, sometimes simply referred to as a remote control system) will be described.

[0024] The remote control system of the present invention is a remote control system configured to remotely control heavy machinery using a remote control device to perform work in a tunnel, an imaging device that photographs the working part of the heavy equipment and the working location inside a tunnel, etc.; an imaging device for capturing an image of the control panel and control lever of the remote control device; a head-mounted display that displays real-time images of the working unit and working location and real-time images of the control panel and operating levers; The present invention is characterized by a configuration having the following features.

[0025] The present invention will be described below based on the embodiments shown in FIGS. In this embodiment, a jumbo drill is used as the heavy machinery. The working part in this invention is the drill of the jumbo drill, the working location is the face, and the work is cutting the face. In this embodiment, work inside a tunnel is described, but the present invention is not limited to this content. As shown in FIG. 1, in this system, a drill jumbo 1 for drilling a face S in a tunnel T is remotely operated by a remote control device 2 from a safe location (preferably outside the tunnel T) away from the face S. An imaging device 3 that photographs the drill 11 of the drill jumbo 1 and the cutting part of the face S; an imaging device 4 for photographing the operation panel 21 and the operation lever 22 of the remote control device 2; a head-mounted display 5 that displays real-time images of the drill 11 and the cutting location, and real-time images of the operation panel 21 and the operation lever 22; The main configuration is as follows.

[0026] First, the remote control device 2 will be described. An example of the configuration of the remote control device 2 mainly comprises an operation lever 22 arranged in an upright position from an operation panel 21, and a display device 23 that displays various information about the drill jumbo 1 (various information such as the drill rotation speed and hydraulic pressure value), and is installed outside the tunnel T away from the working face S. There are no limitations on the specific configuration of the remote control device 2, and it can have a configuration similar to that of a publicly known and publicly used remote control system that remotely controls various civil engineering and construction machines by operating the operation lever 22.

[0027] The remote control device 2 is connected to a control unit 14 that controls the imaging device 3 and various operations of the drill jumbo 1 via a relay box 13 disposed in the drill jumbo 1 via a personal computer 6 and a hub 7 . An imaging device 4 for capturing images of the operation panel 21 and operation lever 22 of the remote control device 2, and a head-mounted display 5 are connected to a personal computer 6 that controls remote control by the remote control device 2.

[0028] Furthermore, it is preferable that a monitor device 8 is connected to the personal computer 6. The monitor device 8 displays the same image as that displayed on the head-mounted display 5. With this configuration, other workers can also see and confirm the same image that the operator performing remote operation sees via the head-mounted display 5.

[0029] Next, the image displayed on the head-mounted display 5 will be described. Of the images displayed in Fig. 2, real-time images of the inside of tunnel T, such as the four drills 11·11·11·11 of the drill jumbo 1 and the working face S, are captured by the imaging device 3 mounted on the drill jumbo 1, and are shown in Fig. 3. The imaging device 3 is preferably installed near the operation room 12 of the drill jumbo 1 (above the operation room 12 in Fig. 1). In such a location, remote operation of the drills 11 and the working face S can be performed with the same feeling as if the operator were seated in the operation room 12.

[0030] Similarly, among the images displayed in Fig. 2, real-time images of the control panel 21 and the control lever 22 are captured by the imaging device 4, and are shown in Fig. 4. It is preferable that the display device 23, which displays various information about the drill jumbo 1, is also captured by the imaging device 4, along with the control panel 21 and the control lever 22. It is preferable that the imaging device 4 is installed slightly above the position facing the control panel 21 of the remote control device 2, and preferably in a position close to the operator's line of sight (for example, slightly above the operator's head). In such a position, the operator can perform remote operation with the same feeling as if he or she were sitting opposite the control panel 21 and operating the control lever 22.

[0031] By combining the real-time image captured by the imaging device 3 shown in Fig. 3 with the real-time image captured by the imaging device 4 shown in Fig. 4, two types of real-time images are combined in one field of view and displayed on the head-mounted display 5 as shown in Fig. 2. With this configuration, it is possible to perform work while checking the movement of the operating lever 22 and the cutting status of the face S by the drill 11. Therefore, remote operation can be performed with the same feeling as if one were sitting at an actual site, that is, in the operating room 12 of the drill jumbo 1 facing the cutting of the face S.

[0032] The imaging device 3, which captures images of the drill 11 and the cutting portion of the face S, is preferably configured to use a head tracking function to follow the movement of the operator's line of sight to capture images, and to display the captured images on the head-mounted display 5. With this configuration, the operator can move their head in the direction they want to see and align their line of sight, allowing them to perform work in a state similar to that of actually sitting and operating the drill jumbo 1. Figure 5 shows examples of images displayed when the operator moves their line of sight in various directions using the head tracking function (images facing forward, image facing left, image facing right, and image facing up).

[0033] Furthermore, it is preferable that the real-time images of the control panel 21 and the control lever 22 captured by the imaging device 4 are displayed as pass-through images in a fixed position in a specific area within the display area (the pass-through area PT is shown by a virtual line in FIG. 3). With this configuration, the image area of ​​the drill 11 and the working face S displayed on the head-mounted display 5 moves in various directions as the imaging area captured by the imaging device 3 moves in accordance with the movement of the operator's line of sight due to the head tracking function, but the control panel 21 and the control lever 22 remain displayed in a fixed position in a specific area within the field of view, so that the control panel 21 and the control lever 22 can be operated while always being checked.

[0034] The imaging device 3 can be a known camera used in remote control systems for various civil engineering and construction machines, and preferably a wide-angle camera with a capture area of ​​about 180 degrees. With this configuration, the drill 11 and the cutting portion of the face S can be captured over a wide area, making it possible to confirm a wide area of ​​the face S from the image. Therefore, even when remotely controlled from a position far from the face S, cutting work can be performed without blind spots or oversights. When a wide-angle camera is used, the head-mounted display 5 may display the entire captured area, but it is preferable to set it to display a specific area of ​​your choice centered on the drill 11 and the cutting portion of the face S.

[0035] The imaging device 3 that captures images of the drill 11 and the cutting portion of the face S can also be a PTZ camera capable of zooming. A configuration using a PTZ camera allows the state of the cutting portion to be enlarged and checked. For zooming, it is preferable to display an optical zoom image by disposing a PTZ camera separately from the wide-angle camera, rather than using an image cut out by the wide-angle camera described above.

[0036] When performing zoom photography, it is preferable to display a zoom area marker ZM that specifies the area to be zoomed in the image as shown in FIG. 6, move this zoom area marker ZM to the area to be zoomed in by moving the line of sight, and then determine the zoom area so that a virtual display VD of the zoomed area is displayed in the image as shown in FIG. 7. It is preferable that the operation of turning on / off the display of the zoom region marker ZM, turning on / off the determination of the zoom range, adjusting the zoom magnification, etc. be performed by providing a zoom operation switch or the like on the operation panel 21.

[0037] The communication between the relay box 13 of the drill jumbo 1, the hub 7, the personal computer 6, the remote control device 2, etc. is configured to be connected by wired communication or wireless communication. In the embodiment shown in Fig. 1, wired communication is used in all areas, but wireless communication may be used in some areas or wireless communication may be used entirely. In the case of wired communication, the cables used may be publicly known and publicly used cables such as LAN cables and HDMI (registered trademark) cables. In the case of wireless communication, publicly known and publicly used wireless means such as Wi-Fi, Bluetooth (registered trademark), and mobile communication systems can be used. Wired communication allows stable, high-speed communication, making it possible to display real-time images without stress, even when displaying high-definition images, and allows for remote control in a stable state. Furthermore, wireless communication is convenient because it allows communication without the need to run cables around at civil engineering sites, which can easily become complicated. Furthermore, even if the cutting work site and the remote device location are geographically separated, such as when the cutting work site is in Hokkaido and the remote operation location is in Tokyo, remote operation can be performed by using a mobile communication system or the like.

[0038] The above describes the heavy equipment remote control system using a head-mounted display according to the present invention based on an embodiment, but the present invention is not limited to the above embodiment and other aspects can be adopted within the scope of the present invention.

[0039] For example, in the above embodiment, a drill jumbo is shown as a heavy machine for working inside a tunnel, but the present invention can also be applied to a backhoe, a concrete spraying robot, a bulldozer, etc. When applied to a backhoe as heavy machinery, the work location is the ground near the tunnel face, the working part is a shovel (bucket), and the work is loading the excavated soil onto a means of transport (such as a belt conveyor or truck). Furthermore, when applied as a heavy machinery to a concrete spraying robot, the working location is the tunnel face or inner wall surface, the working part is the spray nozzle, and the work is spraying concrete onto the tunnel face or inner wall surface. Furthermore, when applied to a bulldozer as heavy machinery, the working area is the ground inside the tunnel, the working part is the blade (blade), and the work involves removing the excavated earth and sand. [Explanation of symbols]

[0040] 1. Jumbo Drill 11 Drill 12 Control room 13 Relay box 14 Control Unit 2 Remote control device 21 Control panel 22 Operating lever 23 Display device 3. Imaging device 4. Imaging device 5. Head-mounted display 6. Computer 7. Hub 8. Monitoring Device S Face T-Tunnel PT Pass-Through Area ZM Zoom Area Marker VD Virtual Display

Claims

1. In a remote control system configured to remotely control heavy machinery using a remote control device to perform work, an imaging device that captures images of the working part and the work location of the heavy equipment; an imaging device for capturing an image of the control panel and control lever of the remote control device; a head-mounted display that displays real-time images of the working unit and working location and real-time images of the control panel and operating levers; A heavy machinery remote control system using a head-mounted display, characterized by having a configuration including:

2. 2. A remote control system for heavy machinery using a head-mounted display as described in claim 1, characterized in that it is configured to display real-time images of the work unit and work location and real-time images of the control panel and control levers in a single field of view.

3. The imaging device that captures the images of the working unit and the working area is configured to capture images by following the movement of the operator's line of sight using a head tracking function, and display the captured images on a head-mounted display; The image of the control panel and the control lever is displayed on the head-mounted display as a pass-through image, with the image fixed in a specific area.

3. A heavy machinery remote control system using a head-mounted display according to claim 1 or 2.

4. 3. A remote control system for heavy machinery using a head-mounted display according to claim 1 or 2, characterized in that the working unit and the working location are photographed by a wide-angle camera.

5. 3. A remote control system for heavy machinery using a head-mounted display according to claim 1 or 2, characterized in that the working unit and the working location are photographed by a PTZ camera in a zoomable manner.

6. 3. A heavy machinery remote control system using a head-mounted display according to claim 1 or 2, characterized in that the heavy machinery and the remote control device are connected by wired communication or wireless communication.

Images