Remote control system

Abstract

In remote operation, we provide an efficient remote control system that minimizes equipment costs and network load. [Solution] A remote control system for a work machine having a sensor, an information processing device, a work machine, and a remote control terminal comprises a camera image acquisition unit that acquires camera images from a camera which is a sensor, a surrounding environment map creation unit that creates map information of a 2D map or a 3D map based on information acquired by sensors other than the camera, a screen display unit that simultaneously displays the camera images and map information on the remote control screen of the remote control terminal, and an adjustment unit that adjusts the data capacity of the images displayed by the screen display unit to 20 Mbps or less, characterized in that the camera images are images acquired from a camera mounted on the work machine and cameras installed around the work machine, and the total number of camera images is between 2 and 10.

Description

Technical Field

[0001] The present invention relates to a remote operation system for a work machine, and particularly to a system that improves the sense of depth and distance with a minimum of infrastructure and data volume and realizes efficient remote operation.

Background Art

[0002] In conventional remote operation systems, when operating a work machine or a moving body from a remote location, it is common to use camera images as the main information source. However, it is difficult to grasp the accurate sense of distance and depth only with camera images, and there is a problem that the work efficiency decreases, particularly in large-scale warehouses and construction sites.

[0003] To solve this problem, there is a method of using a high-resolution camera and a number of overhead cameras. In addition, technologies for assisting operations in a virtual space have been proposed (for example, Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When remotely operating a work machine or a moving body, there is a problem that the operation speed decreases and the work efficiency significantly decreases because it is difficult to grasp the accurate sense of distance and depth.

[0006] Particularly, in large-scale warehouses and construction sites, it is difficult to grasp the obstacles and the size of the work area, and although quick and accurate operations are required, there is a limit with only one camera image mounted on the work machine.

[0007] Furthermore, construction sites and warehouses often have limited network bandwidth, and in such environments, using high-resolution cameras and multiple overhead cameras raises concerns about increased data volume and resulting communication delays. These delays impair real-time capabilities, making quick operational decisions difficult.

[0008] Therefore, in remote operation, there is a need for an efficient remote operation system that keeps equipment costs down while also minimizing network load. [Means for solving the problem]

[0009] In embodiments of the present invention, the following solutions are provided for the remote operation of work machines.

[0010] The embodiment is a remote control system for a work machine having a sensor, an information processing device, a work machine, and a remote control terminal, the remote control system comprising: a camera image acquisition unit that acquires camera images from a camera which is a sensor; a surrounding environment map creation unit that creates map information for a two-dimensional map or a three-dimensional map based on information acquired by sensors other than cameras; a screen display unit that simultaneously displays the camera images and map information on the remote control screen of the remote control terminal; and an adjustment unit that adjusts the data capacity of the images displayed by the screen display unit to 20 Mbps or less, characterized in that the camera images are images acquired from a camera mounted on the work machine and cameras installed around the work machine, and the total number of camera images is between two and ten.

[0011] Furthermore, the remote control system is characterized in that the data capacity of the video displayed by the screen display unit is 10 Mbps or less.

[0012] Furthermore, the remote control system is characterized by the fact that the surrounding environment map is a two-dimensional map.

[0013] Furthermore, the remote control system is characterized by the ability to arbitrarily change the size or display position of the camera image and map information on the screen display unit.

[0014] Furthermore, the remote control system is equipped with an acceleration sensor, and the surrounding environment map creation unit includes a self-position estimation unit that estimates its own position based on information about the surrounding environment from the sensor and information from the acceleration sensor, generates estimated information, and reflects the estimated information in the map information.

[0015] Furthermore, the remote control system is characterized in that the work machine is a mobile object, and the mobile object is one of the following: automobile, forklift, shovel, crane, dump truck, automated guided vehicle, or unmanned ground vehicle.

[0016] Furthermore, in the remote control system, the video displayed by the screen display unit on the remote control screen of the remote control terminal is video acquired from at least one to three cameras mounted on the work machine or installed around the work machine, and the map information is a single two-dimensional map.

[0017] Furthermore, the remote control system is characterized by the adjustment unit combining camera images and a map and transmitting them as a single screen. [Effects of the Invention]

[0018] As described above, according to the present invention, by simultaneously displaying camera images and map information on the remote control screen, it becomes possible to accurately grasp the sense of depth and distance during remote operation. This improves the operational efficiency of the work machine. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram of the remote control system according to the embodiment. [Figure 2] This is a diagram illustrating the configuration of the remote control system according to the embodiment. [Figure 3]This is a flowchart of the remote operation system according to the embodiment.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as embodiments) will be described in detail with reference to the accompanying drawings. Note that the drawings are merely examples, and the present invention is not limited to those shown in the drawings. In the following drawings, the same elements are denoted by the same numbers or symbols throughout the description of the embodiments.

[0021] [Basic Configuration] FIG. 1 is a diagram for showing the basic configuration of a remote operation system 1 according to an embodiment of the present invention.

[0022] First, the configuration at the work site is shown as follows. The working machine 3 is equipped with at least a camera 2a, a sensor 2x other than the camera, and an information processing device 4, and a camera 2b capable of overlooking the surrounding environment in which the working machine operates may be installed.

[0023] Examples of the camera 2a mounted on the working machine 3 and the camera 2b installed around the working machine include, for example, a 4K camera, an 8K camera, an HDR ( High Dynamic Range) - compatible camera, a high - resolution digital camera, a wide - angle camera, a fisheye camera, an infrared camera, a thermal camera, a depth camera (DGB - D camera), etc. These cameras acquire visual information of the work area in real time and are used to display it on the remote operation screen. Note that the camera includes a video acquisition unit for acquiring video. The video data is connected to the information processing device 4 via the network NW.

[0024] Sensors 2x other than the camera include an acceleration sensor, a 2D LiDAR (Light Detection and Ranging), a 3D LiDAR, a laser tracking sensor, a gyro sensor, a GPS sensor, an ultrasonic sensor, a magnetic sensor, and an infrared sensor.

[0025] In particular, 2D LiDAR is a laser ranging sensor that acquires distance information on a two-dimensional plane with high accuracy, and its effectiveness is more easily demonstrated because it uses less data compared to 3D information.

[0026] An accelerometer is a sensor that detects motion information such as acceleration, deceleration, vibration, and tilt of a work machine. Using an accelerometer, the work machine can perform self-position estimation, motion stabilization, and tilt correction, thereby improving the accuracy of remote operation. Furthermore, self-estimation, motion stabilization, and tilt correction can also be performed based on information from sensors other than the accelerometer.

[0027] The work machine 3 is preferably a mobile device such as an unmanned ground vehicle (UGV), automobile, forklift, shovel, crane, drone, dump truck, or automated guided vehicle (AGV). Furthermore, the work machine 3 can also be applied to other devices such as boring machines, fixed conveyor systems, fixed cranes, agricultural machinery, small mobility devices, autonomous robots, transport robots, and exploration vehicles. Figure 1 shows an unmanned ground vehicle, but it is not limited to this. Devices capable of autonomous driving and automatic operation are also considered from the perspective of compatibility with remote operation.

[0028] The information processing device 4 is connected via a network NW to enable communication, and performs various types of information transmission and reception, as well as various processes related to the remote control system 1. The information processing device 4 may also be a notebook computer, smartphone, computer, tablet, or handheld computer device, or it may be a distributed server system that operates cooperatively by communicating via a network, or it may include an edge server, or it may be a so-called cloud server. These may also be equipped with a processing unit and memory, and examples of processing units include a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and examples of memory include RAM (Random Access Memory) and ROM (Read Only Memory). The information processing device 4 also includes an adjustment unit.

[0029] The information processing device 4 also functions as a media server. The information processing device 4 is installed at multiple work sites and is connected to multiple sensors, such as cameras that observe the remote operation status of remotely controllable work machinery, such as a plain transport vehicle, and to terminals that display the images captured by the observation devices via the information processing device 4, via a network such as an internet line or software virtualization, enabling data communication.

[0030] The network NW may include wireless networks and wired networks, and may include, for example, wireless LANs (WLANs), wide area networks (WANs), ISDNs (integrated service digital networks), wireless LANs, CDMA (code division multiple access), LTE (long term evolution), LTE-Advanced, 4G, 5G, and 6G and later mobile communication systems, or combinations thereof. It also includes LPWA (Low Power Wide Area) and satellite communication systems.

[0031] Next, the configuration diagram for a remote operation location (office, etc.) is shown below. In Figure 1, the remote operation device and remote operation terminal 6 used by the remote operator display images 5a, 5b and a surrounding environment map 5x on the remote operation screen 5. The work machine 3 is then remotely operated by the remote control unit 7.

[0032] Of the images displayed on the remote control screen 5, the main image 5a is displayed larger than the other images, while the sub-image 5b is displayed smaller than the main image. Although there are two sub-images in the diagram, it is preferable to add one or more. Depending on the task, there may be five to ten sub-images as needed. From the standpoint of equipment cost and visibility, it is preferable that the remote control screen 5 is displayed on a single monitor, but each image may be displayed on multiple monitors. Furthermore, it is preferable that the main image and sub-images can be switched at the discretion of the remote operator, and that the size, contrast, and position of each image can be changed. This makes it easier to adapt when the required image changes depending on the task and work location, even with the same work machine, and improves operability. In addition, a preset function may be provided to pre-determine the setting conditions for switching between each image.

[0033] The remote control terminal 6 can display or control information obtained from the information processing device 4 via a network NW, and may be equipped with a display device or an audio output device as an output device for this purpose. The remote control terminal 6 may be a notebook computer, smartphone, computer (e.g., tablet), or handheld computer device (not limited to, but for example, a PDA (personal digital assistant)). Examples of display devices include liquid crystal displays, PC displays, projectors, etc., and examples of audio output devices include speakers, etc. The present invention is not particularly limited in function by the output method.

[0034] As shown in Figure 1, the remote control device 7 can be, for example, a game controller. Other options include joysticks, mice, styluses, buttons, touch panels, steering wheel controllers, trackballs, keyboards, and voice control devices. Alternatively, a method of directly operating the on-screen interface using a touch panel without using a remote control device is also suitable. This allows for intuitive operations such as drop-and-drop, and is particularly effective for remote control using devices such as tablets and smartphones.

[0035] In Figure 1, for example, one information processing device 4 and three sensors (2a, 2b, 2x) are shown for each device and equipment. However, the configuration is not limited to these, and can be changed to one or more depending on the embodiment.

[0036] Figure 2 is a diagram showing the configuration of a remote control system 1, which includes a sensor 2, an information processing device 4, and a remote control terminal 6 according to this embodiment. The information processing device 4 and the remote control terminal may be implemented as a single device or as multiple devices.

[0037] Sensor 2 includes at least a camera image acquisition unit if it is a camera, and at least a sensor information acquisition unit if it is a sensor other than a camera, and performs input / output with the information processing device 4. The information processing device 4 includes at least an adjustment unit and a surrounding environment map creation unit, and performs information input and output with the sensor 2 and the remote control terminal. The remote control terminal 6 includes at least a screen display unit and a remote control unit, and performs information input and output with the information processing device 4. The remote control unit is operated by either the remote control terminal or a remote control device.

[0038] The adjustment unit can maintain real-time performance even in environments with limited communication bandwidth by limiting the data capacity of camera images and map information to a predetermined value or less per second. This reduces network latency and enables stable remote operation. Specific adjustment methods include, for example, data compression, trimming, data selection, and image quality adjustment, each of which may be performed by existing methods. The predetermined data capacity may be 50Mbps or less, 40Mbps or less, 30Mbps or less, 20Mbps or less, 15Mbps or less, 10Mbps or less, 5Mbps or less, 3Mbps or less, 2Mbps or less, 1.5Mbps or less, or 1Mbps or less. However, considering adaptation to various communication environments, it is preferable to have a data capacity of 10Mbps or less, and more preferably 3Mbps or less. This enables real-time distribution, and the camera images and surrounding environment map information are linked. To improve the visibility of the link, it is preferable to incorporate means such as markers and arrows indicating the direction of movement of the work machine into each image.

[0039] Furthermore, a function may be provided to adjust the frame rate according to the communication bandwidth when transmitting camera images and map information. For example, by setting the transmission interval to 20 fps (Frames Per Second) or higher, the transmission capacity is reduced and a fast communication speed is ensured. On the other hand, by setting the display rate of the monitor on the remote control terminal to 50 fps or higher, it is possible to display smooth captured images without display lag. The respective frame rates may be 5 fps or higher, 10 fps or higher, 15 fps or higher, 20 fps or higher, 30 fps or higher, 40 fps or higher, 60 fps or higher, or 70 fps or higher.

[0040] If there is sufficient bandwidth and data adjustment is not required, the adjustment unit can be configured not to perform any processing. This allows for flexible data transmission tailored to the situation while maintaining optimal video quality and operability.

[0041] In the embodiment, the case where the sensor 2 and work machine 3 are mainly placed at the work site was described. Environments with limitations on data transmission volume may be, for example, enclosed spaces such as underground areas, tunnels, and factories, areas where communication infrastructure is not sufficiently developed domestically or overseas, or areas with congested communication. Furthermore, the environment may be one in which at least one of the devices is moving while communicating.

[0042] In Figure 2, the configurations of each device and equipment are not limited to those shown. Although not shown, each device may also have a control unit. The processing unit can issue execution commands to the imaging unit, communication unit, input / output unit, and storage unit. The processing unit can perform calculations on the data and determine the calculation results. These communication methods may be wireless or wired, and communication between them can occur via a network.

[0043] The remote control system of the present invention utilizes camera images and sensor information to support the efficient remote operation of work machinery. The flowchart of the embodiment shown in Figure 3 will be explained.

[0044] (1) Acquisition of camera images and sensor information (S1) Images are acquired from cameras mounted on the work machinery or from cameras placed in the surrounding area. In addition, sensors such as 2D LiDAR, 3D LiDAR, accelerometers, and gyroscopes are used to detect the surrounding environment and the status of the work machinery, and to acquire data.

[0045] (2) Generation of surrounding environment map (S2) Based on the acquired sensor information, a 2D or 3D map of the surrounding environment is generated. Furthermore, the effects of vibration and tilt may be corrected using an accelerometer or similar sensor to create accurate map information.

[0046] (3) Integration of camera footage and map information (S3) The generated map information and camera footage are integrated and displayed on a single screen while maintaining correlation. This allows for an intuitive understanding of the location of the work machinery and the surrounding environment.

[0047] (4) Adjusting the data displayed on the screen (S4) The adjustment unit optimizes the camera image and map information displayed on the screen, reducing the load on the communication bandwidth and adjusting the information to what is necessary for operation.

[0048] (5) Display on the remote control terminal and execution of operations (S5) The adjusted data is transmitted to a remote control terminal, where camera footage and map information are displayed simultaneously. The remote operator controls the machinery remotely using a remote control device and touch controls on the screen.

[0049] The means and functions described above are realized by the information processing device and various terminals reading and executing a predetermined program. The program may be provided, for example, from one or more terminals via a network (cloud service, SaaS: Software as a Service). Alternatively, the program may be provided, for example, in the form of being recorded on a computer-readable recording medium. In this case, the computer reads the program from the recording medium, transfers it to an internal or external recording device, records it, and executes it. Alternatively, the program may be pre-recorded on a recording device (recording medium) such as a magnetic disk, optical disk, or magneto-optical disk, and provided to the terminal from that recording device via a communication line.

[0050] Furthermore, artificial intelligence (AI) may be used to perform adjustments and processing based on various pre-trained models, and the generating AI may generate voice, video, text, and control based on information acquired from sensors. Considering work efficiency, it is desirable that the generating AI processes in real time. This would allow remote operators to take appropriate actions in real time based on video information.

[0051] Although embodiments of the present invention have been described above, the present invention is not limited to these embodiments. Furthermore, the effects described in the embodiments of the present invention are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments. [Explanation of symbols]

[0052] 1. Remote control system 2a, 2b Sensor (Camera) 3. Working Machines 4. Information Processing Device 5a Remote control screen (main) 5b Remote control screen (sub) 5x Surrounding Environment Map 6 Remote control terminal 7 Remote control 10 Camera image acquisition unit 11 Sensor information acquisition unit 20. Surrounding Environment Map Creation Department 21 Adjustment part 30 Screen display section 31 Remote Control Unit

Claims

1. A remote control system for a work machine comprising a sensor, an information processing device, a work machine, and a remote control terminal, The remote control system is A camera image acquisition unit that acquires camera images from the aforementioned sensor, a camera, A surrounding environment map creation unit that creates map information for a two-dimensional map or a three-dimensional map based on information acquired by the aforementioned sensor other than the camera, A screen display unit that simultaneously displays the camera image and the map information on the remote control screen of the remote control terminal, The aforementioned screen display unit includes an adjustment unit that adjusts the data capacity of the video displayed to 20 Mbps or less. A remote control system in which the camera images are acquired from a camera mounted on the work machine and cameras installed around the work machine, and the total number of camera images is between two and ten.

2. The remote control system according to claim 1, characterized in that the data capacity of the video displayed by the screen display unit is 10 Mbps or less.

3. The remote control system according to claim 1, characterized in that the surrounding environment map is a two-dimensional map.

4. The remote control system according to claim 1, characterized in that the screen display unit can arbitrarily change the size or display position of the camera image and the map information.

5. The remote control system is further equipped with an accelerometer, The surrounding environment map creation unit estimates its own position based on the surrounding environment information from the sensors and the information from the acceleration sensor, and generates estimated information. The remote control system according to claim 1, characterized in that it includes a self-position estimation unit that reflects the estimation information in the map information.

6. The aforementioned work machine is a mobile unit, The remote control system according to claim 1, characterized in that the moving body is one of an automobile, forklift, shovel, crane, dump truck, automated guided vehicle, or unmanned ground vehicle.

7. The remote control system according to claim 1, characterized in that the video displayed by the screen display unit simultaneously on the remote control screen of the remote control terminal is video acquired from at least one to three cameras mounted on the work machine or installed around the work machine, and the map information is one two-dimensional map information.

8. The remote control system according to claim 1, characterized in that the adjustment unit combines the camera image and the map and transmits them as a single screen.

Images