Remote control system for industrial machinery

Abstract

To improve safety. [Solution] A remote control system for a work machine according to one embodiment includes a work machine having an imaging device that acquires images of the surroundings, a detection device that detects additional information other than images that represents at least one of the conditions of the work machine and the surroundings of the work machine, and a first communication device that transmits images and additional information; a remote control room having a second communication device that transmits operation information from an operation device to the first communication device and receives images and additional information, a display device that displays images, and an output device that outputs additional information; and a control device that controls the transmission or reception of essential information necessary for operating the work machine, including at least one of images and operation information, and additional information between the first communication device and the second communication device, wherein the control device performs different controls on each of the essential information and additional information based on the communication status between the first communication device and the second communication device.

Description

Technical Field

[0004] , , ,

[0006] , , , ,

[0005] , , ,

[0003] , , , , , ,

[0001] The present invention relates to a remote operation system for a working machine.

Background Art

[0002] Conventionally, techniques for remotely operating a working machine have been proposed. In remote operation, an operator performs an operation while referring to an image reflected on a display device (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a possibility that an abnormality may occur in the communication between the working machine and the remote operation device. In Citation Document 1, it is proposed to transmit a stop command when an abnormality occurs in the communication between the working machine and the remote operation device.

[0005] By the way, a plurality of types of information are transmitted or received between the working machine and the remote operation device. The plurality of types of information includes both information essential for operating the working machine and information that can be used as a reference when operating the working machine. That is, there is information for which remote operation becomes difficult when transmission or reception becomes impossible, and there is also information for which remote operation can continue even when transmission or reception becomes impossible. Therefore, when an abnormality occurs in the communication, it is required to perform control according to the information being transmitted or received.

[0006] In view of the above, by performing different controls for each piece of information to be received or transmitted according to the communication status between the working machine and the remote operation device, an improvement in safety is realized.

Means for Solving the Problems

[0007] A remote control system for a work machine according to one aspect of the present invention comprises: a work machine having an imaging device that acquires images of the surroundings of the work machine; a detection device that detects additional information other than the images that represents at least one of the conditions of the work machine and its surroundings; and a first communication device that transmits the images and the additional information; a remote control room having an operating device; a second communication device that transmits operation information from the operating device to the first communication device and receives the images and the additional information; a display device that displays the images; and an output device that outputs the additional information; and a control device that controls the transmission or reception of essential information necessary for operating the work machine, including at least one of the images and the operation information, and the additional information between the first communication device and the second communication device, wherein the control device performs different controls on each of the essential information and the additional information based on the communication status between the first communication device and the second communication device. [Effects of the Invention]

[0008] According to one aspect of the present invention, security is improved by performing different controls for each piece of information to be received or transmitted, depending on the communication conditions. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of a remote control system according to the first embodiment. [Figure 2] This figure shows an example of the configuration of the drive control system for an excavator according to the first embodiment. [Figure 3] This is a functional block diagram showing an example configuration of a remote control system according to the first embodiment. [Figure 4] This figure shows an example of the layout of a remote control room according to the first embodiment. [Figure 5] This is a conceptual diagram illustrating the information transmitted and received between a work machine and a remote control room according to the first embodiment. [Figure 6]This figure shows an example of volume output control in the output control unit according to the first embodiment. [Figure 7] This figure shows an example of a screen displayed by the display control unit according to the first embodiment. [Figure 8] This flowchart shows the processing procedure according to the communication status of vibration information and tilt information by the remote controller according to the first embodiment. [Figure 9] This flowchart shows the processing procedure according to the communication status of the sound signal by the remote controller according to the first embodiment. [Figure 10] This flowchart shows the processing procedure according to the communication status of essential information by the remote controller according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described below with reference to the drawings. The embodiments described below are illustrative and do not limit the invention. Not all features and combinations thereof in the embodiments of this disclosure are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and redundant descriptions may be omitted.

[0011] The working machine 100 according to the embodiment of this disclosure is a shovel. The working machine 100 may be a machine other than a shovel, such as a crane, an asphalt finisher, or a forklift. In the illustrated example, the shovel as the working machine 100 is an excavator equipped with a bucket 6 as an end attachment, but it may be an applied machine such as a forestry machine equipped with an end attachment other than the bucket 6. Furthermore, it may be a crawler crane equipped with a lower traveling body, an upper rotating body, and an attachment provided on the upper rotating body.

[0012] (First Embodiment) First, referring to FIG. 1, the outline of the remote operation system SYS according to the first embodiment will be described. FIG. 1 is a schematic diagram showing an example of the remote operation system SYS according to the first embodiment.

[0013] <Devices constituting the remote operation system> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes a work machine 100 and a remote operation room RC.

[0014] The work machine 100 and the remote operation room RC are connected so as to be able to transmit and receive data via a communication line NW

[0015] The work machine 100 enables wireless communication. And the work machine 100 can transmit and receive data to and from a device (for example, the remote operation room RC) connected to the communication line NW.

[0016] The work machine 100 exists at the work site where the work machine 100 performs work. And the work machine 100 can transmit information about the work site to the remote operation room RC. Thereby, the remote operation room RC can confirm the work site according to the information from the work machine 100. In addition, in this embodiment, the device for measuring the work site is not limited to the work machine 100, and other devices such as a fixed-point measuring device existing at the work site, a drone flying over the work site, or an imaging device that can be carried by a user may be used.

[0017] The number of work machines 100 included in the remote operation system SYS may be one or a plurality. Thereby, the remote operation system SYS can provide information about the work site to the remote operation room RC through one or a plurality of work machines 100.

[0018] <Configuration example of the remote operation room> The remote control room RC is equipped with a communication device T2, a remote controller R40, an operating device R42, an operating sensor R43, a display device D1E, a sound collection device M1E, a sound output device SP1E, a vibration device R44, and a rocking device R45. The remote control room RC also has an operating seat DS where the remote operator OP sits to remotely control the work machine 100.

[0019] The communication device (an example of a second communication device) T2 is configured to control communication with the communication device T1 (see Figure 2) attached to the work machine 100.

[0020] The remote controller (an example of a control device) R40 is an information processing device that performs various calculations. In this embodiment, the remote controller R40 is composed of a microcomputer including a CPU and memory. The various functions of the remote controller R40 are realized by the CPU executing a program stored in memory.

[0021] The display device D1E displays a screen based on information transmitted from the work machine 100, allowing the remote operator OP in the remote control room RC to visually check the area around the work machine 100. The display device D1E allows the operator to check the conditions of the work site, including the area around the work machine 100, even though the operator is in the remote control room RC.

[0022] An operating device R42 (an example of an operating unit) is equipped with an operating sensor R43 for detecting the operation of the operating device R42. The operating sensor R43 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle of the operating lever around its pivot axis. The operating sensor R43 may also consist of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operating sensor R43 outputs information regarding the operation of the operating device R42 that it has detected to the remote controller R40. The remote controller R40 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operating sensor R43 may also be configured to generate an operation signal. In this case, the operating sensor R43 may output the operation signal to the communication device T2 without going through the remote controller R40. This enables remote control of the work machine 100 from the remote control room RC.

[0023] The sound collection device M1E is a device that collects sounds generated within the remote control room RC. In the illustrated example, the sound collection device M1E is an indoor microphone and is configured to pick up the voice of the remote operator OP within the remote control room RC.

[0024] The sound output device SP1E is a device capable of outputting various types of sound information. The sound output device SP1E outputs sound based on information transmitted from the work machine 100 so that the remote operator OP in the remote control room RC can hear the sounds emitted at the work site. The sound output device SP1E is configured to output sound captured by, for example, a sound collection device M1 (see Figure 2) attached to the outside of the work machine 100. The sound output device SP1E may be a stationary device such as a speaker, or a wearable device such as earphones or headphones. The speaker may be a monaural speaker, a stereo speaker, or a surround speaker. The speaker may also be an omnidirectional speaker or a directional speaker. The wearable device may have a noise-canceling function, a spatial audio function (3D sound function), or a bone conduction function.

[0025] The vibration device R44 vibrates the operator's seat DS, where the remote operator OP is seated, with an amplitude and frequency corresponding to the control from the remote controller R40. The vibration device R44 is configured to output vibrations detected by, for example, the vibration detection device S7 (see Figure 2) attached to the work machine 100. Note that the object to be vibrated by the vibration device R44 in this embodiment is not limited to the operator's seat DS, but may also be, for example, the control device R42, etc.

[0026] The rocking device R45 rocks (tilts) the operator's seat DS where the remote operator OP is seated, in response to control from the remote controller R40. The vibration device R44 is configured to reproduce, for example, the tilt of the work machine 100 detected by the machine tilt sensor S4 (see Figure 2) attached to the work machine 100, relative to the operator's seat DS. In this embodiment, an example in which the rocking device R45 rocks (tilts) the work machine 100 relative to the operator's seat DS will be described. However, in this embodiment, the method of making the remote operator OP aware of the tilt of the work machine 100 is not limited to the method in which the rocking device R45 rocks (tilts) the operator's seat DS. For example, the remote controller R40 may display the current tilt angle of the work machine 100 on the display device D1E in a recognizable manner.

[0027] The work machine 100 comprises a lower traveling body 1, an upper rotating body 3 mounted on the lower traveling body 1 so as to be rotatable via a slewing mechanism 2, an attachment AT for performing various tasks, and a driver's cab 10. The driver's cab 10 is also called a cabin or cab. The front side of the work machine 100 (upper rotating body 3) corresponds to the side on which the attachment AT is attached to the upper rotating body 3 when the work machine 100 is viewed from directly above along the slewing axis of the upper rotating body 3. The left, right, and rear sides of the work machine 100 (upper rotating body 3) correspond to the left, right, and rear sides as seen from the perspective of an operator seated in the driver's seat inside the driver's cab 10, respectively.

[0028] The lower travel body 1 includes, for example, a pair of left and right crawlers (not shown). Specifically, the crawlers include a left crawler and a right crawler. The left crawler is driven by a left travel hydraulic motor 2ML (see Figure 2), and the right crawler is driven by a right travel hydraulic motor 2MR (see Figure 2). The left travel hydraulic motor 2ML is a travel drive unit that drives the left crawler, which is the driven part, and can rotate the left crawler. The right travel hydraulic motor 2MR is a travel drive unit that drives the right crawler, which is the driven part, and can rotate the right crawler. Note that the travel drive units may also be electric motors.

[0029] A boom 4 is rotatably mounted to the front center of the upper slewing body 3, an arm 5 is rotatably mounted to the tip of the boom 4, and a bucket 6 is rotatably mounted to the tip of the arm 5. In the illustrated example, the boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of attachment AT. The boom 4, arm 5, and bucket 6 are driven by a boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively.

[0030] Bucket 6 is an example of a work tool (end attachment). Bucket 6 is used, for example, for excavation work. Depending on the work content, other work tools may be attached to the tip of arm 5 instead of bucket 6. Other work tools may be other types of buckets, such as large buckets, slope buckets, or dredging buckets. Other work tools may also be types of work tools other than buckets, such as agitators, breakers, grapples, or lifting magnets. The excavation attachment may be provided with a bucket tilt mechanism.

[0031] Furthermore, the work machine 100 may have all or part of its driven parts, such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6, electrically driven. In other words, the work machine 100 may be a hybrid excavator or electric excavator, in which all or part of its driven parts are driven by electric actuators.

[0032] The work machine 100 according to this embodiment is configured to be remotely operated from outside the work machine 100. When the work machine 100 is remotely operated, the inside of the operator's cab 10 may be unoccupied.

[0033] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operator's actions. This enables the work machine 100 to automatically operate at least a portion of the driven parts, such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, that is, to achieve a so-called "machine control function".

[0034] Figure 2 is a schematic diagram showing an example of the configuration of the work machine 100. In Figure 2, the mechanical power transmission system, hydraulic fluid line, pilot line, and electrical control system are indicated by double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0035] The drive system of the work machine 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 also includes hydraulic actuators such as a slewing hydraulic motor 2A, a left travel hydraulic motor 2ML, a right travel hydraulic motor 2MR, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0036] The engine 11 is an example of a power source for the work machine 100, and is mounted, for example, at the rear of the upper rotating body 3. The power source for the work machine 100 may also be a combination of a battery or fuel cell and an electric motor. Specifically, the engine 11 rotates at a constant speed at a preset target rotational speed under direct or indirect control by the controller 30, driving the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 may also be a gasoline engine or a hydrogen engine, etc.

[0037] The regulator 13 controls the discharge rate of the main pump 14. For example, the regulator 13 controls the discharge rate of the main pump 14 by adjusting the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0038] The main pump 14, for example, is mounted at the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic fluid to the control valve unit 17 through the hydraulic fluid line. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump.

[0039] The control valve unit 17 is one of the hydraulic control devices that control the hydraulic system in the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171 to 176. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control the flow rate of hydraulic fluid flowing from the main pump 14 to the hydraulic actuators, and the flow rate of hydraulic fluid flowing from the hydraulic actuators to the hydraulic fluid tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-travel hydraulic motor 2ML, a right-travel hydraulic motor 2MR, and a slewing hydraulic motor 2A. Specifically, control valve 171 corresponds to the left-travel hydraulic motor 2ML, control valve 172 corresponds to the right-travel hydraulic motor 2MR, and control valve 173 corresponds to the slewing hydraulic motor 2A. Furthermore, control valve 174 corresponds to bucket cylinder 9, control valve 175 corresponds to boom cylinder 7, and control valve 176 corresponds to arm cylinder 8.

[0040] The pilot pump 15 is an example of a pilot pressure generating device and is configured to supply hydraulic fluid to a hydraulic control device via a pilot line. In the illustrated example, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pressure generating device may be implemented by the main pump 14. That is, the main pump 14 may have the function of supplying hydraulic fluid to the control valve unit 17 via a hydraulic fluid line, as well as the function of supplying hydraulic fluid to various hydraulic control devices via a pilot line. In this case, the pilot pump 15 may be omitted.

[0041] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the example shown in the figure, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0042] The operating device 26 is a device used by the operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator may be a hydraulic actuator or an electric actuator.

[0043] The operation sensor 29 is configured to detect the operator's actions using the operation device 26. In this embodiment, the operation sensor 29 detects the operating direction and amount of the operation device 26 corresponding to each actuator and outputs the detected values ​​to the controller 30. In the illustrated example, the controller 30 can control the opening area of ​​the proportional valve 31 according to the output of the operation sensor 29. The controller 30 then supplies the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the hydraulic fluid supplied to each pilot port (pilot pressure) is, in principle, the pressure corresponding to the operating direction and amount of the operation device 26 corresponding to each hydraulic actuator. Thus, the operation device 26 is configured to supply the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0044] The proportional valve 31, which functions as a control valve for machine control, is located in the pipeline connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow area of ​​the pipeline. In the illustrated example, the proportional valve 31 operates in response to control commands output by the controller 30. Therefore, the controller 30 can adjust the pilot pressure acting on the pilot port of the control valve by the proportional valve 31, independently of the operation of the operating device 26 by the operator.

[0045] This configuration allows the controller 30 to operate the hydraulic actuator corresponding to a specific operating device 26 even when no operation is being performed on that particular operating device 26.

[0046] Furthermore, as shown in Figure 2, the control system of the work machine 100 includes a controller 30, angle sensors S1, S2, S3, machine tilt sensor S4, rotation sensor S5, imaging device S6, positioning device PS, vibration detection device S7, sound collection device M1, sound output device SP1, display device D1, and communication device T1, etc.

[0047] The communication device T1 communicates with external devices through a communication network including a mobile communication network, a satellite communication network, or the Internet. The communication device T1 is, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0048] The boom angle sensor S1 detects the boom angle, which is the rotation angle of the boom 4 relative to the upper slewing body 3. The arm angle sensor S2 detects the arm angle, which is the rotation angle of the arm 5 relative to the boom 4. The bucket angle sensor S3 detects the bucket angle, which is the rotation angle of the bucket 6 relative to the arm 5.

[0049] The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may each be, for example, a rotary encoder, acceleration sensor, 6-axis sensor, IMU (Inertial Measurement Unit), or a potentiometer using a variable resistor, or a cylinder stroke sensor for detecting the stroke amount of a hydraulic cylinder.

[0050] The detection signals corresponding to the boom angle from the boom angle sensor S1, the detection signals corresponding to the arm angle from the arm angle sensor S2, and the detection signals corresponding to the bucket angle from the bucket angle sensor S3 are input to the controller 30. The detection signals may include angular velocity in addition to angle.

[0051] In this embodiment, the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 are collectively referred to as angle sensors S1, S2, and S3.

[0052] The machine tilt sensor (an example of a detection device) S4 detects the tilt state of the machine (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. The machine tilt sensor S4 is, for example, attached to the upper rotating body 3 and detects the tilt angle of the work machine 100 (i.e., the upper rotating body 3) around two axes: the front-rear direction and the left-right direction. The machine tilt sensor S4 may be, for example, an acceleration sensor, a 6-axis sensor, or an IMU. The detection signal corresponding to the tilt angle from the machine tilt sensor S4 is input to the controller 30.

[0053] The rotation sensor S5 outputs information regarding the rotation of the upper rotating body 3. The rotation sensor S5 detects, for example, the rotational angular velocity of the upper rotating body 3 relative to the lower traveling body 1. The rotation sensor S5 may also detect the rotation angle. The rotation sensor S5 may be, for example, a gyro sensor, a resolver, or a rotary encoder. The detection signal corresponding to the rotation angle or rotational angular velocity of the upper rotating body 3 detected by the rotation sensor S5 is input to the controller 30.

[0054] The imaging device S6 is mounted on the upper rotating body 3 and captures images of the area around the work machine 100, acquiring images representing the area around the work machine 100. The imaging device S6 includes, for example, a front camera, a left camera, a right camera, and a rear camera.

[0055] The front camera is a camera that captures images in front of the work machine 100 and is mounted on the outside of the operator's cab 10, such as on the roof of the operator's cab 10 or the side of the boom 4. The left camera is a camera that captures images to the left of the work machine 100, the right camera is a camera that captures images to the right of the work machine 100, and the rear camera is a camera that captures images behind the work machine 100. Specifically, the front camera, left camera, right camera, and rear camera are all monocular wide-angle cameras equipped with an image sensor such as a CCD or CMOS, and the information of the captured images is taken into the controller 30. Alternatively, the images captured by the imaging device may be output to the display device D1.

[0056] The imaging device S6 may constitute an object detection device that detects objects in the vicinity of the work machine 100. The object detection device may consist of devices other than a camera. For example, the object detection device may be a LiDAR. A LiDAR is, for example, a device capable of measuring the distance between a point cloud of 1 million or more points within the monitoring range and the LiDAR (laser source). Alternatively, the object detection device may be other devices capable of measuring the distance to an object, such as a stereo camera, a depth image camera, or a millimeter-wave radar. When a millimeter-wave radar or the like is used as the object detection device, the object detection device may determine the distance and direction of the object by transmitting a large number of signals (such as laser light) toward the object and receiving the reflected signals. Alternatively, the object detection device may be a combination of two or more types of devices. For example, the object detection device may be a combination of an imaging device and a LiDAR, or a combination of an imaging device and a millimeter-wave radar, or a combination of an imaging device and a stereo camera.

[0057] The positioning device PS measures the position of the upper rotating body 3. The positioning device PS is, for example, a GNSS (Global Navigation Satellite System) compass and detects the position and orientation of the upper rotating body 3. The detection signals corresponding to the position and orientation of the upper rotating body 3 are received by the controller 30. The function of detecting the orientation of the upper rotating body 3 may be realized by an orientation sensor attached to the upper rotating body 3. In this embodiment, the positioning device PS measures the current position of the work machine 100 in a globally identifiable reference coordinate system.

[0058] A reference coordinate system is, for example, the World Geodetic System, which can determine a location on Earth. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with its origin at the Earth's center of mass, the X-axis pointing in the direction of the intersection of the Greenwich Meridian and the equator, the Y-axis pointing in the direction of 90 degrees east longitude, and the Z-axis pointing in the direction of the North Pole.

[0059] The vibration detection device (an example of a detection device) S7 is a device that detects vibrations occurring in the work machine 100. The vibration detection device S7 may be an IMU (Inertial Measurement Unit) or the like, and it detects vibrations that are caused by external factors in the work machine 100.

[0060] The sound collection device (an example of a detection device) M1 is a device that collects external sounds and is also called a microphone. For example, the sound collection device M1 is installed in the upper rotating body 3 or the operator's cab 10 and converts sound (air vibrations) generated around the work machine 100 into mechanical vibrations, and then converts those mechanical vibrations into electrical signals. Specifically, the sound collection device M1 may include a front microphone, a left microphone, a right microphone, and a rear microphone.

[0061] The front microphone collects sounds generated in front of the work machine 100, the left microphone collects sounds generated to the left of the work machine 100, the right microphone collects sounds generated to the right of the work machine 100, and the rear microphone collects sounds generated behind the work machine 100. The electrical signals generated by the front microphone, left microphone, right microphone, and rear microphone are then input to the controller 30.

[0062] The sound output device SP1 is a device that outputs sound towards the surroundings of the work machine 100. The sound output device SP1 is an omnidirectional speaker and is configured to output sound uniformly in all directions. However, the sound output device SP1 may also be a directional speaker that outputs sound in a specific direction, such as forward.

[0063] The controller 30 is an example of a control device and is composed of a computer including, for example, a CPU, a volatile memory device, a non-volatile memory device, and various input / output interfaces. The controller 30 implements various functions, for example, by reading a program from the non-volatile memory device, loading it into the volatile memory device, and having the CPU execute it. In the illustrated example, the controller 30 is configured to implement various functions and control the work machine 100. These functions include, for example, a machine guidance function that guides the operator in manually operating the work machine 100. The functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and objects within the monitoring range around the work machine 100.

[0064] The controller 30 is configured to output control commands to the regulator 13 as needed, thereby changing the discharge rate of the main pump 14.

[0065] Furthermore, the controller 30 may be configured to perform control related to a machine guidance function that guides the manual operation of the work machine 100 by the operator through the operating device 26. Alternatively, the controller 30 may be configured to perform control related to a machine control function that automatically assists the manual operation of the work machine 100 by the operator through the operating device 26.

[0066] Furthermore, some of the functions of controller 30 may be implemented by other controllers (control devices). In other words, the functions of controller 30 may be implemented in a manner distributed among multiple controllers. For example, machine guidance functions and machine control functions may be implemented by dedicated controllers (control devices).

[0067] <Block configuration of the remote control system> Figure 3 is a functional block diagram showing an example configuration of the remote control system SYS according to this embodiment. In the example shown in Figure 3, the block configurations of the remote control room RC and the work machine 100, which are included in the remote control system SYS, are shown. The hardware configuration of the work machine 100 will not be explained.

[0068] <Configuration of the Remote Control Room (RC)> The remote control room RC includes a remote controller R40, a communication device T2, an operation sensor R43, an operation device R42, a vibration device R44, an oscillating device R45, a sound collection device M1E, a sound output device SP1E, a display device D1E, and an input device D2E. The communication device T2, operation sensor R43, operation device R42, vibration device R44, oscillating device R45, sound collection device M1E, and sound output device SP1E have been described above, so their explanation is omitted.

[0069] The input device D2E is located within reach of the operator seated in the remote control room RC and receives various operation inputs from the operator, outputting signals corresponding to the operation inputs to the remote controller R40. The input device D2E includes a touch panel mounted on the display of a display device that shows various information images, a knob switch located at the tip of the lever part of the operation device R42, and button switches, levers, toggles, rotary dials, etc., installed around the display device D1E. Signals corresponding to the operations performed on the input device D2E are received by the remote controller R40.

[0070] Next, we will explain the remote control room RC. Figure 4 shows an example of the layout of the remote control room RC. The remote control room RC is equipped with multiple control devices R42, with the operator's seat DS as the reference point.

[0071] In this embodiment, the display device D1E is a multi-display consisting of six monitors arranged in two vertical rows and three horizontal columns, as shown in Figure 4. Specifically, the display device D1E includes the central monitor D1Ea, the upper monitor D1Eb, the left monitor D1Ec, the right monitor D1Ed, the upper left monitor D1Ee, and the upper right monitor D1Ef.

[0072] <<Explanation of communication status for remote control>> In this embodiment, the work machine 100 transmits information to the remote control room RC, which is located inside the remote control room, to allow the remote operator OP to recognize the status of the work machine 100 or the surrounding environment.

[0073] Figure 5 is a conceptual diagram illustrating the information transmitted and received between the work machine 100 and the remote control room RC according to this embodiment.

[0074] As shown in Figure 5, for example, the image captured by the imaging device S6 is transmitted from the work machine 100 to the remote control room RC.

[0075] In the remote control room RC, the remote controller R40 displays the received video on the display device D1E. The video contains information necessary for the remote operator OP to operate the work machine 100, and in this embodiment, it is considered one of the essential pieces of information.

[0076] Operation information, controlled by the control device R42 in the remote control room RC, is transmitted from the remote control room RC to the work machine 100. The work machine 100 operates according to the received operation information. Thus, operation information is necessary for the remote operator OP to operate the work machine 100, and in this embodiment, it is considered one of the essential pieces of information.

[0077] In this embodiment, among the information transmitted or received between the remote control room RC and the work machine 100, the information necessary for remotely operating the work machine 100 is defined as essential information.

[0078] On the other hand, additional information that is not video and is used as reference for the remote operator OP to recognize the status of the work machine 100 or the surrounding environment of the work machine 100, among the information transmitted or received between the remote control room RC and the work machine 100, is defined as additional information.

[0079] Furthermore, sound signals collected by the sound collection device M1 are transmitted from the work machine 100 to the remote control room RC. In addition, vibration information detected by the vibration detection device S7 is transmitted from the work machine 100 to the remote control room RC. Furthermore, tilt information detected by the machine tilt sensor S4 is transmitted from the work machine 100 to the remote control room RC.

[0080] In the remote control room RC, the remote controller R40 outputs the received sound signal from the sound output device SP1E. The remote controller R40 also outputs the received vibration information as vibration transmitted to the remote operator OP via the operator's seat DS from the vibration device R44. Furthermore, the remote controller R40 tilts the operator's seat DS via the rocking device R45 to achieve the current tilt state of the work machine 100, as indicated by the received tilt information.

[0081] In this embodiment, sound signals, vibration information, and tilt information are each considered as additional information. However, this embodiment does not limit the additional information to sound signals, vibration information, and tilt information, but rather includes any information that indicates at least one of the conditions of the work machine 100 and its surroundings, and is not video information. For example, it may include force information for recognizing the load on the attachment AT.

[0082] Furthermore, the additional information is not limited to information transmitted from the work machine 100 to the remote control room RC, but may also include information transmitted from the remote control room RC to the work machine 100. For example, an audio signal transmitted from the remote control room RC to the work machine 100 may be included in the additional information.

[0083] In this embodiment, the communication device T1 of the work machine 100 uses multiple mobile communication lines to secure a communication bandwidth that enables real-time transmission and reception of information with the remote control room RC. In the example shown in Figure 5, the communication device T1 of the work machine 100 uses a first mobile communication line and a second mobile communication line for communication.

[0084] Furthermore, the work machine 100 may, for example, use the mobile communication line with the higher communication quality among multiple mobile communication lines for transmitting or receiving essential information, and use the mobile communication line with the lower communication quality for transmitting or receiving additional information.

[0085] The communication device T1 of the work machine 100 enables the transmission and reception of essential information and additional information in real time by using multiple mobile communication lines.

[0086] However, the communication status between the work machine 100 and the remote control room RC is constantly changing due to various factors. Therefore, the remote controller R40 and controller 30 monitor the communication status in order to detect changes in the communication status.

[0087] Any method can be used to monitor the communication status. For example, the remote controller R40 may detect whether or not an abnormality such as packet loss or communication delay has occurred by executing a "PING" command with the timestamp function enabled to the address assigned according to the information being transmitted or received. For example, if it is estimated that a delay has occurred based on the response rate detected by executing the "PING" command, if there is no response to the "PING" command, or if there is a discrepancy (larger time difference) in the timestamp relative to the execution interval of the "PING" command, then packet loss or communication delay may be recognized as having occurred. Furthermore, the controller 30 or remote controller R40 may embed a timestamp or bucket identification information in the header of each packet it transmits, and the remote controller R40 or controller 30 may check the information embedded in the received packet to recognize whether or not packet loss or communication delay has occurred.

[0088] The remote controller R40 then performs different controls for essential information and additional information, respectively, based on the communication status between communication device T1 and communication device T2. Specifically, the remote controller R40 processes differently depending on whether there is an abnormality in the transmission or reception of essential information or an abnormality in the transmission or reception of additional information.

[0089] In other words, if an abnormality occurs in the transmission or reception of essential information, safety must be ensured, so the remote controller R40 and controller 30 have a method for stopping operations based on the operation information. The method for stopping operations may be the transmission of a stop signal from the remote controller R40 to controller 30.

[0090] On the other hand, if an abnormality occurs in the transmission or reception of additional information, the remote operator (OP) can continue to operate the work machine 100, provided that no abnormality occurs in the transmission or reception of essential information.

[0091] However, if an anomaly occurs in the communication between communication device T1 and communication device T2 (e.g., packet loss or communication delay), and the received additional information is output to the remote control room RC, the remote operator OP may misinterpret the working machine 100 or the situation around the working machine 100. For example, if the working machine 100 has already completed moving from the inclined surface to the horizontal surface, but the remote controller R40 continues to control the tilting of the operator's seat DS based on delayed inclination information, the remote operator OP may misinterpret the working machine 100 as still moving on the inclined surface. In this way, when there is a communication delay in the received additional information, the output based on the additional information may cause the remote operator OP to misinterpret the situation of the working machine 100 or the situation around the working machine 100.

[0092] Furthermore, if the remote operator OP operates the work machine 100 based on an incorrect perception of the work machine 100's condition or the condition of the work machine 100's surroundings, the work machine 100 may perform actions that are inconsistent with the current situation.

[0093] Therefore, in this embodiment, the remote controller R40, based on the communication status between communication device T1 and communication device T2, will, for example, if it is determined that there is a delay or packet loss in the reception of vibration information, stop outputting vibrations based on vibration information from the vibration device R44, while continuing to output essential information (e.g., video). Similarly, if it is determined that there is a delay or packet loss in the reception of sound signals and tilt information, the remote controller R40 will stop outputting sound signals from the sound output device SP1E, or tilt control by the rocking device R45 based on tilt information.

[0094] <<Excavator Functional Blocks>> Returning to Figure 3, we will now describe each functional block within the controller 30 of the work machine 100. Each functional block within the controller 30 is conceptual and does not necessarily need to be physically configured as shown in the figure. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed in each functional block is realized, all or any part thereof, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The controller 30, by realizing the program, includes an acquisition unit 301, a transmission control unit 302, a reception control unit 303, and an actuator drive unit 304.

[0095] The acquisition unit 301 acquires signals from various detection devices installed on the work machine 100. For example, the acquisition unit 301 acquires inclination information from the machine inclination sensor S4, which shows the inclination state of the machine (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. The acquisition unit 301 also acquires position information from the positioning device PS, which shows the measurement results such as the position and orientation of the work machine 100. Furthermore, the acquisition unit 301 acquires vibration information from the vibration detection device S7, which shows the detection results of vibrations occurring in the work machine 100.

[0096] Furthermore, the acquisition unit 301 acquires video from the imaging device S6. In addition, the acquisition unit 301 acquires sound signals from the sound collection device M1 that represent sounds collected from around the work machine 100.

[0097] The transmission control unit 302 controls the transmission of various information based on the acquisition results of the acquisition unit 301 to the remote control room RC via a communication device (an example of a first communication device) T1. For example, the transmission control unit 302 controls the transmission of tilt information detected by the machine tilt sensor S4, vibration information indicating vibrations occurring in the work machine 100, video images captured by the imaging device S6, sound signals collected by the sound collection device M1, and position information indicating the position and orientation of the work machine 100 to the remote control room RC.

[0098] The receiving control unit 303 controls the reception of various information from the remote control room RC via the communication device T1. For example, the receiving control unit 303 receives operation signals from the remote control room RC to control the operation of the work machine 100.

[0099] The actuator drive unit 304 is configured to drive the actuator mounted on the work machine 100. In this embodiment, the actuator drive unit 304 generates and outputs an operating signal for each of the multiple solenoid valves included in the proportional valve 31 based on the operation signal transmitted from the remote control room RC.

[0100] Upon receiving an activation signal, each solenoid valve increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0101] <<Functional Blocks of the Remote Control Room>> This section describes the various functional blocks within the remote controller (an example of a control unit) R40 of the remote control room RC. Each functional block within the remote controller R40 is conceptual and does not necessarily need to be physically configured as shown in the diagram. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed by each functional block is realized, in whole or in any part, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The remote controller R40, by implementing the program, includes a receiving control unit 401, a display control unit 402, an output control unit 403, a determination unit 404, a signal generation unit 405, and a transmission control unit 406.

[0102] The receiving control unit 401 controls the receiving of various information from the work machine 100 via the communication device T2.

[0103] For example, the receiving control unit 401 controls the reception of essential information from the work machine 100. Specifically, the receiving control unit 401 controls the reception of video captured by the imaging device S6.

[0104] Another example is the receiving control unit 401, which controls the reception of additional information from the work machine 100. Specifically, the receiving control unit 401 controls the reception of tilt information from the machine tilt sensor S4, vibration information from the vibration detection device S7, and sound signals from the sound collection device M1.

[0105] Furthermore, the receiving control unit 401 controls the receiving of position information from the work machine 100, indicating the position and orientation of the work machine 100.

[0106] The display control unit 402 controls the display of various information on the display device D1E. For example, the display control unit 402 controls the display of the image captured by the imaging device S6 as an image showing the surroundings of the work machine 100 on the display device D1E. In addition, the display control unit 402 controls the display of information indicating the current status of the work machine 100 on the display device D1E based on the detection results of the various detection devices of the work machine 100 received by the receiving control unit 401.

[0107] The output control unit 403 performs output control based on additional information. For example, the output control unit 403 controls the output of the received sound signal from the sound output device SP1E.

[0108] Furthermore, the output control unit 403 controls the tilt of the operator's seat DS based on the received tilt information via the rocking device R45. Moreover, this embodiment does not limit the output based on tilt information to the tilt control of the operator's seat DS. For example, the display control unit 402 may perform control to display information that allows recognition of the tilt angle of the work machine 100 (e.g., an icon of the work machine 100) on the display device D1E.

[0109] Furthermore, the output control unit 403 controls the vibration of the operator's seat DS based on the received vibration information via the vibration device R44.

[0110] The determination unit 404 determines whether or not there is an abnormality in the communication status for each of the essential information and additional information being received. For example, the determination unit 404 determines whether or not packet loss or communication delay is occurring for each of the essential information and additional information being received. Any method can be used to determine whether or not packet loss or communication delay is occurring. For example, the system may detect whether or not an abnormality such as packet loss or communication delay is occurring by executing a "PING" command, or it may detect an abnormality such as packet loss or communication delay by referring to information embedded in the header of each packet.

[0111] Furthermore, the determination unit 404 may also determine whether an abnormality has occurred in any way other than whether an abnormality has occurred in the communication status. For example, the determination unit 404 may determine whether a delay or abnormality has occurred in the encoding or decoding of the sound signal included in the additional information. Any method may be used to determine whether a delay or abnormality has occurred in the encoding of the sound signal. For example, the determination unit 404 may determine whether a delay or abnormality has occurred based on notifications (for example, PING with a timestamp function enabled) output from the controller 30 of the work machine 100 at the start and end of the encoding of the sound signal.

[0112] The signal generation unit 405 generates operation signals to control the operation of the work machine 100 according to the information received from the operation sensor R43.

[0113] Furthermore, if the signal generation unit 405 determines, based on the essential information it has received, that an abnormality has occurred, it generates an operation signal to stop the operation of the work machine 100.

[0114] The transmission control unit 406 controls the transmission of various types of information to the work machine 100. For example, the transmission control unit 406 controls the transmission of operation signals generated by the signal generation unit 405 to the work machine 100.

[0115] If the determination unit 404 determines that there is an abnormality in the communication status for each of the received additional information, it stops outputting the additional information from the various output devices and continues outputting essential information (e.g., video) to the display device D1E.

[0116] For example, if the determination unit 404 determines that packet loss or communication delay has occurred in the received tilt information, the output control unit 403 performs control to stop the tilt control of the operator's seat DS by the rocking device R45. In addition, when performing the control to stop the tilt, the output control unit 403 also performs control to reduce the amount of tilt based on the tilt information.

[0117] Furthermore, if the determination unit 404 determines that packet loss or communication delay has occurred in the received vibration information, the output control unit 403 controls the vibration output from the vibration device R44 to stop. At that time, the output control unit 403 controls the amount of vibration output from the vibration device R44 to decrease based on the vibration information.

[0118] Furthermore, if the determination unit 404 determines that packet loss or communication delay has occurred in the received sound signal, it determines whether or not there are people around the work machine 100. Only if it determines that there are no people, the output control unit 403 controls the output of sound from the sound output device SP1E to stop. At that time, the output control unit 403 controls the volume output from the sound output device SP1E to decrease. In other words, if there are people around the work machine 100, there is a possibility that those people will speak. It is considered better to output the voice spoken by a person to the remote operator OP, even if it is interrupted or delayed. Therefore, in this embodiment, even if packet loss or delay occurs in the sound signal, if there are people around the work machine 100, the output of sound from the sound signal is continued.

[0119] The determination unit 404 determines whether or not the abnormality in the communication status for receiving the additional information has been resolved after the output based on the additional information has been stopped.

[0120] For example, if the determination unit 404 determines that packet loss and communication delay have been eliminated in the received tilt information, the output control unit 403 initiates tilt control of the operator's seat DS by the rocking device R45. Furthermore, when initiating the control, the output control unit 403 increases the amount of tilt based on the tilt information.

[0121] Similarly, if the determination unit 404 determines that packet loss and communication delay have been eliminated in the received vibration information or sound signal, the output control unit 403 controls the start of vibration output from the vibration device R44 or sound output from the sound output device SP1E. At that time, the output control unit 403 controls the amount of vibration output from the vibration device R44 or the volume output from the sound output device SP1E to increase based on the vibration information.

[0122] Figure 6 shows an example of volume output control in the output control unit 403 according to this embodiment. In the example shown in Figure 6, the vertical axis represents volume and the horizontal axis represents time. Although the example shown in Figure 6 shows the case where the vertical axis is volume, the explanation of vibration amount or tilt amount is omitted as they are converted in the same way. In the example shown in Figure 6, it is assumed that there are no people around the work machine 100.

[0123] In the example shown in Figure 6, at time t1, the determination unit 404 determines that packet loss or communication delay has occurred in the received sound signal. Therefore, the output control unit 403 starts controlling the volume output from the sound output device SP1E to decrease. Then, at time t2, the output control unit 403 stops the sound output from the sound output device SP1E.

[0124] Subsequently, at time t3, the determination unit 404 determines that packet loss and communication delay have been eliminated in the received sound signal. Therefore, the output control unit 403 starts outputting sound from the sound output device SP1E. The output control unit 403 then starts controlling the volume output from the sound output device SP1E to increase its output volume. Finally, at time t4, the control to increase the volume output from the sound output device SP1E is stopped.

[0125] In this embodiment, by controlling the vibration, tilt, or volume based on additional information, or by controlling the vibration, tilt, or volume, sudden stops or starts of the output can be suppressed, thereby preventing startling the remote operator (OP). Therefore, it is possible to ensure the safety of operation by the remote operator (OP).

[0126] In this embodiment, the display control unit 402 performs control to display a message on the display device D1E indicating that the output of additional information will be stopped when it stops outputting additional information from various output devices.

[0127] Figure 7 shows an example of a screen displayed by the display control unit 402 according to this embodiment. The example screen shown in Figure 7 is an example of a screen displayed by the display control unit 402 when the determination unit 404 determines that packet loss and communication delay have occurred in the vibration information and tilt information it has received.

[0128] In the example shown in Figure 7, the central monitor D1Ea of the display device D1E displays an image captured by the front camera of the imaging device S6. The display control unit 402 then superimposes the message "Communication error; Vibration and tilt control will be stopped." onto the image. In this embodiment, the message is displayed, for example, semi-transparently so that the image can be viewed through the message. Therefore, the display of the message is prevented from interfering with the operation of the work machine 100. After the display, the output control unit 403 starts control to reduce the amount of vibration and tilt.

[0129] This display allows the remote operator (OP) to recognize that a communication anomaly has occurred without taking their eyes off the device. Furthermore, the remote operator (OP) can recognize that the vibration control and tilt control are ceasing. The remote operator (OP) can infer the current communication status from this display and control. Therefore, the remote operator (OP) can perform operations that take the communication status into consideration, thereby improving safety.

[0130] Furthermore, when the display control unit 402 resumes outputting additional information from various output devices, it performs control to display a message on the display device D1E indicating that output will be resumed. For example, if the determination unit 404 determines that packet loss or communication delay of vibration information and tilt information has been resolved, the display control unit 402 displays a message on the display device D1E that reads, "Communication restored; vibration and tilt control will begin." The remote operator OP can recognize that vibration control and tilt control will resume through this display and control. Subsequently, the remote operator OP can infer the status of the work machine 100 or the surrounding environment based on the vibration control and tilt control and perform operations accordingly, thereby improving safety.

[0131] This section describes the control performed by the remote controller R40 according to the communication status of vibration information and tilt information among the additional information. Figure 8 is a flowchart showing the processing procedure by the remote controller R40 according to the communication status of vibration information and tilt information according to this embodiment.

[0132] The receiving control unit 401 of the remote controller R40 starts control to receive vibration information and tilt information (S1801).

[0133] The determination unit 404 determines whether or not a communication delay or packet loss has occurred in each of the received vibration information and tilt information (S1802). The criteria for determining whether or not a communication delay or packet loss has occurred may be any criteria, or conventional criteria may be used.

[0134] If the determination unit 404 determines that there is no communication delay or packet loss (S1802: NO), the output control unit 403 controls the vibration output to the operator's seat DS based on the received vibration information and controls the tilt of the operator's seat DS based on the tilt information (S1803). After that, control is performed again from S1801.

[0135] On the other hand, if the determination unit 404 determines that a communication delay or packet loss has occurred in at least one of the received vibration information and tilt information (S1802: YES), the display control unit 402 displays a message indicating that at least one of the vibration output control and tilt control will be stopped (S1804).

[0136] The output control unit 403 controls at least one of the vibrations output according to the vibration information and the tilt amount controlled according to the tilt information to decrease, and then stops at least one of the vibration output control and tilt control (S1805). The vibration output control and tilt control that are stopped are those controls corresponding to information (one or more of the vibration information and tilt information) where communication delay or packet loss is occurring.

[0137] The determination unit 404 then determines whether or not the communication delay and packet loss have been resolved in each of the received vibration information and tilt information (S1806).

[0138] If the determination unit 404 determines that the communication delay or packet loss has not been resolved (S1806: NO), it performs the determination in S1806 again after a predetermined time has elapsed.

[0139] On the other hand, if the determination unit 404 determines that the communication delay or packet loss has been resolved (S1806: YES), the display control unit 402 displays a message indicating that it will resume at least one of the vibration output control and tilt control outputs (S1807).

[0140] The output control unit 403 starts outputting at least one of the vibration output control and tilt control, and then increases at least one of the vibration output according to the vibration information and the tilt amount controlled according to the tilt information (S1808). After that, it resumes processing from S1801. Note that the vibration output control and tilt control to be started is the control corresponding to information for which communication delay or packet loss has been resolved (at least one of the vibration information and tilt information).

[0141] As described above, the remote controller R40 according to this embodiment stops outputting one or more additional information, either vibration information or tilt information, and then starts outputting additional information based on the communication status between communication device T1 and communication device T2. When the communication delay or the like is resolved, one or more of the vibration output and tilt control are resumed, so the remote operator OP can recognize the status of the work machine 100 or the status of the work machine 100, thereby improving safety.

[0142] Next, we will describe the control performed by the remote controller R40 according to this embodiment, in accordance with the communication status of the sound signal among the additional information. Figure 9 is a flowchart showing the processing procedure by the remote controller R40 according to the communication status of the sound signal according to this embodiment.

[0143] The receiver control unit 401 of the remote controller R40 starts the control to receive the sound signal (S1901).

[0144] The determination unit 404 determines whether or not a communication delay or packet loss has occurred in the received audio signal (S1902). The criteria for determining whether or not a communication delay or packet loss has occurred may be any criteria, including conventional criteria.

[0145] If the determination unit 404 determines that there is no communication delay or packet loss (S1902: NO), the output control unit 403 controls the output of the sound indicated by the received sound signal from the sound output device SP1E (S1903). After that, control is performed again from S1901.

[0146] On the other hand, if the determination unit 404 determines that there is a communication delay or packet loss in the received sound signal (S1902: YES), the determination unit 404 determines whether or not there are people in the surrounding area of ​​the received video (S1904). If it determines that there are people in the surrounding area (S1904: YES), the display control unit 402 displays a message indicating that there is an abnormality in the sound output (e.g., delay or interruption) (S1905). Then, the output control unit 403 controls the output of the sound indicated by the received sound signal from the sound output device SP1E (S1903). After that, control is performed again from S1901.

[0147] On the other hand, if the determination unit 404 determines that there are no people in the surrounding area in the received video (S1904: NO), the display control unit 402 displays a message indicating that it will stop outputting sound (S1906).

[0148] Subsequently, the output control unit 403 controls the output volume based on the sound signal, and then stops the sound output control (S1907).

[0149] The determination unit 404 then determines whether or not the communication delay and packet loss in the received audio signal have been eliminated (S1908).

[0150] If the determination unit 404 determines that the communication delay or packet loss has not been resolved (S1908: NO), it performs the determination in S1908 again after a predetermined time has elapsed.

[0151] On the other hand, if the determination unit 404 determines that the communication delay or packet loss has been resolved (S1908: YES), the display control unit 402 displays a message indicating that it will resume outputting sound (S1909).

[0152] After the output control unit 403 starts outputting sound, it controls the output volume to increase according to the sound signal (S1910). Then, it resumes processing from S1901.

[0153] In this embodiment, if the remote controller R40 determines that a person is present in the received video, it continues to output sound, indicated by an audio signal, regardless of the communication status. Therefore, although there may be delays or interruptions in the output sound, it is possible to hear voices emitted from people present around the work machine 100. Thus, the remote operator OP can recognize the situation around the work machine 100, thereby improving safety.

[0154] In this embodiment, the remote controller R40 continues to display the video included in the essential information on the display device D1E while performing the processing shown in Figures 8 and 9. This prevents the remote operator OP from making incorrect judgments about the surrounding conditions or the status of the work machine 100 based on the additional information by stopping the output based on the additional information if an abnormality occurs in the additional information. The remote operator OP can then judge the surrounding conditions from the video and continue operating the work machine 100.

[0155] (Second embodiment) In the first embodiment, the case where an abnormality occurs in the communication status when receiving additional information was described. Therefore, in the second embodiment, the case where an abnormality occurs in the communication status when receiving or receiving essential information will be described. Note that the configuration of the remote control system SYS in the second embodiment is the same as in the first embodiment and will not be described further.

[0156] Next, we will describe the control performed by the remote controller R40 according to the communication status of essential information according to this embodiment. Figure 10 is a flowchart showing the processing procedure by the remote controller R40 according to the communication status of essential information according to this embodiment.

[0157] The receiving control unit 401 of the remote controller R40 starts control to receive essential information, including video (S2001).

[0158] The determination unit 404 determines whether or not there is a communication delay or packet loss in the essential information being received (S2002).

[0159] If the determination unit 404 determines that there is no communication delay or packet loss (S2002: NO), for example, the display control unit 402 controls the display device D1E to display the video included in the received essential information (S2003).

[0160] On the other hand, if the determination unit 404 determines that there is a communication delay or packet loss in the essential information being received (S2002: YES), the display control unit 402 displays a message indicating that the communication status has deteriorated and that output related to additional information will be stopped (S2004).

[0161] Subsequently, the output control unit 403 controls the output amount of the additional information (for example, vibration information, tilt information, and sound signals, at least one of these) to be reduced, and then stops the output of the additional information (S2005).

[0162] Then, the transmission control unit 406 sends a request to the work machine 100 to stop transmitting the additional information, and a request to transmit the essential information using the communication line that was used for communication (S2006).

[0163] Subsequently, the receiving control unit 401 of the remote controller R40 starts the control to receive essential information, including video, from the switched communication line (S2007).

[0164] In this embodiment, if the remote controller R40 determines that an abnormality has occurred in the communication status of essential information, it stops transmitting additional information to the controller 30 and uses the communication bandwidth of the additional information for the communication of essential information, thereby continuing the communication of essential information. In this embodiment, as an example of when it is determined that the communication status of essential information does not meet a predetermined standard, the case in which it is determined that an abnormality has occurred in the communication status of essential information, in other words, when it is determined that a communication delay or packet loss has occurred, has been described. However, this embodiment does not limit the case in which it is determined that the communication status of essential information does not meet a predetermined standard to the case in which it is determined that a communication delay or packet loss has occurred, but rather it is sufficient if the communication status of the received essential information is abnormal to the extent that it becomes difficult for the remote operator OP to operate the work machine 100. Furthermore, the remote controller R40 in this embodiment is not limited to the method of stopping the transmission of additional information, but may also reduce the amount of additional information transmitted compared to before the determination and use the communication bandwidth that was used for the additional information for the communication of essential information, thereby continuing the communication of essential information. In this embodiment, depending on the communication status of essential information, the stability of receiving essential information can be improved by using the communication bandwidth of additional information for receiving essential information. Therefore, the remote operator (OP) can easily grasp the status of the work machine 100 or the surrounding environment of the work machine 100 based on essential information, thereby improving safety.

[0165] (Third embodiment) The embodiments described above explained the control when receiving additional or essential information. However, the embodiments described above are not limited to the control when receiving additional or essential information. Therefore, the third embodiment will explain the control when transmitting additional and essential information. The configuration of the remote control system SYS in the third embodiment is the same as in the first embodiment, so its description will be omitted.

[0166] In this embodiment, the communication device T2 transmits essential information, including operation signals, to the communication device T1 of the work machine 100, in accordance with the control of the remote controller R40.

[0167] In this embodiment, the sound collection device M1E in the remote control room RC collects the voice emitted by the remote operator OP as an audio signal. The communication device T2 transmits the audio signal to the communication device T1 of the work machine 100 according to the control of the remote controller R40. In this embodiment, the audio signal transmitted to the communication device T1 of the work machine 100 is used as additional information.

[0168] Furthermore, the remote controller R40 performs the same control when transmitting additional information and essential information as when receiving additional information and essential information as described in the above-described embodiment.

[0169] For example, if the determination unit of the remote controller R40 determines that there is an abnormality in the communication status of the essential information being transmitted, it controls the transmission control unit 406 to stop transmitting additional information to the work machine 100 and to transmit the essential information using the communication line that was being used for transmitting the additional information. Any method can be used for switching the communication line, not limited to well-known methods.

[0170] In this embodiment, if the remote controller R40 determines that an abnormality has occurred in the communication status of essential information, it stops the transmission of supplementary information and uses the communication bandwidth of the supplementary information for the transmission of essential information, thereby continuing the transmission of essential information. However, the remote controller R40 in this embodiment is not limited to stopping the transmission of supplementary information; it may also reduce the amount of supplementary information communicated compared to before the determination and use the communication bandwidth that was used for supplementary information for the transmission of essential information, thereby continuing the transmission of essential information. In this embodiment, by using the communication bandwidth of supplementary information for the transmission of essential information according to the communication status of essential information, the stability of the transmission of essential information can be improved. Therefore, in this embodiment, by stabilizing the transmission of essential information including operation signals, operability can be improved, as can the stability of the operation of the work machine 100.

[0171] <effect> In the embodiment described above, safety is improved by enabling the output or transmission of information according to the current communication status based on the communication status between the communication device T1 of the work machine 100 and the communication device T2 of the remote control room RC.

[0172] In the embodiment described above, if an abnormality occurs in the communication status of additional information between the communication device T1 of the work machine 100 and the communication device T2 of the remote control room RC, control is performed to stop the output of the additional information and continue the output of essential information. This makes it possible to continue operating the work machine 100 by outputting the essential information, and also suppresses erroneous judgments by the remote operator OP based on the additional information, thereby improving safety.

[0173] Preferred embodiments and modifications of the present disclosure have been described above. However, the inventions of the present disclosure are not limited to the embodiments described above. Various modifications, substitutions, etc., can be applied to the embodiments described above without departing from the scope of the inventions of the present disclosure. Furthermore, each of the features described with reference to the embodiments described above may be combined as appropriate, as long as they do not conflict technically. [Explanation of symbols]

[0174] 100 working machines 1. Lower running body 2. Swivel mechanism 3. Upper rotating body 4 Boom 5 Arms 6 buckets S4 Aircraft tilt sensor S6 imaging device S7 Vibration detection device M1 Sound collection device SP1 Audio Output Device T1 Communication Device PS positioning device 30 controllers 301 Acquisition Department 302 Transmission Control Unit 303 Receiving Control Unit 304 Actuator drive unit 31 Proportional valve RC Remote Control Room T2 Communication Device D1E display device R40 Remote Controller 401 Receiving Control Unit 402 Display Control Unit 403 Output Control Unit 404 Judgment section 405 Signal Generation Unit 406 Transmission Control Unit R42 operating device R43 Operation Sensor R44 Vibration Device R45 Oscillating device

Claims

1. A work machine comprising: an imaging device that acquires images of the surroundings of the work machine; a detection device that detects additional information other than the images that represents at least one of the conditions of the work machine and its surroundings; and a first communication device that transmits the images and the additional information. A remote control room having an operating device, a second communication device that transmits operation information from the operating device to the first communication device and receives the video and the additional information, a display device that displays the video, and an output device that outputs the additional information, Between the first communication device and the second communication device, there is a control device that controls the transmission or reception of essential information necessary for operating the work machine, including at least one of the video and the operation information, and the additional information. The control device performs different controls on the essential information and the additional information, respectively, based on the communication status between the first communication device and the second communication device. Remote control system for industrial machinery.

2. The control device performs control to stop outputting the additional information from the output device and to continue outputting the essential information, based on the communication status of the additional information between the first communication device and the second communication device. A remote control system for a work machine according to claim 1.

3. The additional information received by the second communication device from the first communication device includes an audio signal indicating sounds collected from around the work machine. If the control device determines that a person is present in the received video, it will continue to output the sound indicated by the sound signal included in the additional information, regardless of the communication status. A remote control system for a work machine according to claim 2.

4. After stopping the output of the additional information, the control device starts outputting the additional information from the output device based on the communication status between the first communication device and the second communication device. A remote control system for a work machine according to claim 2.

5. The control device performs control to reduce the output amount of the additional information when stopping the output of the additional information, or performs control to increase the output amount of the additional information output by the output device when starting the output of the additional information. A remote control system for a work machine according to claim 4.

6. When the control device stops outputting the additional information from the output device, it outputs to the display device a message indicating that it has stopped outputting the additional information from the output device, or when it starts outputting the additional information, it outputs to the display device a message indicating that it has started outputting the additional information from the output device. A remote control system for a work machine according to claim 4.

7. If the control device determines that the communication status of the essential information between the first communication device and the second communication device does not meet a predetermined standard, it reduces the amount of additional information communicated compared to before the determination and continues to transmit or receive the essential information. A remote control system for a work machine according to claim 1.

Images