Control system and control device for industrial machinery

Abstract

To facilitate remote operation of industrial machinery. [Solution] The control system for the work machine comprises a work machine having a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable, an attachment having a boom and an arm, a sensor for detecting the posture of the attachment, and a first communication device for transmitting information indicating the detected posture; a second communication device for receiving the information; an operating device for receiving operation information for operating the attachment; and a display device that displays a predicted image showing the posture obtained by operating the attachment according to the operation information, based on the operation information received by the operating device and the posture indicated in the information, from the posture indicated in the information.

Description

Technical Field

[0001] The present invention relates to a control system for a working machine and a control device.

Background Art

[0002] Conventionally, in order to remotely operate a working machine to perform work, an image captured by an imaging device attached to the working machine is transmitted to a device for performing remote operation, and the operator refers to the image to visually recognize the posture of the working machine operating by remote operation. A technique has been proposed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when transmitting an image or the like from the working machine to a device for performing remote operation, a communication delay occurs, so the posture of the working machine represented in the received image or the like is deviated from the current posture of the working machine. For this reason, there is a problem that it is difficult for the operator to grasp the current posture and the like of the working machine.

[0005] In view of the above, by allowing the operator to grasp the posture of the working machine, remote operation of the working machine is facilitated.

Means for Solving the Problems

[0006] A control system for a work machine according to one aspect of the present invention comprises a work machine having a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable, an attachment having a boom and an arm, a sensor for detecting the posture of the attachment, and a first communication device for transmitting information indicating the detected posture; a second communication device for receiving the information; an operating device for receiving operation information for operating the attachment; and a display device for displaying a predicted image showing the posture obtained by operating the attachment according to the operation information, based on the operation information received by the operating device and the posture indicated in the information, from the posture indicated in the information. [Effects of the Invention]

[0007] According to one aspect of the present invention, remote operation of a work machine is facilitated by allowing the posture of the work machine to be understood. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing an example of a remote control system according to the first embodiment. [Figure 2] This is a side view showing a shovel (excavator) according to the first embodiment. [Figure 3] This figure shows an example of the configuration of the drive control system for an excavator according to the first embodiment. [Figure 4] This is a functional block diagram showing an example configuration of a remote control system according to the first embodiment. [Figure 5] This figure shows an example of the layout of a remote control room according to the first embodiment. [Figure 6] This figure illustrates the concept of remote operation of a work machine by an operator from a remote control room according to the first embodiment. [Figure 7] This is a flowchart showing the processing procedure for displaying the orientation of the attachment of the work machine in the remote controller according to the first embodiment. [Figure 8] This figure shows a first example of a screen displayed on a display device by the output control unit according to the first embodiment. [Figure 9] This figure shows a second example of a screen displayed on a display device by the output control unit according to the first embodiment. [Figure 10] This figure shows a third example of a screen displayed on a display device by the output control unit according to the first embodiment. [Figure 11] This figure shows an example of a screen displayed on a display device by the output control unit according to Modification 3. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described below with reference to the drawings. The embodiments described below are illustrative and do not limit the invention. Not all features and combinations thereof in the embodiments of this disclosure are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and redundant descriptions may be omitted.

[0010] The working machine 100 according to the embodiment of this disclosure is a shovel. The working machine 100 may be a machine other than a shovel, such as a crane, an asphalt finisher, or a forklift. In the illustrated example, the shovel as the working machine 100 is an excavator equipped with a bucket 6 as an end attachment, but it may be an applied machine such as a forestry machine equipped with an end attachment other than the bucket 6. Furthermore, it may be a crawler crane equipped with a lower traveling body, an upper rotating body, and an attachment provided on the upper rotating body.

[0011] (First Embodiment) First, with reference to Figure 1, an overview of the remote control system SYS according to the first embodiment will be described. Figure 1 is a schematic diagram showing an example of the remote control system SYS according to the first embodiment.

[0012] <Equipment that constitutes a remote control system> As shown in FIG. 1, a remote operation system SYS (an example of a control system for a work machine) according to the first embodiment includes a work machine 100 and a remote operation room RC.

[0013] The work machine 100 and the remote operation room RC are connected so as to be able to transmit and receive data via a communication line NW.

[0014] The work machine 100 enables transmission and reception of data to and from a device (for example, the remote operation room RC) connected to the communication line NW.

[0015] The work machine 100 is present at the work site. Thus, in this embodiment, a plurality of types of devices are provided at the work site. And the work machine 100 can transmit information regarding the work site to the remote operation room RC. Thereby, the remote operation room RC can confirm the work site according to the information from the work machine 100. Note that this embodiment does not limit the device for measuring the work site to the work machine 100, and other types of devices such as a fixed-point measurement device, a drone flying over the work site, or an imaging device that can be carried by a user may be used.

[0016] The number of work machines 100 included in the remote operation system SYS may be one or a plurality. Thereby, the remote operation system SYS can provide information regarding the work site to the remote operation room RC through one or a plurality of work machines 100.

[0017] <Configuration example of remote operation room> The remote operation room RC includes a communication device T2, a remote controller 40, an operation device 42, an operation sensor 43, and a display device D1E. Also, an operation seat DS on which an operator OP who remotely operates the work machine 100 sits is installed in the remote operation room RC.

[0018] The communication device (an example of a second communication device) T2 is configured to control communication with a communication device T1 (see FIG. 2) attached to the work machine 100.

[0019] The remote controller 40 is an information processing device that performs various calculations. In this embodiment, the remote controller 40 is composed of a microcomputer including a CPU and memory. The various functions of the remote controller 40 are realized by the CPU executing a program stored in memory.

[0020] The display device D1E displays a screen based on information transmitted from the work machine 100, allowing the operator OP in the remote control room RC to visually check the area around the work machine 100. The display device D1E allows the operator to check the conditions of the work site, including the area around the work machine 100, even though the operator is in the remote control room RC.

[0021] An operating device 42 (an example of an operating unit) is equipped with an operating sensor 43 for detecting the operation of the operating device 42. The operating sensor 43 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle of the operating lever around its pivot axis. The operating sensor 43 may also consist of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operating sensor 43 outputs information regarding the detected operation of the operating device 42 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operating sensor 43 may also be configured to generate an operation signal. In this case, the operating sensor 43 may output the operation signal to the communication device T2 without going through the remote controller 40. This enables remote control of the work machine 100 from the remote control room RC.

[0022] <Example of shovel configuration> Next, with reference to Figure 2, an overview of the work machine 100 according to this embodiment will be described. Figure 2 is a side view of the work machine 100 as a work machine according to the first embodiment.

[0023] In Figure 2, +X represents one direction of the X-axis in the three-dimensional Cartesian coordinate system, and (not shown) -X represents the other direction of the X-axis. +Y represents one direction of the Y-axis in the three-dimensional Cartesian coordinate system, and (not shown) -Y represents the other direction of the Y-axis. +Z represents one direction of the Z-axis in the three-dimensional Cartesian coordinate system, and (not shown) -Z represents the other direction of the Z-axis. In Figure 2, the +X side of the work machine 100 corresponds to the front side of the work machine 100, and the -X side of the work machine 100 corresponds to the rear side of the work machine 100. Also, the +Y side of the work machine 100 corresponds to the left side of the work machine 100, and the -Y side of the work machine 100 corresponds to the right side of the work machine 100. Furthermore, the +Z side of the work machine 100 corresponds to the top side of the work machine 100, and the -Z side of the work machine 100 corresponds to the bottom side of the work machine 100. The same applies to other figures.

[0024] The work machine 100 comprises a lower traveling body 1, an upper rotating body 3 mounted on the lower traveling body 1 so as to be rotatable via a slewing mechanism 2, an attachment AT for performing various tasks, and a driver's cab 10. The driver's cab 10 is also called a cabin or cab. The front side of the work machine 100 (upper rotating body 3) corresponds to the side on which the attachment AT is attached to the upper rotating body 3 when the work machine 100 is viewed from directly above along the slewing axis of the upper rotating body 3. The left, right, and rear sides of the work machine 100 (upper rotating body 3) correspond to the left, right, and rear sides as seen from the perspective of an operator seated in the driver's seat inside the driver's cab 10, respectively.

[0025] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the crawlers 1C include a left crawler and a right crawler. The left crawler is driven by a left-travel hydraulic motor 2ML (see Figure 2), and the right crawler is driven by a right-travel hydraulic motor 2MR (see Figure 2). The left-travel hydraulic motor 2ML is a traveling drive unit that drives the left crawler as the driven part, and can rotate the left crawler. The right-travel hydraulic motor 2MR is a traveling drive unit that drives the right crawler as the driven part, and can rotate the right crawler. Note that the traveling drive units may be electric motors. The work machine 100 according to this embodiment is not limited to a crawler-type excavator having crawlers 1C, but may also be a wheeled excavator with tires.

[0026] A boom 4 is rotatably mounted to the front center of the upper slewing body 3, an arm 5 is rotatably mounted to the tip of the boom 4, and a bucket 6 is rotatably mounted to the tip of the arm 5. In the illustrated example, the boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of attachment AT. The boom 4, arm 5, and bucket 6 are driven by a boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively.

[0027] Bucket 6 is an example of a work tool (end attachment). Bucket 6 is used, for example, for excavation work. Depending on the work content, other work tools may be attached to the tip of arm 5 instead of bucket 6. Other work tools may be other types of buckets, such as large buckets, slope buckets, or dredging buckets. Other work tools may also be types of work tools other than buckets, such as agitators, breakers, grapples, or lifting magnets. The excavation attachment may be provided with a bucket tilt mechanism.

[0028] The slewing hydraulic motor 2A, the left travel hydraulic motor 2ML, the right travel hydraulic motor 2MR, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 are hydraulic actuators driven by hydraulic fluid discharged from a hydraulic pump.

[0029] Furthermore, the work machine 100 may have all or part of its driven parts, such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6, electrically driven. In other words, the work machine 100 may be a hybrid excavator or electric excavator, in which all or part of its driven parts are driven by electric actuators.

[0030] The imaging device S6 is mounted on the upper rotating body 3 and captures images of the area around the work machine 100, acquiring image information representing the area around the work machine 100. In the illustrated example, the imaging device S6 includes a front camera S6F, a left camera S6L, a right camera S6R, and a rear camera S6B.

[0031] The front camera S6F is a camera that captures images in front of the work machine 100 and is mounted on the outside of the operator's cab 10, such as on the roof of the operator's cab 10 or the side of the boom 4. The left camera S6L is a camera that captures images to the left of the work machine 100, the right camera S6R is a camera that captures images to the right of the work machine 100, and the rear camera S6B is a camera that captures images behind the work machine 100. Specifically, the front camera S6F, left camera S6L, right camera S6R, and rear camera S6B are all monocular wide-angle cameras equipped with an image sensor such as a CCD or CMOS, and the information of the captured images is taken up by the controller 30. Alternatively, the images captured by the imaging device S6 may be output to the display device D1 (see Figure 2).

[0032] In the illustrated example, the front camera S6F is mounted on the roof of the driver's cab 10, the left camera S6L is mounted on the upper left end of the upper surface of the upper rotating body 3, the right camera S6R is mounted on the upper right end of the upper surface of the upper rotating body 3, and the rear camera S6B is mounted on the upper rear end of the upper surface of the upper rotating body 3.

[0033] The controller 30 is composed of a computer including, for example, a CPU, a volatile memory device, a non-volatile memory device, and various input / output interfaces. The controller 30 implements various functions, for example, by reading a program from the non-volatile memory device, loading it into the volatile memory device, and having the CPU execute it. In the illustrated example, the controller 30 is configured to implement various functions and control the work machine 100. These functions include, for example, a machine guidance function that guides the operator in manually operating the work machine 100. The functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and objects within the monitoring range around the work machine 100.

[0034] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor that can detect the rotation angle of the boom 4 relative to the upper slewing body 3 (hereinafter referred to as "boom angle") which changes per unit time. The boom angle sensor S1 can detect the angular velocity of the boom 4, which indicates the change in boom angle, and the angular acceleration of the boom 4, which indicates the rate of said change. The boom angle is, for example, at its minimum when the boom 4 is at its lowest position, and increases as the boom 4 is raised.

[0035] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the rotation angle of the arm 5 relative to the boom 4 (hereinafter referred to as "arm angle"). The arm angle sensor S2 can detect the angular velocity of the arm 5, which indicates the change in the arm angle, and the angular acceleration of the arm 5, which indicates the rate of change. The arm angle is, for example, at its minimum when the arm 5 is closed to its shortest extent, and increases as the arm 5 is opened.

[0036] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the rotation angle of the bucket 6 relative to the arm 5 (hereinafter referred to as "bucket angle"). The bucket angle sensor S3 can detect the angular velocity of the bucket 6, which indicates the change in bucket angle, and the angular acceleration of the bucket 6, which indicates the rate of change. The bucket angle is, for example, at its minimum when the bucket 6 is fully closed, and increases as the bucket 6 is opened.

[0037] The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 can be any sensor capable of acquiring the attitude of the attachment (an example of an attitude sensor), and may be an IMU (Inertial Measurement Unit), a 6-axis sensor, a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, a rotary encoder that detects the rotation angle around the connecting pin, a gyro sensor, or a combination of an acceleration sensor and a gyro sensor, respectively. In this embodiment, an example of acquiring the boom angle, arm angle, and bucket angle as attitude is described, but the information indicating attitude is not limited to the boom angle, arm angle, and bucket angle, and may be at least one of the boom angle, arm angle, and bucket angle, or it may be image information that visually captures the attitude of the attachment's AT.

[0038] The detection signals corresponding to the boom angle from the boom angle sensor S1, the detection signals corresponding to the arm angle from the arm angle sensor S2, and the detection signals corresponding to the bucket angle from the bucket angle sensor S3 are input to the controller 30. The detection signals may include angular velocity in addition to angle.

[0039] The machine tilt sensor S4 detects the tilt state of the machine (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. The machine tilt sensor S4 is, for example, attached to the upper rotating body 3 and detects the tilt angle of the work machine 100 (i.e., the upper rotating body 3) around two axes: the longitudinal direction and the lateral direction. The machine tilt sensor S4 may be, for example, an acceleration sensor, a 6-axis sensor, or an IMU. The detection signal corresponding to the tilt angle from the machine tilt sensor S4 is input to the controller 30.

[0040] The rotation sensor S5 outputs information regarding the rotation of the upper rotating body 3. The rotation sensor S5 detects, for example, the rotational angular velocity and rotational angular acceleration of the upper rotating body 3 relative to the lower traveling body 1. The rotation sensor S5 may also detect the rotation angle. The rotation sensor S5 may be, for example, a gyro sensor, a resolver, or a rotary encoder. The detection signals corresponding to the rotation angle, rotational angular velocity, and rotational angular acceleration of the upper rotating body 3 detected by the rotation sensor S5 are input to the controller 30.

[0041] The boom cylinder 7 is equipped with a boom rod pressure sensor S7R and a boom bottom pressure sensor S7B. The arm cylinder 8 is equipped with an arm rod pressure sensor S8R and an arm bottom pressure sensor S8B. The bucket cylinder 9 is equipped with a bucket rod pressure sensor S9R and a bucket bottom pressure sensor S9B. The boom rod pressure sensor S7R, boom bottom pressure sensor S7B, arm rod pressure sensor S8R, arm bottom pressure sensor S8B, bucket rod pressure sensor S9R, and bucket bottom pressure sensor S9B are collectively referred to as "cylinder pressure sensors".

[0042] The boom rod pressure sensor S7R detects the pressure in the rod-side oil chamber of the boom cylinder 7 (hereinafter referred to as "boom rod pressure"), and the boom bottom pressure sensor S7B detects the pressure in the bottom-side oil chamber of the boom cylinder 7 (hereinafter referred to as "boom bottom pressure"). The arm rod pressure sensor S8R detects the pressure in the rod-side oil chamber of the arm cylinder 8 (hereinafter referred to as "arm rod pressure"), and the arm bottom pressure sensor S8B detects the pressure in the bottom-side oil chamber of the arm cylinder 8 (hereinafter referred to as "arm bottom pressure"). The bucket rod pressure sensor S9R detects the pressure in the rod-side oil chamber of the bucket cylinder 9 (hereinafter referred to as "bucket rod pressure"), and the bucket bottom pressure sensor S9B detects the pressure in the bottom-side oil chamber of the bucket cylinder 9 (hereinafter referred to as "bucket bottom pressure").

[0043] In this embodiment, the boom rod pressure sensor S7R and the boom bottom pressure sensor S7B are used as sensors to acquire cylinder pressure (an example of information) for deriving torque related to boom 4. However, this embodiment does not limit the sensors for acquiring information for deriving torque related to boom 4 to the boom rod pressure sensor S7R and the boom bottom pressure sensor S7B. For example, an angular acceleration sensor (or angular velocity sensor) provided around the boom foot pin, or a strain gauge for detecting force generated on boom 4 may be used.

[0044] The positioning device PS measures the position of the upper rotating body 3. The positioning device PS is, for example, a GNSS (Global Navigation Satellite System) compass and detects the position and orientation of the upper rotating body 3. The detection signals corresponding to the position and orientation of the upper rotating body 3 are received by the controller 30. The function of detecting the orientation of the upper rotating body 3 may be realized by an orientation sensor attached to the upper rotating body 3. In this embodiment, the positioning device PS measures the current position of the work machine 100 in a globally identifiable reference coordinate system.

[0045] A reference coordinate system is, for example, the World Geodetic System, which can determine a location on Earth. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with its origin at the Earth's center of mass, the X-axis pointing in the direction of the intersection of the Greenwich Meridian and the equator, the Y-axis pointing in the direction of 90 degrees east longitude, and the Z-axis pointing in the direction of the North Pole.

[0046] The operator's cab 10 is a compartment where the operator sits and is located on the front left side of the upper rotating body 3. However, the operator's cab 10 may be omitted if the work machine 100 is remotely controlled or if the work machine 100 operates by fully automatic operation.

[0047] Communication device T1 (an example of a first communication device) communicates with external devices through a communication network including a mobile communication network, a satellite communication network, or the Internet network. Communication device T1 is, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0048] The work machine 100 operates actuators in response to the operation of the operator seated in the cab 10, driving the driven parts such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6.

[0049] Alternatively, the work machine 100 may be configured to be remotely controlled from outside the work machine 100. When the work machine 100 is remotely controlled, the inside of the operator's cab 10 may be unoccupied.

[0050] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operator's actions. This enables the work machine 100 to automatically operate at least a portion of the driven parts, such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, that is, to achieve a so-called "machine control function".

[0051] Figure 3 is a schematic diagram showing an example of the configuration of the work machine 100. In Figure 3, the mechanical power transmission system, hydraulic fluid line, pilot line, and electrical control system are indicated by double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0052] The drive system of the work machine 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 also includes hydraulic actuators such as a slewing hydraulic motor 2A, a left travel hydraulic motor 2ML, a right travel hydraulic motor 2MR, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0053] The engine 11 is an example of a power source for the work machine 100, and is mounted, for example, at the rear of the upper rotating body 3. The power source for the work machine 100 may also be a combination of a battery or fuel cell and an electric motor. Specifically, the engine 11 rotates at a constant speed at a preset target rotational speed under direct or indirect control by the controller 30, driving the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 may also be a gasoline engine or a hydrogen engine, etc.

[0054] The regulator 13 controls the discharge rate of the main pump 14. For example, the regulator 13 controls the discharge rate of the main pump 14 by adjusting the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0055] The main pump 14, for example, is mounted at the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic fluid to the control valve unit 17 through the hydraulic fluid line. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump.

[0056] The control valve unit 17 is one of the hydraulic control devices that control the hydraulic system in the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171 to 176. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control the flow rate of hydraulic fluid flowing from the main pump 14 to the hydraulic actuators, and the flow rate of hydraulic fluid flowing from the hydraulic actuators to the hydraulic fluid tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-travel hydraulic motor 2ML, a right-travel hydraulic motor 2MR, and a slewing hydraulic motor 2A. Specifically, control valve 171 corresponds to the left-travel hydraulic motor 2ML, control valve 172 corresponds to the right-travel hydraulic motor 2MR, and control valve 173 corresponds to the slewing hydraulic motor 2A. Furthermore, control valve 174 corresponds to bucket cylinder 9, control valve 175 corresponds to boom cylinder 7, and control valve 176 corresponds to arm cylinder 8.

[0057] The pilot pump 15 is an example of a pilot pressure generating device and is configured to supply hydraulic fluid to a hydraulic control device via a pilot line. In the illustrated example, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pressure generating device may be implemented by the main pump 14. That is, the main pump 14 may have the function of supplying hydraulic fluid to the control valve unit 17 via a hydraulic fluid line, as well as the function of supplying hydraulic fluid to various hydraulic control devices via a pilot line. In this case, the pilot pump 15 may be omitted.

[0058] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the example shown in the figure, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0059] The operating device 26 is a device used by the operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator may be a hydraulic actuator or an electric actuator.

[0060] The operation sensor 29 is configured to detect the operator's actions using the operation device 26. In this embodiment, the operation sensor 29 detects the operating direction and amount of the operation device 26 corresponding to each actuator and outputs the detected values ​​to the controller 30. In the illustrated example, the controller 30 can control the opening area of ​​the proportional valve 31 according to the output of the operation sensor 29. The controller 30 then supplies the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the hydraulic fluid supplied to each pilot port (pilot pressure) is, in principle, the pressure corresponding to the operating direction and amount of the operation device 26 corresponding to each hydraulic actuator. Thus, the operation device 26 is configured to supply the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0061] The proportional valve 31, which functions as a control valve for machine control, is located in the pipeline connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow area of ​​the pipeline. In the illustrated example, the proportional valve 31 operates in response to control commands output by the controller 30. Therefore, the controller 30 can adjust the pilot pressure acting on the pilot port of the control valve by the proportional valve 31, independently of the operation of the operating device 26 by the operator.

[0062] This configuration allows the controller 30 to operate the hydraulic actuator corresponding to a specific operating device 26 even when no operation is being performed on that particular operating device 26.

[0063] Furthermore, as shown in Figure 3, the control system of the work machine 100 includes a controller 30, a display device D1, an input device D2, and a communication device T1, etc.

[0064] The display device D1 is located in a place easily visible to a seated operator in the driver's cab 10 and displays various information images under the control of the controller 30. In the illustrated example, the display device D1 is located to the right front of the driver's seat and is connected to the controller 30 via a dedicated line. The display device D1 displays various image information. The display device D1 includes a display screen that displays information such as the working conditions or operating status of the work machine 100. The operator seated in the driver's seat can perform work on the work machine 100 while checking the various information displayed on the display device D1. The display device D1 may also be provided with an input device D2.

[0065] The input device D2 is located within reach of the operator seated in the driver's seat and receives various operation inputs from the operator, outputting signals corresponding to the operation inputs to the controller 30. The input device D2 includes a touch panel mounted on the display of the display device D1 which displays various information images, a knob switch provided at the tip of one or more of the operation levers included in the operation device 26, or a button switch, lever, toggle switch, or rotary dial installed around the display device D1. Signals corresponding to the content of operations on the input device D2 are received by the controller 30.

[0066] The controller 30 is configured to output control commands to the regulator 13 as needed, thereby changing the discharge rate of the main pump 14.

[0067] Furthermore, the controller 30 may be configured to perform control related to a machine guidance function that guides the manual operation of the work machine 100 by the operator through the operating device 26. Alternatively, the controller 30 may be configured to perform control related to a machine control function that automatically assists the manual operation of the work machine 100 by the operator through the operating device 26.

[0068] Furthermore, some of the functions of controller 30 may be implemented by other controllers (control devices). In other words, the functions of controller 30 may be implemented in a manner distributed among multiple controllers. For example, machine guidance functions and machine control functions may be implemented by dedicated controllers (control devices).

[0069] <Block configuration of the remote control system> Figure 4 is a functional block diagram showing an example configuration of the remote control system SYS according to this embodiment. In the example shown in Figure 4, the block configurations of the remote control room RC and the work machine 100, which are included in the remote control system SYS, are shown. The hardware configuration of the work machine 100 will not be explained.

[0070] <Configuration of the Remote Control Room (RC)> The remote control room RC includes a remote controller 40, a communication device T2, an operation sensor 43, an operation device 42, a display device D1E, an input device D2E, and a storage device STE. The communication device T2, the operation sensor 43, and the operation device 42 have been described above, so their explanation is omitted.

[0071] The input device D2E is located within reach of the operator OP seated in the operator's seat DS and receives various operation inputs from the operator, outputting signals corresponding to the operation inputs to the remote controller 40. The input device D2E includes a touch panel mounted on the display of the display device D1E that displays various information images, a knob switch provided at the tip of one or more of the operation levers included in the operation device 26, or a button switch, lever, toggle switch, or rotary dial installed around the operator's seat DS. Signals corresponding to the content of operations on the input device D2E are received by the remote controller 40.

[0072] The storage device STE is, for example, located in a remote control room RC and stores various types of information under the control of a remote controller 40. The storage device STE is, for example, a non-volatile storage medium such as a semiconductor memory. For example, the storage device STE includes a three-dimensional model storage unit STE1.

[0073] Next, the remote control room RC will be described. Figure 5 shows an example of the layout of the remote control room RC. The remote control room RC is equipped with multiple operating devices 42, with the operator's seat DS as the reference point.

[0074] In this embodiment, the display device D1E is a multi-display consisting of six monitors arranged in two vertical rows and three horizontal columns, as shown in Figure 5. Specifically, the display device D1E includes the central monitor D1Ea, the upper monitor D1Eb, the left monitor D1Ec, the right monitor D1Ed, the upper left monitor D1Ee, and the upper right monitor D1Ef.

[0075] <<Excavator Functional Blocks>> Returning to Figure 4, we will now describe each functional block within the controller 30 of the work machine 100. Each functional block within the controller 30 is conceptual and does not necessarily need to be physically configured as shown in the figure. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed in each functional block is realized, all or any part thereof, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The controller 30, by realizing the program, includes an acquisition unit 301, a transmission control unit 302, a reception control unit 303, and an actuator drive unit 304.

[0076] The acquisition unit 301 acquires signals from various detection devices installed on the work machine 100. For example, the acquisition unit 301 acquires the detection results from the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3. The acquisition unit 301 also acquires measurement results such as the position and orientation of the work machine 100 from the positioning device PS.

[0077] Furthermore, the acquisition unit 301 acquires image information from the imaging device S6. In addition, the acquisition unit 301 acquires the cylinder pressure detected by the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B.

[0078] The transmission control unit 302 controls the transmission of various information based on the acquisition results of the acquisition unit 301 to the remote control room RC via the communication device T1.

[0079] For example, the transmission control unit 302 controls the transmission of image information captured by the imaging device S6 to the remote control room RC. The transmission control unit 302 also controls the transmission of position information indicating the position and orientation of the work machine 100 to the remote control room RC. Furthermore, the transmission control unit 302 controls the transmission of pressure information indicating the cylinder pressure detected by the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B to the remote control room RC.

[0080] Furthermore, the transmission control unit 302 controls the transmission of information regarding the posture of the work machine 100, including the attachment AT, to the remote control room RC. This information includes the angle information from the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, as well as the slewing angle information from the slewing sensor S5.

[0081] The receiving control unit 303 controls the reception of various information from the remote control room RC via the communication device T1. For example, the receiving control unit 303 receives operation signals from the remote control room RC to control the operation of the work machine 100.

[0082] The actuator drive unit 304 is configured to drive the actuator mounted on the work machine 100. In this embodiment, the actuator drive unit 304 generates and outputs an operating signal for each of the multiple solenoid valves included in the proportional valve 31 based on an operation command transmitted from the remote control room RC.

[0083] Upon receiving an activation signal, each solenoid valve increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0084] <<Functional Blocks of the Remote Control Room>> This section describes the various functional blocks within the remote controller (an example of a control device) 40 of the remote control room RC. Each functional block within the remote controller 40 is conceptual and does not necessarily need to be physically configured as shown in the diagram. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed by each functional block is realized, in whole or in any part, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The remote controller 40, by implementing the program, includes a receiving control unit 401, an acquisition unit 402, a filtering unit 403, an attitude estimation unit 404, a display screen generation unit 405, an output control unit 406, and a transmission control unit 407.

[0085] Furthermore, the storage device STE connected to the remote controller 40 includes a three-dimensional model storage unit STE1.

[0086] The three-dimensional model storage unit STE1 stores a three-dimensional shape model, which is a three-dimensional model representing the external shape of the work machine 100. The three-dimensional shape model of the work machine 100 is composed of a combination of multiple part models so as to be able to represent the posture of the work machine 100. Therefore, when posture information (e.g., angle information) is input from the work machine 100, the three-dimensional shape model can represent the posture of the work machine 100 based on that information.

[0087] The modeling method for the format of the three-dimensional shape model of the work machine 100 stored in the three-dimensional model memory unit STE1 can be any method, for example, a wireframe model, a surface model, or a solid model. In this embodiment, the surface of the three-dimensional shape model of the work machine 100 is shown with colors that match the surface of the work machine 100.

[0088] In this embodiment, the remote controller 40 uses the three-dimensional shape model stored in the three-dimensional model storage unit STE1 to display the orientation of the attachment AT of the work machine 100, which is derived from the received information, on the display device D1E.

[0089] In this embodiment, data is transmitted and received between the controller 30 and the remote controller 40 via a communication line NW. However, a communication delay occurs when the controller 30 transmits the acquired data to the remote controller 40.

[0090] Therefore, when the remote controller 40 displays the posture of the work machine 100 on the display device D1E based on the information received from the controller 30, there is a discrepancy between the displayed posture and the actual posture of the work machine 100.

[0091] Therefore, the remote controller 40 according to this embodiment displays a predicted image on the display device D1E that shows the posture obtained by operating the attachment AT according to the operation indicated in the operation information, based on the operation information received by the operating device 42 via the operation sensor 43 and the information regarding the posture of the work machine 100 received by the communication device T2, starting from the posture indicated in the information.

[0092] Figure 6 illustrates the concept of remote operation of the work machine 100 by an operator OP from a remote control room RC according to this embodiment. As shown in Figure 6, when the remote controller 40 displays the posture of the work machine 100 according to posture information (e.g., angle information) received from the work machine 100, the image information 1601 will be displayed with a delay due to the communication delay.

[0093] In other words, in this embodiment, the attachment AT of the work machine 100 is already moving in direction 1603 according to the operator OP's operation, compared to the posture shown in the image information 1601.

[0094] Therefore, the posture estimation unit 404 in this embodiment predicts (estimates) the current posture of the attachment AT of the work machine 100 based on the posture of the work machine 100 based on the acquired angle information, and the operation performed by the attachment AT for a delay period. The remote controller 40 then displays image information 1602 on the display device D1E, which shows the result of the prediction that the attachment AT has moved in the direction 1604 according to the operation. In this embodiment, the remote controller 40 can suppress the perceived communication delay for the operator OP by displaying the predicted image.

[0095] In this way, the display device D1E displays a predicted image based on the received angle information, showing what the image would look like if the operation indicated by the operation information were performed for the delay time. This allows the operator OP to recognize the current posture of the work machine 100. In this case, the prediction calculation is based on the assumption that the operation indicated by the operation information has been performed for the delay time, so if the operator OP performs a steep operation, the predicted image will change steeply.

[0096] Therefore, in this embodiment, the filter processing unit 403 applies a low-pass filter to the operation information to suppress abrupt changes in the predicted image and also suppresses the output of operation commands requesting abrupt movements from the remote controller 40. In this embodiment, abrupt movements by the work machine 100 are suppressed, thereby improving safety.

[0097] Furthermore, predictive calculations based on the behavior after the low-pass filter is applied can suppress abrupt changes in the predicted image. By suppressing abrupt changes in the predicted image, the feeling of unnaturalness during operation can be reduced.

[0098] In this embodiment, the cutoff frequency input from the input device D2E is applied to the low-pass filter applied to the operation information. In other words, the operator OP can adjust the degree of suppression of changes by adjusting the cutoff frequency. Therefore, the remote controller 40 according to this embodiment can achieve improved operability.

[0099] Returning to Figure 4, the receiving control unit 401 controls the receiving of various information from the work machine 100 via the communication device T2.

[0100] For example, the receiving control unit 401 controls the receiving of image information captured by the imaging device S6 and position information indicating the position and orientation of the working machine 100 from the working machine 100.

[0101] Furthermore, the receiving control unit 401 controls the reception of information regarding the posture of the work machine 100, including the attachment AT, including the angle information from the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, as well as the slewing angle information from the slewing sensor S5.

[0102] Furthermore, the receiving control unit 401 controls the reception of pressure information indicating the cylinder pressure detected by the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B provided on the attachment AT. In addition, the receiving control unit 401 controls the reception of detection results from various detection devices provided on the work machine 100.

[0103] The acquisition unit 402 acquires operation information indicating the operation received by the operating device 42 from the operator OP via the operation sensor 43. The acquisition unit 402 also acquires input information input from the input device D2E from the operator OP. For example, the acquisition unit 402 acquires from the input device D2E the cutoff frequency which is the boundary between the transition band and the passband of the low-pass filter used in the filter processing unit 403, and the delay time due to the communication delay.

[0104] The filter processing unit 403 filters the operation information acquired by the acquisition unit 402. The filtering process in this embodiment includes, for example, a low-pass filter. The filter processing unit 403 in this embodiment applies a low-pass filter to the operation information acquired by the acquisition unit 402 to generate an operation signal for operating the work machine 100.

[0105] In this embodiment, filtering is performed using a cutoff frequency that serves as the boundary between the transition region and the pass region. For example, the filter processing unit 403 may perform a Fourier transform on operation information that changes over time, remove frequency band signals exceeding the cutoff frequency, and then perform an inverse Fourier transform to return to the original state and generate an operation command. Any method can be used for executing the filtering, not limited to well-known methods.

[0106] The attitude estimation unit 404 predicts (estimates) the attitude of the attachment AT if the operation indicated by the operation command is performed for the delay time, based on the attitude of the attachment AT indicated by the angle information, using the angle information as a basis. This is done based on the angle information of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 received by the receiving control unit 401, the operation command generated by the filtering process performed by the filtering processing unit 403, and the delay time acquired by the acquisition unit 402.

[0107] In this embodiment, an example of predicting (estimating) the attitude of the attachment AT using filtered operation commands is described. However, the method is not limited to predicting (estimating) the attitude using filtered operation commands. For example, the attitude estimation unit 404 may also predict (estimate) the attitude of the attachment AT when the operation indicated in the operation information (which has not been filtered) is performed for a delay time. In this embodiment, the accuracy of estimating the attitude of the attachment AT can be improved by considering the operation indicated in the operation information. Therefore, the operator OP can perform operations while knowing the current attitude of the attachment AT, eliminating the need to perform operations that take communication delays into account, thus reducing the operator's burden.

[0108] The display screen generation unit 405 generates screen information to be displayed on the display device D1E based on the attitude estimation result of the attachment AT by the attitude estimation unit 404 and the information received by the receiving control unit 401.

[0109] For example, the display screen generation unit 405 generates a predicted image representing the three-dimensional shape model of the work machine 100 stored in the three-dimensional model storage unit STE1 with the posture predicted by the posture estimation unit 404. The predicted image is, for example, an image of the three-dimensional shape model of the work machine 100 taken from a viewpoint corresponding to the front camera S6F, but it can be an image taken from any viewpoint.

[0110] In this way, the display screen generation unit 405 generates a predicted image in which a three-dimensional shape model is represented, showing the posture after it has been operated for a delay time (an example of a predetermined time) according to the operation indicated in the operation information, based on the posture indicated by the angle information.

[0111] This embodiment is not limited to a method of predicting (estimating) the posture after delay operation according to the operation information indicated by the operation information, based on the posture indicated by the angle information. For example, the current posture may be estimated by inputting the changes in the received angle information into a trained model. In this way, the display screen generation unit 405 according to this embodiment generates a predicted image adjusted by the posture estimation unit 404 so as to approach the current posture of the attachment AT of the work machine 100. This embodiment allows the operator OP to grasp the current posture of the attachment AT even when there is a communication delay, by displaying a predicted image adjusted to approach the current posture based on the received information (e.g., angle information).

[0112] Furthermore, the display screen generation unit 405 generates screen information that shows the image information captured by the imaging device S6 as information received by the receiving control unit 401. For example, the display screen generation unit 405 generates at least one of the following: image information showing the rear of the work machine 100 captured by the rear camera S6B, image information showing the left side of the work machine 100 captured by the left camera S6L, and image information showing the right side of the work machine 100 captured by the right camera S6R.

[0113] The output control unit 406 then displays the predicted video generated by the display screen generation unit 405, along with the image information received by the reception control unit 401 (captured by the imaging device S6), on the display device D1E simultaneously.

[0114] For example, the output control unit (an example of a display control unit) 406 displays the predicted video generated by the display screen generation unit 405 on the central monitor D1Ea included in the display device D1E. Furthermore, the output control unit 406 outputs (displays) at least one of the following image information: image information showing the area behind the work machine 100 captured by the rear camera S6B, image information showing the area to the left of the work machine 100 captured by the left camera S6L, and image information showing the area to the right of the work machine 100 captured by the right camera S6R, to a monitor other than the central monitor D1Ea among the monitors included in the display device D1E. In this embodiment, by displaying the predicted video and image information simultaneously, the operator OP can recognize the surrounding situation of the work machine 100, thereby improving safety. Note that this embodiment is not limited to the configuration in which the predicted video and image information are displayed simultaneously on the display device D1E; for example, only the predicted video may be displayed on the display device D1E.

[0115] The transmission control unit 407 controls the transmission of various types of information to the remote control room RC. For example, the transmission control unit 407 controls the transmission of operation commands generated by the filter processing unit 403 to the work machine 100.

[0116] The processing procedure performed by the remote controller 40 according to this embodiment will now be described. Figure 7 is a flowchart showing the processing procedure for displaying the posture of the attachment AT of the work machine 100 in the remote controller 40 according to this embodiment.

[0117] First, the acquisition unit 301 reads the initial settings for the delay time and cutoff frequency stored in the memory device STE (S1701). Alternatively, the initial settings for the delay time and cutoff frequency may be information pre-set by the operator OP.

[0118] The acquisition unit 402 acquires operation information from the operating device 42 via the operation sensor 43 (S1702).

[0119] The filter processing unit 403 processes the acquired operation information using a low-pass filter with a set cutoff frequency applied to it, thereby generating an operation command (S1703).

[0120] The transmission control unit 407 transmits the generated operation command to the work machine 100 using the communication device T2 (S1704).

[0121] The receiving control unit 401 receives angle information and the like from the work machine 100 (S1705).

[0122] Based on the generated motion command, delay time, and received angle information, the posture estimation unit 404 predicts the current posture of the work machine 100, including the attachment AT, after it has operated according to the motion command only during the delay time, using the angle information, etc., which is presumed to be experiencing a communication delay (S1706).

[0123] Then, the display screen generation unit 405 generates a predicted video (S1707) that represents a three-dimensional shape model representing the predicted posture of the current work machine 100, based on the prediction results.

[0124] The receiving control unit 401 receives image information captured by the imaging device S6 from the work machine 100 (S1708).

[0125] The output control unit 406 outputs the received image information and the predicted video to the display device D1E (S1709).

[0126] For example, the output control unit 406 outputs image information representing the surrounding conditions of the work machine 100, along with the predicted video, to the display device D1E.

[0127] Furthermore, the output control unit 406 outputs to the display device D1E simultaneously the image information captured by the front camera S6F and a predicted video showing the posture predicted to have occurred by the delay time, in response to an operation from the operator OP. An example of the actual screen will be described later. By comparing the image information with the predicted video showing the posture by the delay time, the operator OP can determine whether the delay time is set appropriately.

[0128] The acquisition unit 402 determines whether it has received an operation from the operator OP via the input device D2E to change settings such as the delay time (S1710). If it determines that it has not received an operation to change settings such as the delay time (S1710: NO), it proceeds to process again from S1702.

[0129] On the other hand, if the acquisition unit 402 determines that it has received an operation from the operator OP via the input device D2E to change settings such as the delay time (S1710: YES), it then receives the setting of the delay time, etc. via the input device D2E (S1711), and processing resumes from S1702.

[0130] In this embodiment, the remote controller 40 can perform the control described above and display a predictive image showing the posture of the work machine 100 according to the settings made by the operator OP.

[0131] The processing procedure shown in Figure 7 is an example and does not limit the processing procedure. For example, the order in which operation information is acquired, angle information and image information are transmitted from the work machine 100 to the remote controller 40, and operation commands are transmitted from the remote controller 40 to the work machine 100 is not limited to the order shown in Figure 7. These processes only need to be performed at predetermined intervals and at appropriate timings depending on the embodiment.

[0132] Figure 8 shows a first example of a screen displayed on the display device D1E by the output control unit 406 according to this embodiment. The screen shown in Figure 8 is displayed, for example, on the central monitor D1Ea (D1E).

[0133] The screen 1801 shown in Figure 8 displays a predicted image including a three-dimensional shape model 1802 representing the predicted posture of the current work machine 100. In addition to displaying the three-dimensional shape model 1802, the predicted image may also have image information captured by the front camera S6F superimposed on it. This allows the operator OP to recognize the positional relationship between the attachment AT and the shape of the soil being worked on.

[0134] Furthermore, the screen shown in Figure 8 displays a predicted time display field 1803 and an option button 1804.

[0135] The predicted time display field 1803 represents the predicted time. The predicted time is the time used to predict the orientation of the three-dimensional shape model 1802 from the received angle information, etc. In this embodiment, the predicted time coincides with the delay time. In other words, in this embodiment, the orientation represented by the received angle information is considered to be the orientation from the past by the delay time. The remote controller 40 then estimates the current orientation by predicting the orientation in the future (which coincides with the delay time) from the orientation represented by the received angle information.

[0136] However, the currently set delay time does not necessarily match the actual delay time that occurs. Therefore, the remote controller 40 according to this embodiment allows the delay time to be set by pressing the option button 1804 as a trigger.

[0137] When the acquisition unit 402 receives a press of the option button 1804 via the input device D2E, the output control unit 406 displays the display field 1805.

[0138] Display area 1805 shows the communication distance selection area 1811, the operation sensitivity selection area 1812, the delay time setting area 1813, the delay time setting slider bar 1814, the operation deviation filter setting area 1815, and the operation deviation filter setting slider bar 1816.

[0139] The communication distance selection field 1811, the delay time setting field 1813, and the delay time setting slider bar 1814 are items for setting the delay time. The acquisition unit 402 changes the delay time setting when it receives a setting for either the delay time setting field 1813 or the delay time setting slider bar 1814.

[0140] The communication distance selection field 1811 is a mode selection field related to delay time. In this embodiment, the remote control room RC is used as the reference point for the location of the work machine 100, and the system can accept a selection from "nearby," "intermediate," and "remote." When the acquisition unit 402 receives a selection from "nearby," "intermediate," and "remote," it sets a delay time corresponding to the selected location (mode).

[0141] The delay time setting field 1813 is a field for receiving a numerical input for the delay time. When the acquisition unit 402 receives a numerical input in the delay time setting field 1813, it sets the delay time using the entered numerical value.

[0142] The delay time setting slider 1814 is a slider used to adjust the delay time. The acquisition unit 402 accepts the operation of moving the slider left or right on the delay time setting slider 1814. When the acquisition unit 402 accepts an adjustment of the value by moving the slider, it sets the delay time with the adjusted value.

[0143] Furthermore, the display area 1805 in this embodiment allows the filter processing unit 403 to accept the setting of an operation delay filter (cutoff frequency) for adjusting the effect of the filter.

[0144] The operation sensitivity selection field 1812, the operation delay filter setting field 1815, and the operation delay filter setting slider bar 1816 are items for setting the operation delay filter (cutoff frequency). The acquisition unit 402 changes the cutoff frequency setting when it receives a setting for either the operation delay filter setting field 1815 or the operation delay filter setting slider bar 1816.

[0145] The operation sensitivity selection field 1812 is a mode selection field related to the cutoff frequency. In this embodiment, in order to adjust the effect of the filter on the operation information, it is possible to select one of "insensitive," "medium," and "sensitive." When the acquisition unit 402 receives a selection from "insensitive," "medium," and "sensitive," it sets the cutoff frequency corresponding to the selected sensitivity (mode).

[0146] The operation delay filter setting field 1815 is a field for receiving a numerical input for the cutoff frequency. When the acquisition unit 402 receives a numerical input in the operation delay filter setting field 1815, it sets the cutoff frequency using the input value.

[0147] The operation delay filter setting slider bar 1816 is a slider bar for adjusting the operation delay filter (cutoff frequency). The acquisition unit 402 accepts the operation delay filter setting slider bar 1816 by moving the slider left or right. When the acquisition unit 402 accepts an adjustment of the value by moving the slider, it sets the cutoff frequency with the adjusted value.

[0148] In this embodiment, the operator OP can refer to the screen shown in Figure 8 to set one or more of the delay time and cutoff frequency, and then view the three-dimensional shape model 1802 based on the set result.

[0149] Figure 9 shows a second example of a screen displayed on the display device D1E by the output control unit 406 according to this embodiment. The screen shown in Figure 9 is displayed, for example, on the central monitor D1Ea (D1E).

[0150] The screen 1901 shown in Figure 9 displays a prediction video showing a three-dimensional shape model 1902 representing the predicted posture of the current work machine 100, a prediction time display field 1903, and option buttons 1904.

[0151] The screen 1901 shown in Figure 9 also displays an operation check area 1905. The operation check area 1905 is a display area for verifying whether the currently set delay time matches the actual delay time.

[0152] In the operation check area 1905, the currently received image information and an image representing the three-dimensional shape model predicted a certain delay time ago are superimposed. In other words, in the operation check area 1905, the display area 1911 of the attachment AT represented in the image information and the display area 1912 of the three-dimensional shape model of the attachment (which is a past predicted image) are superimposed.

[0153] If the communication between the remote control room RC and the work machine 100 is delayed by the set delay time, the display area 1911 of the attachment AT shown in the currently received image information and the display area 1912 of the three-dimensional shape model of the attachment predicted by the delay time earlier are considered to be approximately identical. Conversely, if the display area of ​​the attachment AT shown in the currently received image information and the display area of ​​the three-dimensional shape model of the attachment predicted by the delay time earlier are not approximately identical, in other words, if the display area of ​​one attachment is delayed (or operating faster) than the display area of ​​the other attachment, then the currently set delay time is considered to be incorrect.

[0154] In this case, the operator (OP) presses option button 1904 to perform an operation (input) to change the delay time. The screen displayed when option button 1904 is pressed is the same as the screen shown in Figure 8, so its explanation is omitted.

[0155] Furthermore, in the operation check area 1905 of the screen shown in Figure 9, the display area is small, making it difficult to determine the extent of the discrepancy between the display area 1911 of the attachment AT represented in the image information and the display area 1912 of the three-dimensional shape model of the attachment. Therefore, in this embodiment, the displayed screen can be switched according to the operation (input) from the operator OP.

[0156] Figure 10 shows a third example of a screen displayed on the display device D1E by the output control unit 406 according to this embodiment. The screen shown in Figure 10 is displayed, for example, on the central monitor D1Ea (D1E).

[0157] The screen 2001 shown in Figure 10 is an enlarged view of the operation check area 1905 shown in Figure 9, and displays a display field 2002 labeled "Operation Check" and an option button 2003. Like the operation check area 1905, screen 2001 is a screen used to check whether the delay time is different from the actual delay time.

[0158] The screen 2001 shown in Figure 10 displays the currently received image information and an image representing the three-dimensional shape model predicted a delay time earlier, superimposed on each other. As a result, the display area 2011 of the attachment AT shown in the image information and the three-dimensional shape model 2012 of the attachment AT are displayed superimposed on each other.

[0159] If the display area of ​​the attachment AT shown in the currently received image information does not closely match the display area of ​​the three-dimensional shape model of the attachment AT predicted a delay time earlier, in other words, if the display area of ​​one attachment AT is lagging behind (or operating faster than) the display area of ​​the other attachment AT, the operator OP presses option button 2003 to input a value to change the delay time.

[0160] In the screen 2001 shown in Figure 10, when the option button 2003 is pressed, a display area 1805 is displayed, similar to Figure 8, showing the communication distance selection field 1811, the operation sensitivity selection field 1812, the delay time setting field 1813, the delay time setting slider bar 1814, the operation deviation filter setting field 1815, and the operation deviation filter setting slider bar 1816. The operator OP can then make detailed adjustments to the delay time by operating the delay time setting field 1813 or the delay time setting slider bar 1814.

[0161] In other words, the operator (OP) can adjust the delay time by referring to screen 2001, while checking the degree of shift in the display area of ​​the attachment AT due to the set delay time. Therefore, by setting an appropriate delay time, the accuracy of the predicted image displayed on the screen can be improved. The operator (OP) can also operate the device while being aware of the current orientation of the attachment AT, thus improving the convenience of operation.

[0162] As described above, the display device D1E according to this embodiment superimposes the image information and a past predicted image, which was predicted a delay time earlier as the posture at the time the image information was captured, in order to recognize the degree of agreement between the current posture of the attachment AT and the predicted image based on the received image information. Note that this embodiment is not limited to superimposed display; it is sufficient if the image information and the past predicted image are displayed simultaneously in a way that allows the operator OP to compare them. By referring to the screen, the operator OP can determine whether the delay time is set appropriately. If the operator OP recognizes from the screen that the posture and the predicted image do not match, they adjust the delay time. By adjusting the delay time, the accuracy of predicting the current posture of the attachment AT can be improved. Therefore, the operator OP can operate the device after understanding the current posture.

[0163] This embodiment describes an example of displaying information that allows recognition of the degree of agreement between the current orientation of the attachment AT and the predicted image, specifically the case of simultaneously displaying image information and past predicted images. However, this embodiment does not limit the display of information that allows recognition of the degree of agreement between the current orientation of the attachment AT and the predicted image to the simultaneous display of image information and past predicted images; other information may also be used.

[0164] (Second embodiment) The above-described embodiment illustrates an example of generating a predicted image that approaches the current posture of the attachment AT by estimating the posture using a delay time set by the operator OP. However, the above-described embodiment is not limited to the posture estimation method using a delay time set by the operator OP, and other methods may be used. The second embodiment describes an example in which the delay time and cutoff frequency are set automatically.

[0165] In the second embodiment, the filter processing unit 403 sets a cutoff frequency based on the degree of change of the manipulated variable input to the operating device 42. For example, the filter processing unit 403 stores a history of operation information for a predetermined time and determines whether the degree of change of the manipulated variable in the stored operation information is greater than a reference. If the filter processing unit 403 determines that the degree of change of the manipulated variable is greater than the reference, it derives a first cutoff frequency that is greater than the reference and sets the derived first cutoff frequency. On the other hand, if the filter processing unit 403 determines that the degree of change of the manipulated variable is less than or equal to the reference, it derives a second cutoff frequency that is smaller than the first cutoff frequency and sets the derived second cutoff frequency. Note that the specific method for deriving the cutoff frequency may be any method other than the method described above.

[0166] Furthermore, the posture estimation unit 404 according to this embodiment derives a delay time based on the received image information, the received angle information, and the motion command, and sets the derived delay time.

[0167] Any method may be used to derive the delay time. For example, the posture estimation unit 404 according to this embodiment identifies the display area of ​​the three-dimensional shape model corresponding to each of the multiple configurable delay times. The posture estimation unit 404 then compares the display area of ​​the three-dimensional shape model corresponding to each of the multiple delay times with the display area of ​​the attachment AT that appears in the currently received image information. The posture estimation unit 404 then identifies the three-dimensional shape model among the multiple three-dimensional shape model display areas that has the greatest degree of agreement with the display area of ​​the attachment AT that appears in the image information, and derives the delay time corresponding to the identified three-dimensional shape model as the delay time that is actually occurring.

[0168] This embodiment illustrates an example of automatic setting of delay time and cutoff frequency, and is not limited to this method. The delay time and cutoff frequency may be automatically set by other methods.

[0169] (Modified version of the second embodiment) In a modification of the second embodiment, an example is described in which, instead of automatically setting the delay time and cutoff frequency, recommended values ​​for the delay time and cutoff frequency are presented.

[0170] In this modified example, the filter processing unit 403 derives a cutoff frequency based on the degree of change of the manipulated variable input to the operating device 42, using the same method as in the second embodiment. Similarly, the attitude estimation unit 404 derives a delay time based on the received image information, the received angle information, and the operation command.

[0171] The display screen generation unit 405 then generates screen information in which at least one of the derived cutoff frequency and delay time is presented as a recommended value, and the output control unit 406 outputs the screen information. For example, the output control unit 406 may display the screen shown in the display field 1805 in Figure 8, in which at least one of the derived cutoff frequency and delay time is presented as a recommended value.

[0172] This modified version displays the recommended delay time and the currently set delay time in display field 1805, etc., allowing the operator (OP) to recognize the degree of agreement between the current attitude of the attachment (AT) and the predicted image, and prompting them to make appropriate settings.

[0173] (Variation 1) In the predictive video shown in the above-described embodiment, when the three-dimensional shape model of the attachment AT is displayed in Figure 8 or Figure 9, it is difficult to determine how much soil is loaded into the bucket 6 of the attachment AT. Therefore, in this modified example, the three-dimensional shape model of the work machine 100 that corresponds to the bucket 6 of the attachment AT is set to be semi-transparent. The transparency of the three-dimensional shape model corresponding to the bucket 6 of the attachment AT can be set to 50%, for example, but any value is acceptable.

[0174] In this modified display device D1E, a predicted image representing a three-dimensional shape model is superimposed on the image information captured by the front camera S6F. Since the three-dimensional shape model corresponding to bucket 6 is set to be semi-transparent, the display device D1E displays bucket 6 as it appears in the image information, with the three-dimensional shape model of bucket 6 transparent.

[0175] Bucket 6, as shown in the image information, is displayed shifted from the bucket in the three-dimensional shape model by the delay time, but unless soil is being removed, the contents of bucket 6 are not expected to change during the delay time.

[0176] Therefore, the operator (OP) can perceive the amount of material (e.g., soil) in bucket 6 by viewing bucket 6 as it appears in the image information through the three-dimensional shape model of bucket 6.

[0177] (Modification 2) When displaying a predictive image representing a three-dimensional shape model, the method for recognizing the contents of bucket 6 is not limited to the method shown in Modification Example 1.

[0178] For example, the remote controller 40 calculates the weight of the load in the bucket 6 based on pressure information indicating the cylinder pressure detected by the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B. The method for calculating the weight of the load may be the same as in the conventional method.

[0179] The remote controller 40 stores the weight of the load and an image of the load of that weight in association. The image of the load may be a computer graphic or an actual image.

[0180] Then, the display screen generation unit 405 of the remote controller 40 pastes an image of the load, which is associated with the calculated weight, onto the area corresponding to the bucket 6 of the three-dimensional shape model attachment AT. As a result, the bucket 6 of the three-dimensional shape model shown in the predicted image on the display device D1E will show the load.

[0181] Therefore, the operator (OP) can recognize the amount of material (e.g., soil) in bucket 6 by visually inspecting the image of the load represented in the three-dimensional shape model of the bucket shown in the predicted video.

[0182] (Variation 3) When displaying a predictive image representing a three-dimensional shape model, the method for recognizing the contents of bucket 6 is not limited to the method shown in Modification 1 or Modification 2. Therefore, Modification 3 describes a method for recognizing the contents of bucket 6 using image information captured by the front camera S6F.

[0183] Figure 11 shows an example of a screen displayed on the display device D1E by the output control unit 406 according to this modified example. The screen shown in Figure 11 is displayed, for example, on the central monitor D1Ea (D1E).

[0184] The screen 2101 shown in Figure 11 displays a predicted image showing a three-dimensional shape model 2102 representing the predicted posture of the current work machine 100, a predicted time display area 2103, and option buttons 2104.

[0185] The display screen generation unit 405 then identifies the display area where the bucket 6 is visible from the image information captured by the front camera S6F received by the reception control unit 401, and identifies the area in which the cargo loaded in the bucket 6 is visible. In the example shown in Figure 11, the area 2111 in which the cargo is visible is identified from the image information.

[0186] The display screen generation unit 405 then cuts out the identified region 2111 and superimposes the cut-out image onto the display region 2102A in which the buckets of the three-dimensional shape model 2102 are represented.

[0187] Therefore, the operator OP can recognize the amount of cargo (e.g., soil) in bucket 6 by visually inspecting the image of the cargo attached to the bucket of the three-dimensional shape model 2102 of bucket 6. Note that the imaging device for capturing images of the cargo in bucket 6 is not limited to the front camera S6F; an imaging device provided on the attachment AT for capturing images of bucket 6 may also be used.

[0188] <effect> The remote controller 40 according to the above-described embodiment and modified example displays a predicted image showing the posture of the attachment AT after it has been operated according to the operation information indicated by the operation information, based on the posture indicated by the received angle information. This suppresses the occurrence of posture discrepancies caused by communication delays in the predicted image displayed on the display device D1E. Therefore, the operator OP can grasp the current posture of the work machine 100, including the attachment AT. Furthermore, since the operator OP can operate the work machine 100 without considering communication delays, the burden on the operator is reduced and the remote operation of the work machine 100 can be made easier.

[0189] Incidentally, in conventional systems where a low-pass filter is not applied, when the operation information changes, the operation information after the change is assumed to have continued for the duration of the delay, which could potentially amplify the operation resulting from the operation information and present it as a predicted image. In contrast, in the above-described embodiment and modification, when the operation information changes, the filter processing unit 403 applies a low-pass filter to the operation information, thereby suppressing abrupt changes in the operation amount indicated by the operation information.

[0190] Therefore, the embodiments and modifications described above can suppress abrupt changes in the posture of the attachment AT shown in the predicted image. Furthermore, abrupt changes in the operation of the work machine 100 can also be suppressed.

[0191] Preferred embodiments of the present disclosure have been described above. However, the inventions of the present disclosure are not limited to the embodiments described above. Various modifications, substitutions, etc., can be applied to the embodiments described above without departing from the scope of the inventions of the present disclosure. Furthermore, each of the features described with reference to the embodiments described above may be combined as appropriate, as long as they do not contradict each other technically. [Explanation of Symbols]

[0192] 100 working machines 1. Lower running body 2. Swivel mechanism 3. Upper rotating body 4 Boom 5 Arms 6 buckets S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S6 imaging device S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S7R Boom Rod Pressure Sensor S7B and boom bottom pressure sensor S8R Arm Rod Pressure Sensor S8B Arm Bottom Pressure Sensor S9R Bucket Rod Pressure Sensor S9B Bucket Bottom Pressure Sensor STE storage device STE1 Three-Dimensional Model Memory Unit 30 controllers 301 Acquisition Department 302 Transmission Control Unit 303 Receiving Control Unit 304 Actuator drive unit RC Remote Control Room 40 Remote Controllers T2 Communication Device D1E display device 40 Remote Controllers 401 Receiving Control Unit 402 Acquisition Department 403 Filtering section 404 Posture estimation section 405 Display screen generation section 406 Output Control Unit 407 Transmission Control Unit

Claims

1. A work machine comprising: a lower traveling body; an upper rotating body mounted on the lower traveling body so as to be rotatable; an attachment having a boom and an arm; a sensor for detecting the posture of the attachment; and a first communication device for transmitting information indicating the detected posture. A second communication device that receives the aforementioned information, An operating device that receives operating information for operating the aforementioned attachment, A display device that displays a predicted image showing the posture obtained by operating the attachment according to the operation indicated in the operation information, based on the operation information received by the operating device and the posture indicated in the information, A control system for work machines equipped with the following features.

2. The display device displays the predicted image, which has been adjusted by predetermined control to approximate the current posture of the attachment of the work machine. A control system for a work machine according to claim 1.

3. The display device, as a predetermined control, displays the predicted image showing the posture after operating for a predetermined time according to the operation indicated in the operation information, starting from the posture indicated in the information. A control system for a work machine according to claim 2.

4. The predetermined time is set by user input or automatic input. A control system for a work machine according to claim 3.

5. The aforementioned work machine further includes an imaging device that images the area around the work machine, The first communication device transmits the image information captured by the imaging device. The second communication device receives the image information, The display device displays the image information along with the predicted video. A control system for a work machine according to claim 1.

6. The aforementioned work machine further includes an imaging device that captures images in a recognizable manner of the posture of the attachment of the work machine, The first communication device transmits the image information captured by the imaging device. The second communication device receives the image information, The display device displays information that allows recognition of the degree of agreement between the current orientation of the attachment and the predicted image, based on the image information. A control system for a work machine according to claim 1.

7. The display device simultaneously displays the image information and the predicted image, which is estimated to be the posture at the time the image information was captured. A control system for a work machine according to claim 6.

8. A work machine comprising a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable, an attachment having a boom and an arm, a sensor for detecting the posture of the attachment, and a first communication device for transmitting information indicating the detected posture, and a second communication device for receiving the information from the work machine, An acquisition unit that acquires operation information from an operating device for operating the aforementioned attachment, A display control unit performs control to display on a display device a predicted image showing the posture obtained by operating the attachment according to the operation indicated in the operation information, based on the operation information received by the operating device and the posture indicated in the information, A control device for a work machine equipped with the following features.

Images