Remote control system for work machine

Abstract

To reduce the amount of data being sent.SOLUTION: According to one embodiment, a remote control system for a work machine comprises: a work machine having a lower running body, an upper rotating body mounted on the lower running body so as to be freely rotatable, a detection unit that detects an attitude of the work machine including an attachment, an imaging device provided on the upper rotating body that captures images of at least a front of the work machine, and a transmission unit that transmits information related to the attitude detected by the detection unit and images captured by the imaging device; a receiving unit that receives the information related to the attitude and the images; and a display unit that displays the received images and indicates changes in an attitude of the attachment in accordance with the information related to the attitude.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a remote control system for a work machine. [Background technology]

[0002] Conventionally, techniques for remotely controlling work machines have been proposed. In remote control, an operator operates the work machine while referring to video images. In order for the operator to work with the work machine, it is necessary to display the surrounding situation, including the work machine. For this reason, a technique has been proposed in which images captured by multiple imaging devices provided on the work machine are transmitted to a remote control device (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-207244 Summary of the Invention [Problem to be solved by the invention]

[0004] The remote control system described in Patent Document 1 transmits images captured by multiple imaging devices, and therefore requires a wide communication bandwidth. If the amount of data to be transmitted is large, this can cause strain on the communication bandwidth, so there is a trend toward reducing the amount of data to be transmitted.

[0005] In view of the above, the amount of data to be transmitted is reduced while making it possible to display changes in the attitude of a work machine. [Means for solving the problem]

[0006] A remote control system for a work machine according to one aspect of the present invention comprises a work machine having a lower running body, an upper rotating body mounted on the lower running body so as to be freely rotatable, a detection unit that detects the attitude of the work machine including an attachment, an imaging device provided on the upper rotating body that captures images of at least the front of the work machine, and a transmitting device that transmits information related to the attitude detected by the detection unit and images captured by the imaging device, a receiving device that receives the information related to the attitude and the images, and a display device that displays the received images and shows changes in the attitude of the attachment in accordance with the information related to the attitude. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to display changes in the attitude of a work machine, while reducing the amount of data to be transmitted. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating an example of a remote control system according to a first embodiment. [Figure 2] 1 is a side view showing a shovel (excavator) according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a drive control system of the shovel according to the first embodiment. [Figure 4] 1 is a functional block diagram showing an example of the configuration of a remote control system according to a first embodiment. [Figure 5] 1 is a diagram showing an example of the layout of a remote control room according to a first embodiment. FIG. [Figure 6] 5 is a flowchart showing a processing procedure for displaying on a display device in the remote controller according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a display screen displayed on the display device according to the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a display screen displayed on a central monitor according to the first embodiment. [Figure 9]FIG. 3 is a diagram showing an example of a display screen displayed on a central monitor according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The embodiments described below are illustrative and do not limit the invention. Not all features and combinations thereof in the embodiments of the present disclosure are necessarily essential to the invention. In addition, identical or corresponding components in each drawing are designated by identical or corresponding reference numerals, and redundant description may be omitted.

[0010] The work machine 100 according to the embodiment of the present disclosure is a shovel. The work machine 100 may be a machine other than a shovel, such as a crane, an asphalt finisher, or a forklift. In the illustrated example, the shovel serving as the work machine 100 is an excavator equipped with a bucket 6 as an end attachment, but it may also be an applied machine such as a forestry machine equipped with an end attachment other than the bucket 6. Furthermore, the work machine 100 may be a crawler crane equipped with a lower traveling body, an upper rotating body, and an attachment provided on the upper rotating body.

[0011] (First embodiment) First, an overview of the remote control system SYS according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of the remote control system SYS according to the first embodiment.

[0012] <Devices that make up the remote control system> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes a work machine 100, a fixed point measurement device 500, and a remote operation room RC.

[0013] The work machine 100, the fixed point measurement device 500, and the remote control room RC are connected via a communication line NW so that data can be sent and received.

[0014] The work machine 100 and the fixed point measurement device 500 are capable of wireless communication. The work machine 100 is capable of sending and receiving data with equipment (for example, a remote control room RC) connected to the communication line NW. The fixed point measurement device 500 is capable of sending and receiving data with equipment (for example, a remote control room RC) connected to the communication line NW.

[0015] The work machine 100 and the fixed point measurement device 500 are located at the work site where the work machine 100 performs work. In this way, in this embodiment, multiple types of devices are provided at the work site. The work machine 100 and the fixed point measurement device 500 can transmit information about the work site to the remote control room RC. This allows the remote control room RC to check the work site according to the information from the work machine 100 and the fixed point measurement device 500. Note that this embodiment does not limit the devices that measure the work site to the work machine 100 and the fixed point measurement device 500, and may be other types of devices, such as a drone that flies over the work site or an imaging device that can be carried by the user.

[0016] The remote operation system SYS may include one or more work machines 100. This allows the remote operation system SYS to provide information about the work site to the remote control room RC through one or more work machines 100.

[0017] The remote operation system SYS may include one or more fixed-point measurement devices 500. This allows the remote operation system SYS to provide information about the work site to the remote operation room RC through one or more fixed-point measurement devices 500.

[0018] <Example of remote control room configuration> The remote control room RC is equipped with a communication device T2, a remote controller 40, an operation device 42, an operation sensor 43, a storage device 44, and a display device D1E. The remote control room RC also has an operation seat DS where an operator OP who remotely operates the work machine 100 sits.

[0019] The communication device (an example of a receiving device) T2 is configured to control communication with a communication device T1 (see FIG. 2) attached to the work machine 100.

[0020] The remote controller 40 is an information processing device that executes various calculations. In this embodiment, the remote controller 40 is configured as a microcomputer including a CPU and a memory. The various functions of the remote controller 40 are realized by the CPU executing programs stored in the memory.

[0021] The display device D1E displays a screen based on information transmitted from the work machine 100 so that the operator OP in the remote control room RC can visually confirm the surroundings of the work machine 100. The display device D1E enables the operator to confirm the status of the work site, including the surroundings of the work machine 100, even though he is in the remote control room RC.

[0022] An operation sensor 43 is installed in the operation device 42 (an example of an operation unit) to detect the operation content of the operation device 42. The operation sensor 43 is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever about the swing axis. The operation sensor 43 may be configured with other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 43 outputs information related to the detected operation content of the operation device 42 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operation sensor 43 may be configured to generate an operation signal. In this case, the operation sensor 43 may output the operation signal to the communication device T2 without going through the remote controller 40. This makes it possible to remotely operate the work machine 100 from the remote control room RC.

[0023] <Example of excavator configuration> Next, an overview of the work machine 100 according to this embodiment will be described with reference to Figure 2. Figure 2 is a side view of the work machine 100 as a work machine according to the first embodiment.

[0024] In Fig. 2, +X represents one direction of the X axis that constitutes a three-dimensional Cartesian coordinate system, and -X (not shown) represents the other direction of the X axis. +Y represents one direction of the Y axis that constitutes a three-dimensional Cartesian coordinate system, and -Y (not shown) represents the other direction of the Y axis. +Z represents one direction of the Z axis that constitutes a three-dimensional Cartesian coordinate system, and -Z (not shown) represents the other direction of the Z axis. In Fig. 1, the +X side of the work machine 100 corresponds to the front side of the work machine 100, and the -X side of the work machine 100 corresponds to the rear side of the work machine 100. Furthermore, the +Y side of the work machine 100 corresponds to the left side of the work machine 100, and the -Y side of the work machine 100 corresponds to the right side of the work machine 100. Furthermore, the +Z side of the work machine 100 corresponds to the top side of the work machine 100, and the -Z side of the work machine 100 corresponds to the bottom side of the work machine 100. The same applies to the other figures.

[0025] The work machine 100 comprises a lower traveling body 1, an upper rotating body 3 that is mounted on the lower traveling body 1 so as to be rotatable via a rotating mechanism 2, an attachment AT for performing various tasks, and a driver's cab 10. The driver's cab 10 is also called a cabin or cab. The front side of the work machine 100 (upper rotating body 3) corresponds to the side of the upper rotating body 3 to which the attachment AT is attached, when the work machine 100 is viewed from directly above along the rotation axis of the upper rotating body 3. Furthermore, the left, right, and rear sides of the work machine 100 (upper rotating body 3) correspond to the left, right, and rear sides, respectively, as viewed from the perspective of an operator seated in the driver's seat in the driver's cab 10.

[0026] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the crawlers 1C include a left crawler and a right crawler. The left crawler is driven by a left traveling hydraulic motor 2ML (see FIG. 3), and the right crawler is driven by a right traveling hydraulic motor 2MR (see FIG. 3). The left traveling hydraulic motor 2ML is a traveling drive unit that drives the left crawler as a driven part and can rotate the left crawler. The right traveling hydraulic motor 2MR is a traveling drive unit that drives the right crawler as a driven part and can rotate the right crawler. The traveling drive units may be electric motors.

[0027] A boom 4 is rotatably attached to the center of the front of the upper rotating body 3, an arm 5 is rotatably attached to the tip of the boom 4, and a bucket 6 is rotatably attached to the tip of the arm 5. In the illustrated example, the boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of an attachment AT. The boom 4, arm 5, and bucket 6 are driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0028] The bucket 6 is an example of a work tool (end attachment). The bucket 6 is used, for example, for excavation work. Instead of the bucket 6, another work tool may be attached to the end of the arm 5 depending on the type of work, etc. The other work tool may be, for example, a large bucket, a slope bucket, a dredging bucket, or another type of bucket. The other work tool may also be a type of work tool other than a bucket, such as a mixer, a breaker, a grapple, or a lifting magnet. The excavation attachment may be provided with a bucket tilt mechanism.

[0029] The swing hydraulic motor 2A, the left traveling hydraulic motor 2ML, the right traveling hydraulic motor 2MR, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 are hydraulic actuators driven by hydraulic oil discharged from a hydraulic pump.

[0030] Note that in the work machine 100, all or some of the driven parts, such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6, may be electrically driven. In other words, the work machine 100 may be a hybrid excavator, an electric excavator, or the like, in which all or some of the driven parts are driven by electric actuators.

[0031] The imaging device S6 is provided on the upper rotating body 3, and captures images of the periphery of the work machine 100 to obtain image information showing the periphery of the work machine 100. In the illustrated example, the imaging device S6 includes a front camera S6F, a left camera S6L, a right camera S6R, and a rear camera S6B.

[0032] The front camera S6F is a camera that captures images in front of the work machine 100, and is attached to the exterior of the cab 10, such as on the roof of the cab 10 or on the side of the boom 4. The left camera S6L is a camera that captures images to the left of the work machine 100, the right camera S6R is a camera that captures images to the right of the work machine 100, and the rear camera S6B is a camera that captures images to the rear of the work machine 100. Specifically, the front camera S6F, left camera S6L, right camera S6R, and rear camera S6B are all monocular wide-angle cameras equipped with an imaging element such as a CCD or CMOS, and information about the captured images is taken into the controller 30. In addition, images captured by the imaging device S6 may be output to a display device D1 (see FIG. 3).

[0033] In the illustrated example, the front camera S6F is attached to the roof of the driver's cab 10, the left camera S6L is attached to the left end of the upper surface of the upper rotating body 3, the right camera S6R is attached to the right end of the upper surface of the upper rotating body 3, and the rear camera S6B is attached to the rear end of the upper surface of the upper rotating body 3.

[0034] The imaging device S6 may constitute an object detection device that detects objects in the vicinity of the work machine 100. The object detection device may be constituted by a device other than a camera. For example, the object detection device may be a LiDAR. A LiDAR is a device that can measure the distance between a point cloud of one million or more points within a monitoring range and the LiDAR (laser source). The object detection device may also be another device that can measure the distance to an object, such as a stereo camera, a range imaging camera, or a millimeter-wave radar. When a millimeter-wave radar or the like is used as the object detection device, the object detection device may derive the distance and direction of the object by emitting a number of signals (laser light, etc.) toward the object and receiving the reflected signals. Alternatively, the object detection device may be a combination of two or more types of devices. For example, the object detection device may be a combination of an imaging device and a LiDAR, a combination of an imaging device and a millimeter-wave radar, or a combination of an imaging device and a stereo camera.

[0035] The indoor image capturing device S7 is provided inside the cab 10, and captures an image of the area in front of the work machine 100 from inside the cab 10 as the operator's viewpoint, and acquires image information showing the area in front of the work machine 100. Specifically, the indoor image capturing devices S7 are all monocular wide-angle cameras equipped with an image capturing element such as a CCD or CMOS, and output the captured image to the controller 30.

[0036] The interior imaging device S7 may be provided in the cab 10, and may be provided, for example, near the driver's seat so as to be substantially aligned with the driver's viewpoint, or may be provided on the ceiling of the cab 10.

[0037] The controller 30 is an example of a control device, and is configured, for example, by a computer including a CPU, a volatile storage device, a non-volatile storage device, and various input / output interfaces. The controller 30 then realizes various functions, for example, by reading a program from the non-volatile storage device, loading it into the volatile storage device, and having the CPU execute the program. In the illustrated example, the controller 30 is configured to realize various functions to control the work machine 100. The various functions include, for example, a machine guidance function that guides the operator in manually operating the work machine 100. The various functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and an object that is present within a monitoring range around the work machine 100.

[0038] The boom angle sensor S1 detects the boom angle, which is the rotation angle of the boom 4 relative to the upper rotating body 3. The arm angle sensor S2 detects the arm angle, which is the rotation angle of the arm 5 relative to the boom 4. The bucket angle sensor S3 detects the bucket angle, which is the rotation angle of the bucket 6 relative to the arm 5.

[0039] Each of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may be, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder, etc.

[0040] A detection signal corresponding to the boom angle from the boom angle sensor S1, a detection signal corresponding to the arm angle from the arm angle sensor S2, and a detection signal corresponding to the bucket angle from the bucket angle sensor S3 are input to the controller 30. The detection signals may include angular velocity in addition to the angle.

[0041] The machine body tilt sensor S4 detects the tilt state of the machine body (undercarriage 1 or upper rotating body 3) relative to the horizontal plane. The machine body tilt sensor S4 is attached to, for example, the upper rotating body 3, and detects the tilt angle of the work machine 100 (i.e., the upper rotating body 3) around two axes in the forward / backward and left / right directions. The machine body tilt sensor S4 may be, for example, an acceleration sensor, a six-axis sensor, an IMU, or the like. A detection signal corresponding to the tilt angle detected by the machine body tilt sensor S4 is input to the controller 30.

[0042] The rotation sensor S5 outputs information related to the rotation of the upper rotating body 3. The rotation sensor S5 detects, for example, the rotation angular velocity of the upper rotating body 3 relative to the lower running body 1. The rotation sensor S5 may also detect the rotation angle. The rotation sensor S5 may be, for example, a gyro sensor, a resolver, or a rotary encoder. A detection signal corresponding to the rotation angle or rotation angular velocity of the upper rotating body 3 detected by the rotation sensor S5 is input to the controller 30.

[0043] In this embodiment, the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, and swing sensor S5 each function as a detector that detects the attitude of the work machine. Note that this embodiment shows an example of a detector that detects the attitude of the work machine, and any configuration that can detect the attitude of the work machine will suffice.

[0044] The positioning device PS measures the position of the upper rotating body 3. The positioning device PS is, for example, a GNSS (Global Navigation Satellite System) compass, and detects the position and orientation of the upper rotating body 3. A detection signal corresponding to the position and orientation of the upper rotating body 3 is input to the controller 30. The function of detecting the orientation of the upper rotating body 3 may be realized by an orientation sensor attached to the upper rotating body 3. The positioning device PS according to this embodiment measures the current position of the work machine 100 in a reference coordinate system that can identify positions in the world.

[0045] The reference coordinate system is, for example, the World Geodetic System, which can identify positions on Earth. The World Geodetic System is a three-dimensional Cartesian XYZ coordinate system with its origin at the center of gravity of the Earth, its X axis pointing toward the intersection of the Greenwich Meridian and the equator, its Y axis pointing toward 90 degrees east longitude, and its Z axis pointing toward the North Pole.

[0046] The operator's cab 10 is a compartment space in which an operator rides, and is provided on the front left side of the upper rotating body 3. However, when the work machine 100 is remotely controlled or when the work machine 100 operates by fully automatic driving, the operator's cab 10 may be omitted.

[0047] The communication device T1 communicates with an external device through a communication network including a mobile communication network, a satellite communication network, the Internet, etc. The communication device T1 is, for example, a mobile communication module compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0048] In response to operations by an operator seated in the cab 10, the work machine 100 operates actuators to drive driven parts such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6.

[0049] Alternatively, the work machine 100 may be configured so that it can be remotely operated from outside the work machine 100. When the work machine 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0050] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operation performed by the operator. This allows the work machine 100 to realize a function of automatically operating at least some of the driven parts, such as the lower traveling structure 1, upper rotating structure 3, boom 4, arm 5, and bucket 6, i.e., a so-called "machine control function."

[0051] Figure 3 is a diagram that schematically shows an example of the configuration of a work machine 100. In Figure 3, a mechanical power transmission system, a hydraulic oil line, a pilot line, and an electrical control system are indicated by double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0052] The drive system of the work machine 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 also includes hydraulic actuators such as a swing hydraulic motor 2A, a left traveling hydraulic motor 2ML, a right traveling hydraulic motor 2MR, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0053] The engine 11 is an example of a power source for the work machine 100, and is mounted, for example, on the rear of the upper rotating body 3. The power source for the work machine 100 may be a combination of a power source such as a battery or a fuel cell and an electric motor. Specifically, the engine 11 rotates at a constant speed at a preset target rotation speed under direct or indirect control by the controller 30, and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 may also be a gasoline engine, a hydrogen engine, or the like.

[0054] The regulator 13 controls the discharge amount of the main pump 14. For example, the regulator 13 controls the discharge amount of the main pump 14 by adjusting the angle (tilting angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0055] The main pump 14 is mounted on the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic oil to the control valve unit 17 through a hydraulic oil line. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump.

[0056] The control valve unit 17 is one of the hydraulic control devices that controls the hydraulic system of the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171-176. The control valve unit 17 is configured to selectively supply hydraulic oil discharged by the main pump 14 to one or more hydraulic actuators via the control valves 171-176. The control valves 171-176 control the flow rate of hydraulic oil flowing from the main pump 14 to the hydraulic actuators, and the flow rate of hydraulic oil flowing from the hydraulic actuators to a hydraulic oil tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left traveling hydraulic motor 2ML, a right traveling hydraulic motor 2MR, and a swing hydraulic motor 2A. Specifically, the control valve 171 corresponds to the left traveling hydraulic motor 2ML, the control valve 172 corresponds to the right traveling hydraulic motor 2MR, and the control valve 173 corresponds to the swing hydraulic motor 2A. Furthermore, the control valve 174 corresponds to the bucket cylinder 9 , the control valve 175 corresponds to the boom cylinder 7 , and the control valve 176 corresponds to the arm cylinder 8 .

[0057] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to be able to supply hydraulic oil to hydraulic control devices via a pilot line. In the illustrated example, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generating device may also be realized by the main pump 14. That is, the main pump 14 may have a function to supply hydraulic oil to various hydraulic control devices via pilot lines, in addition to a function to supply hydraulic oil to the control valve unit 17 via a hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0058] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the example shown in the figure, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0059] The operating device 26 is a device used by an operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator may be a hydraulic actuator or an electric actuator.

[0060] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each actuator and outputs the detected value to the controller 30. In the illustrated example, the controller 30 can control the opening area of ​​the proportional valve 31 in accordance with the output of the operation sensor 29. The controller 30 then supplies the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17. The pressure of the hydraulic oil supplied to each pilot port (pilot pressure) is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to supply the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17.

[0061] Proportional valve 31, which functions as a control valve for machine control, is disposed in a pipe connecting pilot pump 15 and a pilot port of a control valve in control valve unit 17, and is configured so that the flow path area of ​​that pipe can be changed. In the illustrated example, proportional valve 31 operates in response to a control command output by controller 30. Therefore, controller 30 can adjust the pilot pressure acting on the pilot port of the control valve using proportional valve 31, regardless of the operation of operating device 26 by the operator.

[0062] With this configuration, the controller 30 can operate the hydraulic actuator corresponding to a specific operating device 26 even when the specific operating device 26 is not being operated.

[0063] As shown in FIG. 3, the control system of the work machine 100 includes a controller 30, a display device D1, a communication device T1, and the like.

[0064] The controller 30 is configured to output a control command to the regulator 13 as necessary, thereby changing the discharge rate of the main pump 14.

[0065] Furthermore, the controller 30 may be configured to perform control relating to a machine guidance function that guides (provides guidance for) the manual operation of the work machine 100 by the operator via the operation device 26. Furthermore, the controller 30 may be configured to perform control relating to a machine control function that automatically assists the manual operation of the work machine 100 by the operator via the operation device 26.

[0066] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers. For example, the machine guidance function and the machine control function may be realized by a dedicated controller (control device).

[0067] <Block diagram of remote control system> Fig. 4 is a functional block diagram showing an example configuration of the remote operation system SYS according to this embodiment. The example shown in Fig. 4 shows the block configurations of the remote operation room RC, fixed point measurement device 500, and work machine 100 included in the remote operation system SYS. Note that a description of the hardware configurations of the work machine 100 and fixed point measurement device 500 will be omitted.

[0068] <Configuration of the remote control room RC> The remote control room RC includes a remote controller 40, a communication device T2, an operation sensor 43, an operation device 42, and a display device D1E. The communication device T2, the operation sensor 43, and the operation device 42 have been described above, so a description thereof will be omitted.

[0069] Next, the remote control room RC will be described. Fig. 5 is a diagram showing an example of the layout of the remote control room RC. In the remote control room RC, a plurality of operating devices 42 are provided with respect to the operating seat DS.

[0070] In this embodiment, the display device D1E is a multi-display consisting of six monitors arranged in two vertical rows and three horizontal columns, as shown in Fig. 5. Specifically, the display device D1E includes a center monitor D1Ea, an upper monitor D1Eb, a left monitor D1Ec, a right monitor D1Ed, an upper-left monitor D1Ee, and an upper-right monitor D1Ef.

[0071] <<Excavator function blocks>> Returning to Figure 4, each functional block within the controller 30 of the work machine 100 will be described. Each functional block within the controller 30 is conceptual and does not necessarily have to be physically configured as shown in the figure. All or some of the functional blocks can be configured in any unit by functionally or physically distributing or integrating them. All or any part of the processing functions performed by each functional block are realized by a program executed by a CPU. Alternatively, each functional block may be realized as hardware using wired logic. By realizing the program, the controller 30 is provided with an acquisition unit 301, an identification unit 402, a transmission control unit 302, a reception control unit 303, and an actuator drive unit 304.

[0072] Returning to Fig. 4, the acquisition unit 301 acquires signals from various detection devices provided on the work machine 100. For example, the acquisition unit 301 acquires the detection results of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3. The acquisition unit 301 also acquires measurement results such as the position and orientation of the work machine 100 from the positioning device PS.

[0073] The acquisition unit 301 also acquires image information from the image capture device S6. Furthermore, the acquisition unit 301 also acquires image information of an image of the front view captured from inside the driver's cab 10 by the interior image capture device S7.

[0074] The transmission control unit 302 controls the transmission of various information based on the results of acquisition by the acquisition unit 301 to the remote control room RC via the communication device (an example of a transmission device) T1. For example, the transmission control unit 302 controls the transmission of image information captured by the indoor image capture device S7 and position information indicating the position and orientation of the work machine 100 to the remote control room RC. Note that if the image information captured by the indoor image capture device S7 includes the right side of the cab 10, where the boom 4 is located, in its imaging range, the transmission control unit 302 deletes the imaging range on the right side from the image information before transmitting the image information.

[0075] Furthermore, the transmission control unit 302 performs control to transmit image information captured by the imaging device S6 to the remote control room RC in response to a transmission request from the remote control room RC.

[0076] Furthermore, the transmission control unit 302 controls the transmission of angle information from the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3, as well as rotation angle information from the rotation sensor S5, to the remote control room RC as information regarding the posture of the work machine 100 including the attachment AT.

[0077] In this manner, in this embodiment, at least angle information is transmitted as information relating to the attitude of the attachment AT of the work machine 100. Note that in this embodiment, angular velocity information may be transmitted in addition to angle information as information relating to the attitude of the attachment AT of the work machine 100.

[0078] In conventional remote operation of a work machine, the work machine transmits an image of the right side, where the attachment is located, taken from inside the cab to the remote control room. When such an image is displayed, the operator in the remote control room can understand the posture of the attachment (for example, the boom of an excavator) by referring to the image, but it is difficult to understand objects that exist outside the attachment (for example, the boom of an excavator). However, when an image of the right side taken from inside the cab is not displayed, the operator in the remote control room cannot understand the posture of the attachment.

[0079] In this way, conventionally, in order for an operator in a remote control room to understand the posture of the work machine, the work machine would capture an image of its own posture from inside the cab, transmit the captured image to the remote control room, and the remote control room would display the received image. However, when the work machine transmits such an image to the remote control room, there is a possibility that the communication bandwidth will be congested.

[0080] Therefore, the controller 30 according to this embodiment transmits information relating to the attitude of the work machine 100 detected by various detection devices (for example, boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3) provided on the work machine 100, and the display device D1E in the remote control room RC displays changes in the attitude of the attachment AT in accordance with the received attitude information. The operator OP in the remote control room RC can recognize the attitude of the attachment AT of the work machine 100 by referring to the display device D1E. The controller 30 also transmits image information captured by the indoor imaging device S7 to the remote control room RC, and the display device D1E in the remote control room RC displays the received image information. This allows the operator OP in the remote control room RC to check the situation around the work machine 100 from a perspective similar to that from inside the cab 10. Therefore, the operator OP can grasp the attitude of the work machine 100 and the situation around the work machine 100 and operate the work machine 100 with the same feeling as if he or she were actually present at the work machine 100.

[0081] The reception control unit 303 controls the reception of various information from the remote control room RC via the communication device T1. For example, the reception control unit 303 receives an operation signal for controlling the operation of the work machine 100 from the remote control room RC.

[0082] The actuator driving unit 304 is configured to drive the actuators mounted on the work machine 100. In this embodiment, the actuator driving unit 304 generates and outputs actuation signals for each of the multiple solenoid valves included in the proportional valve 31, based on an operation signal transmitted from the remote control room RC.

[0083] Each solenoid valve that receives the actuation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed that corresponds to the stroke amount of the control valve.

[0084] <<Remote Control Room Function Blocks>> Each functional block in the remote controller (an example of a control unit) 40 of the remote control room RC will be described. Each functional block in the remote controller 40 is conceptual and does not necessarily have to be physically configured as shown in the figure. All or part of each functional block can be functionally or physically distributed or integrated in any unit. All or any part of the processing functions performed by each functional block are realized by a program executed by a CPU. Alternatively, each functional block may be realized as hardware using wired logic. By realizing the program, the remote controller 40 includes a reception control unit 401, an identification unit 402, a display control unit 403, a determination unit 404, a signal generation unit 405, and a transmission control unit 406.

[0085] Furthermore, a storage device 44 connected to the remote controller 40 stores three-dimensional shape models of the boom 4, arm 5, and bucket 6 of the work machine 100. The three-dimensional shape models of the boom 4, arm 5, and bucket 6 are created based on the actual sizes of the boom 4, arm 5, and bucket 6. Therefore, the current posture of the attachment AT can be recognized from the three-dimensional shape models of the boom 4, arm 5, and bucket 6 and the current angle information of the boom 4, arm 5, and bucket 6.

[0086] The three-dimensional shape models of the boom 4, the arm 5, and the bucket 6 may be manually registered in the storage device 44 connected to the remote controller 40 by an operator OP or the like. Furthermore, the remote controller 40 may generate the three-dimensional shape models based on image information captured by the imaging device S6.

[0087] In this embodiment, a case will be described in which the remote controller 40 in the remote operation room RC is a control unit for a display device (an example of a display unit) D1E. In other words, the configuration in the remote operation room RC functions as a display device for supporting the remote operation of the work machine 100.

[0088] The reception control section 401 performs control for receiving various information from the work machine 100 via the communication device T2.

[0089] For example, the reception control unit 401 controls the reception of image information captured by the indoor image capture device S7 and position information indicating the position and orientation of the work machine 100 from the work machine 100. The reception control unit 401 also controls the reception of detection results from various detection devices provided on the work machine 100.

[0090] The reception control unit 401 also controls the reception of image information captured by the image capturing device S6 in response to a transmission request from the remote control room RC. The transmission request will be described later.

[0091] Furthermore, the reception control unit 401 controls the reception of angle information from the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3, as well as rotation angle information from the rotation sensor S5, as information regarding the posture of the work machine 100 including the attachment AT.

[0092] Furthermore, the reception control unit 401 controls the reception of image information of the work site captured from the fixed point measurement device 500 .

[0093] The identification unit 402 identifies the posture of the work machine 100, including the attachment AT, based on the information received by the reception control unit 401. In this embodiment, the posture of the work machine 100 that is identified includes the posture of the attachment AT of the work machine 100 (for example, the positions of the boom 4, arm 5, and bucket 6) and the orientation of the crawler 1C of the work machine 100.

[0094] The posture of the attachment AT (e.g., the positions of the boom 4, arm 5, and bucket 6) can be determined using a conventional method using the detection results of the angle sensors (boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3) and the three-dimensional shape models of the boom 4, arm 5, and bucket 6 stored in the memory device 44.

[0095] Furthermore, the identifying unit 402 identifies the orientation of the crawler 1C with respect to the upper rotating body 3 based on the received rotation angle information.

[0096] The display control unit 403 controls the display of a three-dimensional shape model of the attachment AT on the display device D1 in accordance with the attitude identified by the identification unit 402, along with image information captured by the indoor image capture device S7. The display control unit 403 also controls the display of information indicating the current status of the work machine 100, based on the detection results of the various detection devices of the work machine 100, received by the reception control unit 401. The display control unit 403 also controls the display of image information of the work site, including the work machine 100, captured by the fixed point measurement device 500, received by the reception control unit 401.

[0097] Furthermore, when the reception control unit 401 receives image information captured by the imaging device S6, the display control unit 403 performs control to display the image information captured by the imaging device S6 as image information showing the periphery of the work machine 100.

[0098] Specifically, the display control unit 403 generates overhead image information showing the periphery of the work machine 100 from a viewpoint above the work machine 100 based on the image information captured by the imaging device S6, and performs control to display the overhead image information. Note that a well-known method may be used for generating the overhead image information, and a description thereof will be omitted. Furthermore, the display control unit 403 performs control to display the image information captured by the imaging device S6 as is, in response to an operation from the operator OP.

[0099] Based on the identified posture, the determination unit 404 determines whether the distance from the tip of the bucket 6 included in the attachment AT to the crawler 1C is shorter than a predetermined threshold. The predetermined threshold may be determined according to the embodiment, and is a distance determined according to the possibility that the tip of the bucket 6 will come into contact with (the track of) the crawler 1C.

[0100] That is, when the determination unit 404 determines that the distance from the tip of the bucket 6 to the crawler 1C is shorter than a predetermined threshold, the display control unit 403 displays a warning that there is a possibility that the tip of the bucket 6 will come into contact with (the track of) the crawler 1C. Note that in this embodiment, the object of contact detection is not limited to the crawler 1C, and the possibility of contact with the cab 10 may also be detected. For example, the determination unit 404 determines whether or not the distance between the tip of the bucket 6 and the cab 10 is shorter than a predetermined threshold.

[0101] The signal generation unit 405 generates an operation signal for controlling the operation of the work machine 100 in accordance with the operation received by the operation sensor 43 .

[0102] The transmission control unit 406 controls the transmission of various types of information to the remote control room RC. For example, the transmission control unit 406 controls the transmission of an operation signal generated by the signal generation unit 405 to the work machine 100. The transmission control unit 406 also controls the transmission of a request to transmit image information captured by the imaging device S6 in accordance with an operation from the operator OP.

[0103] The following describes the processing procedure executed by the remote controller 40 according to this embodiment. Fig. 6 is a flowchart showing the processing procedure for displaying on the display device D1E in the remote controller 40 according to this embodiment.

[0104] First, the reception control unit 401 controls the reception of image information captured by the indoor image capture device S7, angle information from the angle sensors (boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3), and swing angle information from the swing sensor S5 from the work machine 100 (S1601). Furthermore, the reception control unit 401 controls the reception of detection results from the various detection devices of the work machine 100, and also controls the reception of image information captured of the work site from the fixed point measurement device 500.

[0105] The identification unit 402 identifies the posture of the attachment AT of the work machine 100 (e.g., the positions of the boom 4, arm 5, and bucket 6) and the direction of the crawler 1C of the work machine 100 based on the information received by the reception control unit 401 (S1602).

[0106] The display control unit 403 displays the three-dimensional shape model of the attachment AT according to the identified attitude, and also displays the three-dimensional shape model of the crawler 1C (its track) according to the identified orientation by superimposing it on the image information captured by the indoor image capturing device S7 (S1603). The display control unit 403 also displays the image information received from the fixed point measuring device 500.

[0107] The determination unit 404 determines, based on the identified attitude, whether or not the distance from the tip of the bucket 6 included in the attachment AT to the crawler 1C is shorter than a predetermined threshold value (S1604).

[0108] If the determination unit 404 determines that the distance to the crawler 1C is shorter than the predetermined threshold (S1604: YES), the display control unit 403 displays a warning that the toe of the bucket 6 is approaching the crawler 1C (S1605). On the other hand, if the determination unit 404 determines that the distance to the crawler 1C is equal to or greater than the predetermined threshold (S1604: NO), no particular control is performed and the process proceeds to S1606.

[0109] Furthermore, the remote controller 40 determines whether or not an operation to display an overhead image (information) has been accepted (S1606). Note that, although an example of an operation to display an overhead image (information) will be described in the example shown in Fig. 6, an operation to display image information captured by the imaging device S6 as is may also be accepted.

[0110] When it is determined that the operation to display the overhead image (information) has not been accepted (S1606: NO), the remote controller 40 performs the process again from S1601.

[0111] On the other hand, if the remote controller 40 determines that it has received an operation to display an overhead image (information) (S1606: YES), the transmission control unit 406 transmits a request to transmit image information captured by the imaging device S6 to the work machine 100 (S1607).

[0112] Then, the reception control unit 401 receives from the work machine 100 the image information captured by the indoor image capturing device S7, the angle information of each angle sensor, the turning angle information from the turning sensor S5, and the image information captured by the image capturing device S6 (S1608).

[0113] The identification unit 402 identifies the posture of the attachment AT of the work machine 100 (e.g., the positions of the boom 4, arm 5, and bucket 6) and the direction of the crawler 1C of the work machine 100 based on the information received by the reception control unit 401 (S1609).

[0114] The display control unit 403 generates an overhead image (information) showing the surroundings of the work machine 100 based on the image information captured by the imaging device S6 (S1610).

[0115] Then, the display control unit 403 displays a three-dimensional shape model of the attachment AT and an overhead image (information) according to the identified posture, and also displays a three-dimensional shape model of the crawler 1C (its tracks) according to the identified orientation, superimposed on the image information captured by the indoor imaging device S7 (S1611).

[0116] The determination unit 404 determines, based on the identified posture, whether or not the distance from the tip of the bucket 6 included in the attachment AT to the crawler 1C is shorter than a predetermined threshold value (S1612).

[0117] If the determination unit 404 determines that the distance to the crawler 1C is shorter than the predetermined threshold (S1612: YES), the display control unit 403 displays a warning that the toe of the bucket 6 is approaching the crawler 1C (S1613). On the other hand, if the determination unit 404 determines that the distance to the crawler 1C is equal to or greater than the predetermined threshold (S1612: NO), no particular control is performed and the process proceeds to S1614.

[0118] The remote controller 40 determines whether or not an operation to stop the display of the overhead image (information) has been accepted (S1614).

[0119] When it is determined that an operation to stop the display of the overhead image (information) has not been received (S1614: NO), the remote controller 40 performs the process again from S1608.

[0120] On the other hand, if the remote controller 40 determines that an operation to display an overhead image (information) has been accepted (S1614: YES), the transmission control unit 406 transmits a request to stop transmission of image information captured by the imaging device S6 to the work machine 100 (S1615). Thereafter, the processing is repeated from S1601.

[0121] Next, a screen displayed on the display device D1E by the display control unit 403 will be described. Fig. 7 is a diagram showing an example of a display screen displayed on the display device D1E according to this embodiment. In the example screen shown in Fig. 7, image information captured by the imaging device S6 and overhead image information generated based on the image information captured by the imaging device S6 are displayed in accordance with settings input by the operator OP.

[0122] The upper left monitor D1Ee displays overhead image information generated by the display control unit 403. In the center of the overhead image information 1703, an icon 1703a representing the work machine 100 is displayed.

[0123] The left monitor D1Ec displays image information captured by the imaging device S6. For example, image information of the area behind the work machine 100 captured by the rear camera S6B is displayed.

[0124] The upper monitor D1Eb displays image information captured by the fixed point measurement device 500. For example, the fixed point measurement device 500 captures images of the situation around the work site, including the work machine 100. As a result, the upper monitor D1Eb displays the situation around the work machine 100, with the area 1705 showing the work machine 100 at the center.

[0125] However, when the operator OP performs an operation, it may be difficult to understand the position of the bucket 6 of the work machine 100 and the positional relationship between the work object and the work target from the image information captured by the indoor image capturing device S7 and the image information captured by the image capturing device S6. Therefore, in this embodiment, image information captured by the fixed point measurement device 500, which is installed in a different position from the work machine 100, is displayed. This allows the operator OP to recognize the position of the bucket 6 of the work machine 100 and the positional relationship between the work object and the work target by referring to the image information captured by the fixed point measurement device 500, which is a third-party viewpoint. Therefore, it is possible to achieve improved work efficiency.

[0126] The upper right monitor D1Ef displays the detection results of various detection devices of the work machine 100. For example, a date and time display area 1704a, a driving mode display area 1704b, an attachment display area 1704c, a fuel consumption display area 1704d, an engine control status display area 1704e, a coolant temperature display area 1704g, a remaining fuel amount display area 1704h, an RPM level display area 1704i, a urea water remaining amount display area 1704j, a hydraulic oil temperature display area 1704k, a weather display area 1704l, and an engine operating time display area 1704m are displayed.

[0127] The date and time display area 1704a is an area for displaying the current date and time. The driving mode display area 1704b is an area for displaying the current driving mode. The attachment display area 1704c is an area for displaying an image representing the currently attached attachment. The fuel efficiency display area 1704d is an area for displaying fuel efficiency information calculated by the controller 30.

[0128] The engine control status display area 1704e is an area that displays the control status of the engine 11. The coolant temperature display area 1704g is an area that displays the current temperature status of the engine coolant. The remaining fuel amount display area 1704h is an area that displays the remaining amount of fuel stored in the fuel tank.

[0129] The rotation speed level display area 1704i is an area that displays the current level set by a dial (not shown) as an image. A number indicating the selected level is displayed in the rotation speed level display area 1704i. The number "1" displayed in the rotation speed level display area 1704i indicates that the selected rotation speed level is the "first level." The number "n" displayed in the rotation speed level display area 1704i indicates that the selected rotation speed level is the "nth level." "n" is a natural number. When the operator rotates the dial (not shown), the number displayed in the rotation speed level display area 1704i changes.

[0130] The urea water remaining amount display area 1704j is an area that displays an image of the remaining amount of urea water stored in the urea water tank. The hydraulic oil temperature display area 1704k is an area that displays the temperature of the hydraulic oil in the hydraulic oil tank.

[0131] The weather display area 1704l displays the weather in the location where the work machine 100 is located as text. For example, information acquired by the remote controller 40 from an external server that provides weather information based on the position measured by the positioning device PS is displayed. The engine operating time display area 1704m is an area that displays the accumulated operating time of the engine 11.

[0132] On the central monitor D1Ea, three-dimensional shape models 1701 and 1702 of the crawler 1C (the track of the crawler) are displayed in accordance with the orientation identified by the identification unit 402, superimposed on image information captured by the indoor imaging device S7.

[0133] In this way, the central monitor D1Ea of the display device D1E according to this embodiment displays the shape of at least a portion of the undercarriage 1 (for example, three-dimensional shape models 1701, 1702) so that the attitude of the crawler 1C of the undercarriage 1 relative to the upper rotating body 3 can be recognized based on the received information (swing angle information) related to the attitude of the upper rotating body 3. The three-dimensional shape models 1701, 1702 of the crawler 1C (the track of the crawler) indicate "Front" on the front side of the crawler 1C. When the operator OP performs an operation to travel the undercarriage 1, the operator OP can recognize the direction in which the work machine 100 will travel, thereby reducing the operational burden.

[0134] In this embodiment, when the upper rotating body 3 rotates, the change in posture of the crawler 1C is displayed as an animation. That is, in this embodiment, the controller 30 of the work machine 100 continues to transmit information about the posture (for example, the swing angle or swing angular velocity included in the detection signal of the swing sensor S5) at time intervals that allow for animation display. As a result, the remote controller 40 continues to display the change in posture of the crawler 1C as an animation based on the received information about the posture.

[0135] Furthermore, the three-dimensional shape models 1701, 1702 of the crawler 1C (the tracks thereof) may be displayed as outlines only, or may have a predetermined transparency, which allows the operator OP to view the image captured by the indoor image capturing device S7 without being obstructed by the three-dimensional shape models 1701, 1702 of the crawler 1C (the tracks thereof).

[0136] The image information displayed on the central monitor D1Ea shows the situation in front of the work machine 100 as seen from inside the cab 10. In other words, when the operator OP in the operator seat DS looks at the central monitor D1Ea, the operator OP can be given the feeling that he or she is in the cab 10 of the work machine 100.

[0137] Furthermore, the image information displayed on the central monitor D1Ea is image information captured forward from the indoor imaging device S7, and is different from image information captured to the right from the indoor imaging device S7. In other words, the image information captured forward from the indoor imaging device S7 and transmitted by the communication device T1 does not include in its imaging range the right direction (an example of a lateral direction) where the attachment AT is located from the cab 10 of the work machine 100. Therefore, transmission of image information in which the attachment AT is shown in a large size is suppressed, thereby preventing congestion in the communication band.

[0138] Incidentally, when an operator is present in the cab 10, the attachment AT of the work machine 100 can be seen from the window on the right side of the cab 10. This allows the operator to recognize changes in the posture of the attachment AT. Even when operating from the remote control room RC, there is a need for a display that allows the operator to recognize changes in the posture of the attachment AT, just as when operating from the cab 10.

[0139] Therefore, in this embodiment, the display control unit 403 uses the central monitor D1Ea, which displays an image showing the front of the work machine 100, as a reference and displays changes in the posture of the attachment AT on the upper monitor D1Eb, right monitor D1Ed, and upper right monitor D1Ef, which are located in the direction in which the attachment AT is located from the cab of the work machine 100 (in other words, to the right and upward). In this embodiment, changes in the posture of the attachment AT are displayed as animation using three-dimensional shape models 1710, 1711, and 1712.

[0140] In this embodiment, an example is taken in which changes in the posture of the attachment AT are displayed as animation. That is, in this embodiment, the controller 30 of the work machine 100 continues to transmit information about the posture (for example, angles or angular velocities contained in the detection signals of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3) at time intervals that allow for display as animation. As a result, the remote controller 40 continues to display changes in the posture of the attachment AT as animation based on the received posture information. Therefore, the operator OP can recognize the current posture of the attachment AT.

[0141] This embodiment describes an example in which changes in the posture of the attachment AT, based on one or more of the angles and angular velocities contained in the detection signals of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, are represented using three-dimensional shape models 1710, 1711, and 1712 of the attachment AT. However, this embodiment is not limited to a method of representing changes in the posture of the attachment AT using three-dimensional shape models 1710, 1711, and 1712 of the attachment AT, and, for example, image information representing the attachment AT may also be used. In other words, any method may be used as long as the changes in the posture of the attachment AT can be recognized.

[0142] Furthermore, as shown in Figure 7, the three-dimensional shape models 1710, 1711, 1712 of the attachment AT displayed on the upper monitor D1Eb, right monitor D1Ed, and upper right monitor D1Ef are represented as if they were viewed from the cab 10 of the work machine 100.

[0143] Furthermore, the three-dimensional shape models 1710, 1711, 1712 of the attachment AT displayed on the upper monitor D1Eb, right monitor D1Ed, and upper right monitor D1Ef are displayed at approximately the same scale as the object appearing in the image displayed on the central monitor D1Ea. Therefore, in this embodiment, the attachment AT appearing on the upper monitor D1Eb, right monitor D1Ed, and upper right monitor D1Ef and the bucket 6 displayed on the central monitor D1Ea are displayed as if they are connected.

[0144] In other words, the three-dimensional shape models 1710, 1711, 1712 of the attachment AT displayed on the screen of the display device D1E are the shapes when viewed from the cab 10 of the work machine 100, and are displayed at approximately the same scale as the objects shown in the image displayed on the central monitor D1Ea, so that the situation is expressed as if the operator were in the cab 10 of the work machine 100. Therefore, the operator OP can intuitively grasp the current posture of the work machine 100 by referring to the information displayed on the display device D1E. Furthermore, the operational burden on the operator OP can be reduced.

[0145] Furthermore, the upper monitor D1Eb displays image information captured by the fixed point measurement device 500. The upper monitor D1Eb then displays the outline of the three-dimensional shape model 1710 of the attachment AT superimposed on the image information (an example of the second image) captured by the fixed point measurement device 500. In this way, because the outline of the three-dimensional shape model 1710 of the attachment AT is superimposed on the image information, the operator OP can recognize both the content displayed as an image and the posture of the attachment AT. This reduces the operational burden on the operator OP.

[0146] In this embodiment, an example will be described in which the contour of the three-dimensional shape model 1710 of the attachment AT is superimposed on image information captured by the fixed point measurement device 500, but the destination on which the contour of the three-dimensional shape model 1710 of the attachment AT is superimposed is not limited to image information captured by the fixed point measurement device 500, and it can be any image information other than image information captured by the indoor image capture device S7. For example, the contour of the three-dimensional shape model 1710 of the attachment AT may be superimposed on the detection results of various detection devices of the work machine 100. Therefore, by referring to the upper monitor D1Eb, the operator OP can confirm the situation around the work machine 100 from an objective perspective and can also auxiliary recognize the posture (angle) of the attachment AT.

[0147] This embodiment shows an example of the display of changes in the posture of the attachment AT, and is not limited to the display mode described above. In other words, the display control unit 403 only needs to be able to display changes in the posture of the attachment AT on the display device D1E in accordance with information about the posture (for example, angles or angular velocities contained in the detection signals of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3), and for example, may display a three-dimensional shape model of the attachment AT from a viewpoint different from that shown in FIG. 7, or may display a three-dimensional shape model of the attachment AT at a scale different from that shown in FIG.

[0148] Fig. 8 is a diagram showing an example of a display screen displayed on the central monitor D1Ea according to this embodiment. In the example screen shown in Fig. 8, a three-dimensional shape model 1801 of the crawler 1C (the track of the crawler) is displayed superimposed on image information captured by the indoor imaging device S7 in accordance with the orientation identified by the identification unit 402. In the example shown in Fig. 8, the three-dimensional shape model 1801 with "Front" on the left side is arranged horizontally. Therefore, when an operation to move forward is performed using the operating device 42, the operator OP can recognize that the upper rotating body 3 will travel leftward.

[0149] As shown in Figures 7 and 8, changes in the posture of the crawler 1C (its track) are displayed in three-dimensional shape models 1701, 1702, and 1801 according to the orientation of the crawler 1C identified based on information regarding the posture of the received work machine 100.

[0150] In this embodiment, changes in the posture of the work machine 100 that are not captured by the image capture device S6 and the indoor image capture device S7 are displayed in a three-dimensional shape model in accordance with the received information about the posture. Therefore, the operator OP can recognize changes in the posture of the work machine 100 and understand what operation to perform. Furthermore, this embodiment can reduce the operational burden on the operator OP.

[0151] 9 is a diagram showing an example of a display screen displayed on the central monitor D1Ea according to this embodiment. In the example screen shown in Fig. 9, three-dimensional shape models 1901 and 1902 of the crawler 1C (the track of the crawler) are displayed superimposed on image information captured by the indoor image capturing device S7 in accordance with the orientation identified by the identifying unit 402.

[0152] Display area 1911 in Figure 9 shows a situation in which the toe of the bucket 6 is moving downward. Because the crawler 1C (the track of the crawler) is not visible, it is difficult for the operator OP to determine whether the toe of the bucket 6 will come into contact with the crawler 1C. Furthermore, if image information that includes the crawler 1C in the imaging range is transmitted from the work machine 100 to the remote controller 40, the amount of data transmitted from the work machine 100 to the remote controller 40 will increase. Therefore, in this embodiment, the transmission of image information that includes the crawler 1C in the imaging range is suppressed, and the determination unit 404 determines whether the distance from the toe of the bucket 6 to the crawler 1C (the track of the crawler) is closer than a predetermined threshold, and the display control unit 403 displays a warning based on the determination result.

[0153] Specifically, the determination unit 404 can identify the position of the tip of the bucket 6, relative to a predetermined position of the work machine 100, from information related to the posture (for example, angles or angular velocities included in the detection signals of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3) and a three-dimensional shape model of the attachment AT. Similarly, the determination unit 404 can identify the area in which the crawler 1C exists, relative to a predetermined position of the work machine 100, from the swing angle information and a three-dimensional shape model of the crawler 1C. Therefore, the determination unit 404 can determine whether the distance between the crawler 1C and the tip of the bucket 6 is shorter than a predetermined threshold.

[0154] In this embodiment, when the determination unit 404 determines that the distance from the tip of the bucket 6 included in the attachment AT to the crawler 1C is shorter than a predetermined threshold, the display control unit 403 displays an exclamation mark 1912 near the display area 1911 of the bucket 6. Furthermore, the display control unit 403 displays a warning such as "The bucket may come into contact with the track. Please operate with caution" on the right monitor D1Ed. The operator OP can recognize that the tip of the bucket and (the track of) the crawler 1C are close to each other, and can therefore refrain from performing an operation that would bring the tip of the bucket and (the track of) the crawler 1C closer together. This makes it possible to achieve improved safety.

[0155] Furthermore, when the determination unit 404 determines that the distance from the tip of the bucket 6 included in the attachment AT to the crawler 1C is shorter than a predetermined threshold, the signal generation unit 405 may generate a signal to stop the operation of the attachment AT, and the transmission control unit 406 may transmit the generated signal. This control can prevent contact between the tip of the bucket 6 and the crawler 1C, thereby improving safety.

[0156] In the work machine 100, it is possible to replace an end attachment such as the bucket 6. When the end attachment is replaced, the determination unit 404 determines whether or not the distance between the tip of the replaced end attachment and the crawler 1C is shorter than a predetermined threshold value. The determination unit 404 may also determine whether or not the distance between the tip of the replaced end attachment and the cab 10 is shorter than a predetermined threshold value.

[0157] To change the three-dimensional shape model in response to a change in the end attachment, for example, the operator OP inputs identification information of the replaced end attachment via the operation device 42 or the like. The remote controller 40 then controls to use the three-dimensional shape model of the end attachment corresponding to the input identification information. Furthermore, the method is not limited to input by the operator OP; the controller 30 may automatically detect the replacement of the end attachment. For example, when the end attachment is replaced, the controller 30 detects that the end attachment has been replaced based on the flow rate setting, etc., and transmits the detection result to the remote controller 40. The remote controller 40 then changes the three-dimensional shape model of the end attachment in accordance with the detection result. Furthermore, the remote controller 40 may detect that the end attachment has been replaced based on image information captured by the imaging device S6, and change the three-dimensional shape model of the end attachment in accordance with the detection result. Furthermore, a serviceman who performs maintenance and inspection of the work machine 100 may directly input the depth, height, width, etc. of the end attachment after the change to the remote controller 40, and the remote controller 40 may display a three-dimensional shape model based on the input depth, height, width, etc. as the end attachment.

[0158] (Variation 1) In the above-described embodiment, an example has been described in which the remote controller 40 determines the posture of the attachment AT based on the detection results of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, and the three-dimensional shape models of the boom 4, the arm 5, and the bucket 6. However, the above-described embodiment is not limited to the method of determining the posture of the attachment AT in the remote controller 40.

[0159] In this modification, the controller 30 determines the posture of the attachment AT based on the detection results of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, and the three-dimensional shape models of the boom 4, the arm 5, and the bucket 6. To this end, the controller 30 and the remote controller 40 each store information relating to the three-dimensional shape models of the boom 4, the arm 5, and the bucket 6.

[0160] Then, the controller 30 transmits information indicating the determination result of the attitude of the attachment AT to the remote controller 40 via the communication device T1. The information indicating the determination result of the attitude of the attachment AT may be, for example, information indicating position coordinates for displaying three-dimensional shape models of each of the boom 4, arm 5, and bucket 6 on the display device D1E.

[0161] In this way, the information regarding posture transmitted by the controller 30 may be other than the detection results of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3, and may be information processed to identify the posture based on the detection results, such as the position coordinates for displaying the three-dimensional shape model of the attachment AT described above.

[0162] The remote controller 40 according to this modification performs display on the display device D1E in the same manner as in the above-described embodiment, based on information indicating the result of identifying the posture of the attachment AT received from the controller 30. Therefore, the same effects as in the above-described embodiment can be obtained, and the processing load on the remote controller 40 can be reduced.

[0163] (Variation 2) In the above-described embodiment and modified examples, an example has been described in which the controller 30 or the remote controller 40 determines the posture of the attachment AT based on the detection results of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, and the three-dimensional shape models of each of the boom 4, the arm 5, and the bucket 6. However, the posture of the attachment AT may be determined by a device other than the controller 30 or the remote controller 40. Therefore, modified example 2 will describe a case in which the determination is performed by a management device.

[0164] The management device is a device for managing the work site, and is connected to the work machine 100, the fixed point measurement device 500, and the remote control room RC via a communication line NW.

[0165] The management device manages information relating to the work machine 100. For example, the management device stores three-dimensional shape models of the boom 4, arm 5, and bucket 6 of the work machine 100.

[0166] The management device manages the information transmitted from the work machine 100 and the remote control room RC as a work history. The information transmitted from the work machine 100 to the management device includes the detection results of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3.

[0167] The management device then identifies the posture of the attachment AT based on the detection results of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3, and the three-dimensional shape models of the boom 4, the arm 5, and the bucket 6. The management device then transmits information indicating the identified posture of the attachment AT to the remote controller 40.

[0168] The remote controller 40 according to this modification performs display on the display device D1E based on the information indicating the result of identifying the posture of the attachment AT received from the management device, in the same manner as in the above-described embodiment and modification. Therefore, the same effects as in the above-described embodiment can be obtained, and the processing load on the remote controller 40 can be reduced.

[0169] <effect> In the above-described embodiment, it is difficult for the operator OP to recognize the posture of the work machine 100 that is not shown on the display device D1E, but if images captured by an imaging device or the like are to be displayed on the display device D1E so that the operator OP can recognize all postures of the work machine 100, the amount of data transmitted from the work machine 100 to the remote controller 40 increases. Therefore, in the above-described embodiment, the controller 30 of the work machine 100 transmits information regarding the posture of the work machine 100 (including the attachment AT) obtained from an angle sensor or the like to the remote controller 40, and the remote controller 40 displays changes in the posture of the work machine 100, including the attachment AT, based on the received posture information.

[0170] Therefore, the remote operation system SYS according to the embodiment described above can reduce the amount of data transmitted. Furthermore, the operator OP can recognize the posture of the work machine 100, including the attachment AT. Furthermore, even in an environment where the communication bandwidth between the work machine 100 and the remote controller 40 is narrow, the operator OP can easily and intuitively recognize the posture of the work machine 100, including the attachment AT, the orientation of the crawler 1C, and other conditions. Furthermore, the posture of the work machine 100, including the attachment AT, is displayed in animation, improving visibility. Furthermore, the operator OP can operate the attachment AT, etc., after understanding the posture of the work machine 100, including the attachment AT, thereby reducing the burden on the operator.

[0171] The preferred embodiments and modifications of the present disclosure have been described above. However, the invention according to the present disclosure is not limited to the above-described embodiments. Various modifications, substitutions, etc. may be applied to the above-described embodiments without departing from the scope of the invention according to the present disclosure. Furthermore, each of the features described with reference to the above-described embodiments may be combined as appropriate unless technically inconsistent. [Explanation of symbols]

[0172] 100 Work Machinery 1 Undercarriage 2. Swivel mechanism 3 Upper rotating body 4. Boom 5 Arm 6 buckets S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S6 imaging device S7 Indoor Imaging Device T1 communications equipment PS Positioning Device 30 Controllers 301 Acquisition Department 302 Transmission control section 303 Reception control section 304 Actuator drive unit 31 Proportional valve RC remote control room T2 communications equipment D1E display device 40 Remote Controller 401 Reception control section 402 Specific section 403 Display control unit 404 Judgment section 405 Signal Generation Unit 406 Transmission control section 42 Operating device 43 Operation Sensor 44 Storage device

Claims

1. a work machine having a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, a detection unit that detects the attitude of the work machine including an attachment, an imaging device that is provided on the upper rotating body and captures images of at least the front of the work machine, and a transmission device that transmits information related to the attitude detected by the detection unit and images captured by the imaging device; a receiving device that receives the posture information and the image; a display device that displays the image and that displays a change in the posture of the attachment according to the posture information; A remote control system for a work machine comprising:

2. the image transmitted by the transmitting device does not include, in an imaging range, a side direction from the driver's seat of the upper rotating body of the work machine to which the attachment is attached; 2. The remote control system for a work machine according to claim 1.

3. the display device displays a change in the attitude of the attachment in a direction from a cab of the work machine to a position where the attachment is located, using the image showing the front of the work machine as a reference.

2. The remote control system for a work machine according to claim 1.

4. The attachment whose posture change is displayed by the display device is expressed as a shape as viewed from the driver's seat of the work machine.

4. The remote control system for a work machine according to claim 3.

5. the display device displays the attachment according to substantially the same scale as the object depicted in the image; 5. A remote control system for a work machine according to claim 3 or 4.

6. The display device displays a second image other than the image representing the front, and displays a contour of the attachment superimposed on the second image.

2. The remote control system for a work machine according to claim 1.

7. The information about the attitude received by the receiving device includes information about the attitude of the upper rotating body, the display device displays a shape of at least a part of the lower traveling body so that the attitude of the lower traveling body relative to the upper rotating body can be recognized based on the received information regarding the attitude of the upper rotating body.

2. The remote control system for a work machine according to claim 1.

8. the display device displays a change in the posture of the attachment as an animation in accordance with the information related to the posture.

2. The remote control system for a work machine according to claim 1.

Images