Remote control system for industrial machinery

Abstract

To reduce the burden on the operator. [Solution] A remote control system for a work machine according to one embodiment comprises a work machine having a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable, an attachment having an end attachment connected to the upper rotating body 3, a posture detection device capable of identifying the posture of the attachment, and an imaging device capable of imaging the area around the work machine body; a display device; and a control device that acquires information representing the shape of the ground that is the work target of the work machine based on the change in the posture of the attachment identified by the detection result of the posture detection device, and displays the shape of the ground represented by the acquired information on the display device.

Description

Technical Field

[0001] The present invention relates to a remote control system for a working machine.

Background Art

[0002] Conventionally, when remotely operating a working machine, an operator operates the working machine while referring to image data captured by a camera provided at the work site or on the working machine. However, when image data captured by a monocular camera is displayed on a normal monitor, it is difficult for the operator to grasp the three-dimensional sense of the work target, and the work efficiency may decrease.

[0003] Therefore, in the technique described in Patent Document 1, a technique of grid-displaying the ground shape using point cloud data acquired by LiDAR has been proposed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in remote operation, since an imaging device is an essential component, when attempting to implement the technique described in Patent Document 1, in addition to the imaging device, it is necessary to separately provide a three-dimensional distance measuring sensor such as LiDAR. When providing a three-dimensional distance measuring sensor such as LiDAR, there is a problem that the cost increases.

[0006] In view of the above, by displaying the shape of the ground based on the change in the attitude of the attachment, while suppressing the increase in cost, the operator is made to grasp the three-dimensional sense, thereby reducing the operation burden.

Means for Solving the Problems

[0007] A remote control system for a work machine according to one aspect of the present invention comprises a work machine having a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable, an attachment having an end attachment connected to the upper rotating body 3, a posture detection device capable of identifying the posture of the attachment, and an imaging device capable of imaging the area around the work machine body; a display device; and a control device that acquires information representing the shape of the ground that is the work target of the work machine based on the change in the posture of the attachment identified by the detection result of the posture detection device, and displays the shape of the ground represented by the acquired information on the display device. [Effects of the Invention]

[0008] According to one aspect of the present invention, the shape of the ground is displayed based on the change in the posture of the attachment, thereby allowing the operator to grasp the three-dimensionality and reducing the burden on the operator. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of a remote control system according to the first embodiment. [Figure 2] This is a side view showing a work machine according to the first embodiment. [Figure 3] This diagram schematically shows an example of the configuration of a work machine according to the first embodiment. [Figure 4] This is a functional block diagram showing an example configuration of a remote control system according to the first embodiment. [Figure 5] This figure shows an example of the layout of a remote control room according to the first embodiment. [Figure 6] This diagram illustrates the excavation work performed by the work machine according to the first embodiment. [Figure 7] This figure shows an example of a display screen shown in the display device according to the first embodiment. [Figure 8] This is a sequence diagram showing the overall processing flow in the remote control system according to the first embodiment. [Figure 9]This is a sequence diagram showing the overall processing flow in the remote control system SYS according to Modification 1 of the first embodiment. [Figure 10] This is a schematic diagram showing an example of a remote control system according to the second embodiment. [Figure 11] This is a conceptual diagram showing the relationship between multiple work machines 100 and a management server according to the second embodiment. [Figure 12] This figure shows an example of a screen displayed by a remote controller in a remote control room for operating a work machine according to the second embodiment. [Figure 13] This figure shows an example of a screen displayed by a remote controller in a remote control room for operating a work machine according to the second embodiment. [Figure 14] This is a sequence diagram showing the overall processing flow in the remote control system according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described below with reference to the drawings. The embodiments described below are illustrative and do not limit the invention. Not all features and combinations thereof in the embodiments of this disclosure are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and redundant descriptions may be omitted.

[0011] The working machine 100 according to the embodiment of this disclosure is a shovel. The working machine 100 may be any working machine equipped with an attachment. In the illustrated example, the shovel as the working machine 100 is an excavator equipped with a bucket 6 as an end attachment, but it may also be an applied machine such as a forestry machine equipped with an end attachment other than a bucket 6.

[0012] (First Embodiment) First, referring to FIG. 1, the outline of a remote operation system (an example of a remote control system) SYS according to the first embodiment will be described. FIG. 1 is a schematic diagram showing an example of the remote operation system SYS according to the first embodiment.

[0013] <Devices constituting the remote operation system> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes a working machine 100 and a remote operation room RC.

[0014] The working machine 100 and the remote operation room RC are connected so as to be able to transmit and receive data via a communication line NW.

[0015] The working machine 100 enables wireless communication. And the working machine 100 can transmit and receive data to and from a device (for example, the remote operation room RC) connected to the communication line NW.

[0016] The working machine 100 exists at the work site where the working machine 100 performs work. Thus, in this embodiment, a plurality of types of devices are provided at the work site. And the working machine 100 can transmit information regarding the work site to the remote operation room RC. Thereby, the remote operation room RC can confirm the work site according to the information from the working machine 100. Note that this embodiment does not limit the device for measuring the work site to the working machine 100, and other types of devices such as a drone flying over the work site or an imaging device that can be carried by a user may be used.

[0017] The number of working machines 100 included in the remote operation system SYS may be one or a plurality. Thereby, the remote operation system SYS can provide information regarding the work site to the remote operation room RC through one or a plurality of working machines 100.

[0018] <Example of the configuration of the remote operation room> The remote control room RC is equipped with a communication device T2, a remote controller R40, an operating device R42, an operating sensor R43, and a display device D1E. The remote control room RC also has an operator's seat DS where the operator OP sits to remotely control the work machine 100.

[0019] A communication device (an example of a receiving device) T2 is configured to control communication with a communication device T1 (see Figure 2) attached to the work machine 100.

[0020] The remote controller R40 is an information processing device that performs various calculations. In this embodiment, the remote controller R40 is composed of a microcomputer including a CPU and memory. The various functions of the remote controller R40 are realized by the CPU executing a program stored in memory.

[0021] The display device D1E displays a screen based on information transmitted from the work machine 100, allowing the operator OP in the remote control room RC to visually check the area around the work machine 100. The display device D1E allows the operator to check the conditions of the work site, including the area around the work machine 100, even though the operator is in the remote control room RC. In the illustrated example, the display device D1E is a liquid crystal display that displays images captured by the imaging device S6 mounted on the work machine 100. The display device D1E may also be a display or projector that enables naked-eye stereoscopic viewing, or it may be a VR goggle or the like.

[0022] An operating device R42 (an example of an operating unit) is equipped with an operating sensor R43 for detecting the operation of the operating device R42. The operating sensor R43 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle of the operating lever around its pivot axis. The operating sensor R43 may also consist of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operating sensor R43 outputs information regarding the operation of the operating device R42 that it has detected to the remote controller R40. The remote controller R40 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operating sensor R43 may also be configured to generate an operation signal. In this case, the operating sensor R43 may output the operation signal to the communication device T2 without going through the remote controller R40. This enables remote control of the work machine 100 from the remote control room RC.

[0023] <Example of a work machine configuration> Next, with reference to Figure 2, an overview of the work machine 100 according to this embodiment will be described. Figure 2 is a side view of the work machine 100 as a work machine according to the first embodiment.

[0024] In Figure 2, +X represents one direction of the X-axis in the three-dimensional Cartesian coordinate system, and (not shown) -X represents the other direction of the X-axis. +Y represents one direction of the Y-axis in the three-dimensional Cartesian coordinate system, and (not shown) -Y represents the other direction of the Y-axis. +Z represents one direction of the Z-axis in the three-dimensional Cartesian coordinate system, and (not shown) -Z represents the other direction of the Z-axis. In Figure 1, the +X side of the work machine 100 corresponds to the front side of the work machine 100, and the -X side of the work machine 100 corresponds to the rear side of the work machine 100. Also, the +Y side of the work machine 100 corresponds to the left side of the work machine 100, and the -Y side of the work machine 100 corresponds to the right side of the work machine 100. Furthermore, the +Z side of the work machine 100 corresponds to the top side of the work machine 100, and the -Z side of the work machine 100 corresponds to the bottom side of the work machine 100. The same applies to other figures.

[0025] The work machine 100 comprises a lower traveling body 1, an upper rotating body 3 mounted on the lower traveling body 1 so as to be rotatable via a slewing mechanism 2, an attachment AT for performing various tasks, and a driver's cab 10. The driver's cab 10 is also called a cabin or cab. The front side of the work machine 100 (upper rotating body 3) corresponds to the side on which the attachment AT is attached to the upper rotating body 3 when the work machine 100 is viewed from directly above along the slewing axis of the upper rotating body 3. The left, right, and rear sides of the work machine 100 (upper rotating body 3) correspond to the left, right, and rear sides as seen from the perspective of an operator seated in the driver's seat inside the driver's cab 10, respectively.

[0026] The lower travel body 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the crawlers 1C include a left crawler and a right crawler. The left crawler is driven by a left travel hydraulic motor 2ML (see Figure 3), and the right crawler is driven by a right travel hydraulic motor 2MR (see Figure 3). The left travel hydraulic motor 2ML is a travel drive unit that drives the left crawler, which is the driven part, and can rotate the left crawler. The right travel hydraulic motor 2MR is a travel drive unit that drives the right crawler, which is the driven part, and can rotate the right crawler. Note that the travel drive units may also be electric motors.

[0027] A boom 4 is rotatably mounted to the front center of the upper slewing body 3, an arm 5 is rotatably mounted to the tip of the boom 4, and a bucket 6 is rotatably mounted to the tip of the arm 5. In the illustrated example, the boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of attachment AT. The boom 4, arm 5, and bucket 6 are driven by a boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively.

[0028] Bucket 6 is an example of a work tool (end attachment). Bucket 6 is used, for example, for excavation work. Depending on the work content, other work tools may be attached to the tip of arm 5 instead of bucket 6. Other work tools may be other types of buckets, such as large buckets, slope buckets, or dredging buckets. Other work tools may also be types of work tools other than buckets, such as agitators, breakers, grapples, or lifting magnets. The excavation attachment may be provided with a bucket tilt mechanism.

[0029] The slewing hydraulic motor 2A, the left travel hydraulic motor 2ML, the right travel hydraulic motor 2MR, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 are hydraulic actuators driven by hydraulic fluid discharged from a hydraulic pump.

[0030] Furthermore, the work machine 100 may have all or part of its driven parts, such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6, electrically driven. In other words, the work machine 100 may be a hybrid excavator or electric excavator, in which all or part of its driven parts are driven by electric actuators.

[0031] The imaging device S6 is mounted on the upper rotating body 3 and captures images of the area around the main body of the work machine 100, acquiring image information representing the area around the main body of the work machine 100. In the illustrated example, the imaging device S6 includes a front camera S6F, a left camera S6L, a right camera S6R, and a rear camera S6B.

[0032] The front camera S6F is a camera that captures images in front of the work machine 100 and is mounted on the outside of the operator's cab 10, such as on the roof of the operator's cab 10 or the side of the boom 4. The left camera S6L is a camera that captures images to the left of the work machine 100, the right camera S6R is a camera that captures images to the right of the work machine 100, and the rear camera S6B is a camera that captures images behind the work machine 100. Specifically, the front camera S6F, left camera S6L, right camera S6R, and rear camera S6B are all monocular wide-angle cameras equipped with an image sensor such as a CCD or CMOS, and the information of the captured images is taken up by the controller 30. Alternatively, the images captured by the imaging device S6 may be output to the display device D1 (see Figure 3).

[0033] In the illustrated example, the front camera S6F is mounted on the roof of the driver's cab 10, the left camera S6L is mounted on the upper left end of the upper surface of the upper rotating body 3, the right camera S6R is mounted on the upper right end of the upper surface of the upper rotating body 3, and the rear camera S6B is mounted on the upper rear end of the upper surface of the upper rotating body 3.

[0034] The imaging device S6 may constitute an object detection device that detects objects in the vicinity of the main body of the work machine 100. The object detection device may consist of devices other than a camera. For example, the object detection device may be a LiDAR. A LiDAR is, for example, a device capable of measuring the distance between a point cloud of 1 million or more points within the monitoring range and the LiDAR (laser source). Alternatively, the object detection device may be other devices capable of measuring the distance to an object, such as a stereo camera, a depth image camera, or a millimeter-wave radar. When a millimeter-wave radar or the like is used as the object detection device, the object detection device may derive the distance and direction of the object by transmitting a large number of signals (such as laser light) toward the object and receiving the reflected signals. Alternatively, the object detection device may be a combination of two or more types of devices. For example, the object detection device may be a combination of an imaging device and a LiDAR, a combination of an imaging device and a millimeter-wave radar, or a combination of an imaging device and a stereo camera.

[0035] The controller 30 is an example of a control device and consists of a computer including, for example, a CPU, a volatile memory device, a non-volatile memory device, and various input / output interfaces. The controller 30 implements various functions, for example, by reading a program from the non-volatile memory device, loading it into the volatile memory device, and having the CPU execute it. In the illustrated example, the controller 30 is configured to implement various functions and control the work machine 100. These functions include, for example, a machine guidance function that guides the operator in manually operating the work machine 100. The functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and objects within a monitoring range around the work machine 100.

[0036] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor that can detect the rotation angle of the boom 4 relative to the upper slewing body 3 (hereinafter referred to as "boom angle") which changes per unit time. The boom angle sensor S1 can detect the angular velocity of the boom 4, which indicates the change in boom angle, and the angular acceleration of the boom 4, which indicates the rate of said change. The boom angle is, for example, at its minimum when the boom 4 is at its lowest position, and increases as the boom 4 is raised.

[0037] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the rotation angle of the arm 5 relative to the boom 4 (hereinafter referred to as "arm angle"). The arm angle sensor S2 can detect the angular velocity of the arm 5, which indicates the change in the arm angle, and the angular acceleration of the arm 5, which indicates the rate of change. The arm angle is, for example, at its minimum when the arm 5 is closed to its shortest extent, and increases as the arm 5 is opened.

[0038] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the rotation angle of the bucket 6 relative to the arm 5 (hereinafter referred to as "bucket angle"). The bucket angle sensor S3 can detect the angular velocity of the bucket 6, which indicates the change in bucket angle, and the angular acceleration of the bucket 6, which indicates the rate of change. The bucket angle is, for example, at its minimum when the bucket 6 is fully closed, and increases as the bucket 6 is opened.

[0039] The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 can be any sensor capable of acquiring information about the attitude of the attachment (an example of an attitude sensor), and may be an IMU (Inertial Measurement Unit), a 6-axis sensor, a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, a rotary encoder that detects the rotation angle around the connecting pin, a gyro sensor, or a combination of an acceleration sensor and a gyro sensor, respectively.

[0040] The detection signals corresponding to the boom angle from the boom angle sensor S1, the detection signals corresponding to the arm angle from the arm angle sensor S2, and the detection signals corresponding to the bucket angle from the bucket angle sensor S3 are input to the controller 30. The detection signals may include angular velocity in addition to angle.

[0041] The machine tilt sensor S4 detects the tilt state of the machine (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. The machine tilt sensor S4 is, for example, attached to the upper rotating body 3 and detects the tilt angle of the work machine 100 (i.e., the upper rotating body 3) around two axes: the longitudinal direction and the lateral direction. The machine tilt sensor S4 may be, for example, an acceleration sensor, a 6-axis sensor, or an IMU. The detection signal corresponding to the tilt angle from the machine tilt sensor S4 is input to the controller 30.

[0042] The rotation sensor S5 outputs information regarding the rotation of the upper rotating body 3. The rotation sensor S5 detects, for example, the rotational angular velocity and rotational angular acceleration of the upper rotating body 3 relative to the lower traveling body 1. The rotation sensor S5 may also detect the rotation angle. The rotation sensor S5 may be, for example, a gyro sensor, a resolver, or a rotary encoder. The detection signals corresponding to the rotation angle, rotational angular velocity, and rotational angular acceleration of the upper rotating body 3 detected by the rotation sensor S5 are input to the controller 30.

[0043] The boom cylinder 7 is equipped with a boom rod pressure sensor S7R and a boom bottom pressure sensor S7B. The arm cylinder 8 is equipped with an arm rod pressure sensor S8R and an arm bottom pressure sensor S8B. The bucket cylinder 9 is equipped with a bucket rod pressure sensor S9R and a bucket bottom pressure sensor S9B. The boom rod pressure sensor S7R, boom bottom pressure sensor S7B, arm rod pressure sensor S8R, arm bottom pressure sensor S8B, bucket rod pressure sensor S9R, and bucket bottom pressure sensor S9B are devices that detect the pressure (an example of driving force) required to drive each component of the attachment AT (for example, boom 4, arm 5, and bucket 6), and are collectively referred to as "cylinder pressure sensors (an example of a driving force detection device)". In this embodiment, the device for detecting the driving force required to drive each component of the attachment AT is not limited to cylinder pressure sensors, and other detection devices such as strain gauges may be used.

[0044] The boom rod pressure sensor S7R detects the pressure in the rod-side oil chamber of the boom cylinder 7 (hereinafter referred to as "boom rod pressure"), and the boom bottom pressure sensor S7B detects the pressure in the bottom-side oil chamber of the boom cylinder 7 (hereinafter referred to as "boom bottom pressure"). The arm rod pressure sensor S8R detects the pressure in the rod-side oil chamber of the arm cylinder 8 (hereinafter referred to as "arm rod pressure"), and the arm bottom pressure sensor S8B detects the pressure in the bottom-side oil chamber of the arm cylinder 8 (hereinafter referred to as "arm bottom pressure"). The bucket rod pressure sensor S9R detects the pressure in the rod-side oil chamber of the bucket cylinder 9 (hereinafter referred to as "bucket rod pressure"), and the bucket bottom pressure sensor S9B detects the pressure in the bottom-side oil chamber of the bucket cylinder 9 (hereinafter referred to as "bucket bottom pressure").

[0045] The operator's cab 10 is a compartment where the operator sits and is located on the front left side of the upper rotating body 3. However, if the work machine 100 operates by fully automatic operation, the operator's cab 10 may be omitted.

[0046] The indoor imaging device S10 is an imaging device installed inside the operator's cab 10, and is a camera that images the area in front of the work machine 100. For example, the indoor imaging device S10 is installed in the operator's cab 10 so as to be near the operator's line of sight from where the operator is seated.

[0047] The communication device T1 communicates with external devices through a communication network including a mobile communication network, a satellite communication network, or the Internet. The communication device T1 is, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0048] The memory device ST is a read-write non-volatile storage medium.

[0049] The work machine 100 operates actuators in response to the operation of the operator seated in the cab 10, driving the driven parts such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6.

[0050] Alternatively, the work machine 100 may be configured to be remotely controlled from outside the work machine 100. When the work machine 100 is remotely controlled, the inside of the operator's cab 10 may be unoccupied.

[0051] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operator's actions. This enables the work machine 100 to automatically operate at least a portion of the driven parts, such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, that is, to achieve a so-called "machine control function".

[0052] Figure 3 is a schematic diagram showing an example of the configuration of the work machine 100 according to this embodiment. In Figure 3, the mechanical power transmission system, hydraulic fluid line, pilot line, and electrical control system are indicated by double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0053] The drive system of the work machine 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 also includes hydraulic actuators such as a slewing hydraulic motor 2A, a left travel hydraulic motor 2ML, a right travel hydraulic motor 2MR, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0054] The engine 11 is an example of a power source for the work machine 100, and is mounted, for example, at the rear of the upper rotating body 3. The power source for the work machine 100 may also be a combination of a battery or fuel cell and an electric motor. Specifically, the engine 11 rotates at a constant speed at a preset target rotational speed under direct or indirect control by the controller 30, driving the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 may also be a gasoline engine or a hydrogen engine, etc.

[0055] The regulator 13 controls the discharge rate of the main pump 14. For example, the regulator 13 controls the discharge rate of the main pump 14 by adjusting the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0056] The main pump 14, for example, is mounted at the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic fluid to the control valve unit 17 through the hydraulic fluid line. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump.

[0057] The control valve unit 17 is one of the hydraulic control devices that control the hydraulic system in the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171 to 176. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control the flow rate of hydraulic fluid flowing from the main pump 14 to the hydraulic actuators, and the flow rate of hydraulic fluid flowing from the hydraulic actuators to the hydraulic fluid tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-travel hydraulic motor 2ML, a right-travel hydraulic motor 2MR, and a slewing hydraulic motor 2A. Specifically, control valve 171 corresponds to the left-travel hydraulic motor 2ML, control valve 172 corresponds to the right-travel hydraulic motor 2MR, and control valve 173 corresponds to the slewing hydraulic motor 2A. Furthermore, control valve 174 corresponds to bucket cylinder 9, control valve 175 corresponds to boom cylinder 7, and control valve 176 corresponds to arm cylinder 8.

[0058] The pilot pump 15 is an example of a pilot pressure generating device and is configured to supply hydraulic fluid to a hydraulic control device via a pilot line. In the illustrated example, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pressure generating device may be implemented by the main pump 14. That is, the main pump 14 may have the function of supplying hydraulic fluid to the control valve unit 17 via a hydraulic fluid line, as well as the function of supplying hydraulic fluid to various hydraulic control devices via a pilot line. In this case, the pilot pump 15 may be omitted.

[0059] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the example shown in the figure, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0060] The operating device 26 is a device used by the operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator may be a hydraulic actuator or an electric actuator.

[0061] The operation sensor 29 is configured to detect the operator's actions using the operation device 26. In this embodiment, the operation sensor 29 detects the operating direction and amount of the operation device 26 corresponding to each actuator and outputs the detected values ​​to the controller 30. In the illustrated example, the controller 30 can control the opening area of ​​the proportional valve 31 according to the output of the operation sensor 29. The controller 30 then supplies the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the hydraulic fluid supplied to each pilot port (pilot pressure) is, in principle, the pressure corresponding to the operating direction and amount of the operation device 26 corresponding to each hydraulic actuator. Thus, the operation device 26 is configured to supply the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0062] The proportional valve 31, which functions as a control valve for machine control, is located in the pipeline connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow area of ​​the pipeline. In the illustrated example, the proportional valve 31 operates in response to control commands output by the controller 30. Therefore, the controller 30 can adjust the pilot pressure acting on the pilot port of the control valve by the proportional valve 31, independently of the operation of the operating device 26 by the operator.

[0063] This configuration allows the controller 30 to operate the hydraulic actuator corresponding to a specific operating device 26 even when no operation is being performed on that particular operating device 26.

[0064] Furthermore, as shown in Figure 3, the control system of the work machine 100 includes a controller 30, a display device D1, and a communication device T1, etc.

[0065] The controller 30 is configured to output control commands to the regulator 13 as needed, thereby changing the discharge rate of the main pump 14.

[0066] Furthermore, the controller 30 may be configured to perform control related to a machine guidance function that guides the manual operation of the work machine 100 by the operator through the operating device 26. Alternatively, the controller 30 may be configured to perform control related to a machine control function that automatically assists the manual operation of the work machine 100 by the operator through the operating device 26.

[0067] Furthermore, some of the functions of controller 30 may be implemented by other controllers (control devices). In other words, the functions of controller 30 may be implemented in a manner distributed among multiple controllers. For example, machine guidance functions and machine control functions may be implemented by dedicated controllers (control devices).

[0068] <Block configuration of the remote control system> Figure 4 is a functional block diagram showing an example configuration of the remote control system SYS according to this embodiment. In the example shown in Figure 4, the block configurations of the remote control room RC and the work machine 100, which are included in the remote control system SYS, are shown. The hardware configuration of the work machine 100 will not be explained.

[0069] <Configuration of the Remote Control Room (RC)> The remote control room RC includes a remote controller R40, a communication device T2, an operation sensor R43, an operation device R42, and a display device D1E. The communication device T2, operation sensor R43, and operation device R42 have been described above, so their explanation is omitted.

[0070] Next, we will explain the remote control room RC. Figure 5 shows an example of the layout of the remote control room RC. The remote control room RC is equipped with multiple control devices R42, with the operator's seat DS as the reference point.

[0071] In this embodiment, the display device D1E is a multi-display consisting of six monitors arranged in two vertical rows and three horizontal columns, as shown in Figure 5. Specifically, the display device D1E includes the central monitor D1Ea, the upper monitor D1Eb, the left monitor D1Ec, the right monitor D1Ed, the upper left monitor D1Ee, and the upper right monitor D1Ef.

[0072] <<Changes in ground shape due to excavation operations by construction machinery>> Figure 6 is a diagram illustrating the excavation work performed by the work machine 100 according to this embodiment. When excavation is performed by the work machine 100 as shown in Figure 6, the bucket 6 changes in the order of positions 6a, 6b, 6c, and 6d.

[0073] When performing excavation work, the controller 30 acquires data representing the shape of the ground formed by the removal of soil (an example of an object) that was present in the area through which the bucket 6 passed, and this data is referred to as topographic data.

[0074] Specifically, the controller 30 can calculate the position of the tip of the bucket 6 at each of the positions 6a, 6b, 6c, and 6d based on the detection results of the angle sensors S1 to S3. The controller 30 can recognize the shape of the ground (after the soil has been removed by the bucket 6), indicated by line 1701, from the change in the position (trajectory) of the tip of the bucket 6 at positions 6a, 6b, 6c, and 6d, which is detected based on the detection results of the angle sensors S1 to S3.

[0075] In other words, the controller 30 can recognize the shape of the ground, indicated by line 1701 which shows the change in the position (trajectory) of the tip of the bucket 6, based on the progression of detection results from angle sensors S1 to S3.

[0076] Therefore, when the work machine 100 performs excavation work, the controller 30 according to this embodiment acquires terrain data representing the shape of the ground that the work machine 100 is working on, based on the change in the posture of the tip of the bucket 6 (an example of an attachment). The terrain data according to this embodiment may be any form of data as long as it is information that can be stored in three dimensions. For example, the controller 30 according to this embodiment may acquire terrain data as a set of position coordinates representing the surface of the ground, where the set of positions where parts of the bucket 6 (e.g., the tip) have moved is used.

[0077] The controller 30 then generates mesh data that represents the shape of the ground in three dimensions by connecting each position that makes up the ground surface with lines, based on the terrain data. The controller 30 then transmits the image data captured by the imaging device S6 and the mesh data representing the shape of the ground to the remote controller R40. The controller 30 may also transmit the image data with the mesh data superimposed on it.

[0078] The remote controller R40 then receives (an example of acquisition) the captured image data with the mesh data superimposed on it, and displays the captured image data with the mesh data superimposed on it on the display device D1E. Note that this embodiment is not limited to superimposing the mesh data onto the captured image data; the mesh data may be displayed on the display device D1E without being superimposed on the captured image data.

[0079] The example shown in Figure 6 illustrates the case where excavation work is performed by the work machine 100. In the case of excavation work, terrain data is generated based on, for example, the change in the posture of the bucket 6's toes as it moves. However, this embodiment does not limit the work required to acquire terrain data indicating the shape of the ground to excavation work, but can also be applied when performing other tasks such as compaction. When compaction is performed, the controller 30 recognizes the shape of the ground from the change in the position of the bottom surface of the bucket 6.

[0080] In this way, the controller 30 changes the part of the attachment AT for recognizing the shape of the ground according to the work performed by the work machine 100. The controller 30 may identify the work performed by the work machine 100 according to the operation performed by the operator OP, or it may accept an operation from the operator OP to select the work mode to be performed by the work machine 100.

[0081] Furthermore, the controller 30 may identify the part of the bucket 6 that is in contact with the ground based on the angle of the bucket 6 detected by the angle sensors S1 to S3, and the angle of the bucket 6 and the direction of the pressure acting on the bucket 6 based on the detection results of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B, and recognize the shape of the ground from the change in the position of that part.

[0082] In this way, the controller 30 identifies the part of the bucket 6 that is in contact with the ground (e.g., the toe or the bottom surface) according to the work performed by the attachment AT, and acquires terrain data representing the shape of the ground based on the change in the position of that part. Therefore, this embodiment can appropriately recognize the shape of the ground according to the work, thereby improving the accuracy of the displayed ground shape.

[0083] <<Functional Blocks of Work Machines>> Returning to Figure 4, we will now describe each functional block within the controller 30 of the work machine 100. Each functional block within the controller 30 is conceptual and does not necessarily need to be physically configured as shown. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed in each functional block is realized, all or any part thereof, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The controller 30, by realizing the program, includes a receiving control unit 301, an actuator driving unit 302, an acquisition unit 303, a terrain data processing unit 304, an image generation unit 305, and a transmission control unit 306. The storage device ST includes a terrain data storage unit ST1.

[0084] The terrain data storage unit ST1 stores terrain data indicating the shape of the ground on which the work machine 100 operates. In this embodiment, the terrain data is three-dimensional shape data representing the surface of the ground. In this embodiment, the terrain data is an example of being represented in a relative coordinate system based on a predetermined position (e.g., center position) of the work machine 100.

[0085] The receiving control unit 301 controls the reception of various information from the remote control room RC via the communication device T1. For example, the receiving control unit 301 receives operation signals from the remote control room RC to control the operation of the work machine 100.

[0086] The actuator drive unit 302 is configured to drive the actuator mounted on the work machine 100. In this embodiment, the actuator drive unit 302 generates and outputs an operating signal for each of the multiple solenoid valves included in the proportional valve 31 based on the operation signal transmitted from the remote control room RC.

[0087] Upon receiving an activation signal, each solenoid valve increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0088] The acquisition unit 303 acquires signals from various detection devices installed on the work machine 100. For example, the acquisition unit 303 acquires the detection results from the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3. The acquisition unit 303 also acquires the detection results from the slewing sensor S5. Furthermore, the acquisition unit 303 acquires detection results from each of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B.

[0089] Furthermore, the acquisition unit 303 acquires captured image data from the imaging device S6. Also, the acquisition unit 303 acquires captured image data from the indoor imaging device S10.

[0090] The terrain data processing unit 304 acquires terrain data relating to the current shape of the ground being worked on. Based on the detection results acquired by angle sensors S1 to S3 and the rotation angle acquired by rotation sensor S5, the terrain data processing unit 304 acquires the coordinates and orientation indicating the current position of the bucket 6, and acquires information relating to the current shape of the ground being worked on based on the changes in the coordinates and orientation of the bucket 6. Furthermore, in addition to the changes in the posture of the attachment AT identified by the detection results of angle sensors S1 to S3 and rotation sensor S5, the terrain data processing unit 304 acquires information representing the shape of the ground being worked on by the work machine 100 based on the pressure related to the attachment AT detected by cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B.

[0091] Specifically, the terrain data processing unit 304 determines whether or not soil and other materials have been removed from the area through which the bucket 6 has passed by excavation work, based on the detection results from the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B.

[0092] The terrain data processing unit 304 determines whether the pressure applied to the tip of the bucket 6, calculated from the detection results of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B, is higher than a predetermined threshold that serves as a criterion for determining whether or not excavation is being performed.

[0093] The terrain data processing unit 304 determines that the pressure applied to the tip of the bucket 6 is above a predetermined threshold, and as long as this pressure is within a certain range, it considers that the bucket 6 is removing soil and other debris from the area it has passed through, and updates the terrain data indicating the current shape of the ground being worked on. In this way, the terrain data processing unit 304 according to this embodiment updates the terrain data of a position derived from the distance from the center of the work machine 100 to the bucket 6, which is derived from the angle sensors S1 to S3, and the slewing angle acquired by the slewing sensor S5, with the work machine 100's predetermined position (e.g., center position) as the reference point.

[0094] The terrain data processing unit 304 determines that the pressure applied to the tip of the bucket 6 is below a predetermined threshold, and assumes that no excavation is being performed by the bucket 6 (for example, it is in the air), and suppresses the acquisition of terrain data representing the shape of the ground based on changes in the attitude of the bucket 6. Therefore, the controller 30 according to this embodiment can suppress the updating of terrain data for areas where no excavation is being performed, thereby improving the accuracy of the ground shape displayed based on the terrain data.

[0095] The line 1701 shown in Figure 6 represents the current ground shape, derived from the change in the attitude (trajectory) of the bucket 6. As described above, the terrain data processing unit 304 updates the terrain data storage unit ST1 with the acquired terrain data showing the current ground shape. Therefore, the terrain data storage unit ST1 stores terrain data showing the current ground shape as it has changed due to the work performed by the work machine 100.

[0096] The image generation unit 305 generates three-dimensional mesh data, which represents the three-dimensional shape of the ground surface using a combination of points and lines, based on the terrain data stored in the terrain data storage unit ST1. The points represented as mesh data do not necessarily correspond to all positions stored in the terrain data storage unit ST1, and may be thinned out at predetermined intervals. The thinning interval may be adjusted by the operator OP.

[0097] The image generation unit 305 then superimposes the generated mesh data onto the image data captured by the imaging device S6. Specifically, the image generation unit 305 superimposes the mesh data onto the image data so that the position of the ground shown in the image data matches the position of the ground shown in the mesh data.

[0098] The transmission control unit 306 controls the transmission of various information based on the acquisition results of the acquisition unit 303 to the remote control room RC via a communication device (an example of a transmission device) T1. For example, the transmission control unit 306 controls the transmission of image data with mesh data superimposed on it to the remote control room RC.

[0099] <<Functional Blocks of the Remote Control Room>> This section describes the functional blocks within the remote controller (an example of a control unit) 40 of the remote control room RC. Each functional block within the remote controller R40 is conceptual and does not necessarily need to be physically configured as shown in the diagram. All or part of each functional block can be configured by distributing and integrating them functionally or physically in any unit. Each processing function performed by each functional block is realized, all or any part thereof, by a program executed on the CPU. Alternatively, each functional block may be realized as hardware using wired logic. The remote controller R40, by implementing the program, includes a receive control unit 401, an output control unit 402, a signal generation unit 403, and a transmit control unit 404.

[0100] The receiving control unit 401 controls the receiving of various information from the work machine 100 via the communication device T2.

[0101] For example, the receiving control unit 401 controls the reception of image data with mesh data superimposed from the work machine 100. The receiving control unit 401 also controls the reception of detection results from various detection devices installed on the work machine 100.

[0102] The output control unit 402 controls the output of various types of information from each of the display devices D1E. For example, the output control unit 402 controls the output of captured image data with mesh data superimposed on it from the display device D1E.

[0103] Therefore, after the soil (an example of an object) has been removed from the area excavated by the bucket 6 of the work machine 100 (an example of a first area), the output control unit 402 displays the surface of the area where soil has not been removed (an example of a second area) as a mesh shape.

[0104] The signal generation unit 403 generates operation signals to control the operation of the work machine 100 according to the operation received by the operation sensor R43.

[0105] The transmission control unit 404 controls the transmission of various types of information to the remote control room RC. For example, the transmission control unit 404 controls the transmission of operation signals generated by the signal generation unit 403 to the work machine 100.

[0106] Next, the output control unit 402 will describe the screen displayed on the display device D1E. Figure 7 is a diagram showing an example of a display screen shown on the display device D1E according to this embodiment. In the example screen shown in Figure 7, a screen based on the information received by the reception control unit 401 is displayed.

[0107] The upper left monitor D1Ee displays overhead image information generated by the output control unit 402. The overhead image information 1703 is generated by the output control unit 402 based on the image data captured by the imaging device S6, which is received by the receiving control unit 401. An icon 1703a representing the work machine 100 is displayed in the center of the overhead image information 1703.

[0108] The left monitor D1Ec displays image information captured by the imaging device S6. For example, it displays image information of the area behind the work machine 100, captured by the rear camera S6B.

[0109] The upper monitor D1Eb displays various information as needed. For example, the upper monitor D1Eb displays messages for the operator (OP).

[0110] The upper right monitor D1Ef displays the detection results of various detection devices of the work machine 100. For example, it displays the date and time display area 1704a, the driving mode display area 1704b, the attachment display area 1704c, the fuel consumption display area 1704d, the engine control status display area 1704e, the coolant temperature display area 1704g, the fuel level display area 1704h, the rotation speed level display area 1704i, the urea solution level display area 1704j, the hydraulic oil temperature display area 1704k, the weather display area 1704l, and the engine operating time display area 1704m.

[0111] The date and time display area 1704a is the area that displays the current date and time. The driving mode display area 1704b is the area that displays the current driving mode. The attachment display area 1704c is the area that displays an image representing the attachment currently installed. The fuel consumption display area 1704d is the area that displays fuel consumption information calculated by the controller 30.

[0112] The engine control status display area 1704e is the area that displays the control status of the engine 11. The coolant temperature display area 1704g is the area that displays the current engine coolant temperature. The fuel level display area 1704h is the area that displays the remaining amount of fuel stored in the fuel tank.

[0113] The rotation speed level display area 1704i is an area that displays the current level set by the dial (not shown) as an image. The rotation speed level display area 1704i displays a number indicating the selected level. The number "1" displayed in the rotation speed level display area 1704i indicates that the selected rotation speed level is "Level 1". The number "n" displayed in the rotation speed level display area 1704i indicates that the selected rotation speed level is "Level n". "n" is a natural number. When the operator OP rotates the dial (not shown), the number displayed in the rotation speed level display area 1704i changes.

[0114] The urea solution level display area 1704j is an area that displays the remaining amount of urea solution stored in the urea solution tank as an image. The hydraulic oil temperature display area 1704k is an area that displays the temperature of the hydraulic oil in the hydraulic oil tank.

[0115] The weather display area 1704l displays the weather in the location where the work machine 100 is located as a string of characters. For example, based on the position measured by the positioning device PS, the remote controller R40 displays information obtained from an external server that provides weather information. The engine operating time display area 1704m is an area that displays the cumulative operating time of the engine 11.

[0116] The central monitor D1Ea and the right monitor D1Ed display the captured image data with mesh data superimposed on it.

[0117] The image data displayed on the central monitor D1Ea shows the situation in front of the work machine 100 from the viewpoint inside the driver's cab 10. The image data displayed on the right monitor D1Ed shows the situation to the right of the work machine 100 from the viewpoint inside the driver's cab 10. In other words, when the operator OP in the operator's seat DS refers to the central monitor D1Ea or the right monitor D1Ed, the operator OP can be given the feeling of being in the driver's cab 10 of the work machine 100.

[0118] The image data on the central monitor D1Ea displays mesh data 1801 aligned with the shape of the ground. Therefore, the operator OP can recognize the shape of the ground by referring to the mesh data 1801.

[0119] Furthermore, in the image data captured by the central monitor D1Ea, marks 1811 and 1812 are shown at positions vertically below both ends of the tip of the bucket 6 within the area where mesh data 1801 is displayed. The superposition of marks 1811 and 1812, which indicate both ends of the tip of the bucket 6, onto the image data may be performed by the controller 30 or by the remote controller R40.

[0120] For example, the controller 30 or remote controller R40 identifies the positions of both ends of the bucket's claws according to the detection results of the angle sensors S1 to S3, identifies the point where the identified positions intersect vertically downward within the display area where the mesh data 1801 is shown, and superimposes marks 1811 and 1812 on the captured image data at the identified point. The remote controller R40 then displays marks 1811 and 1812 at the identified point on the central monitor D1Ea. The operator OP can perceive the depth relative to the bucket 6 by looking at marks 1811 and 1812 displayed on the captured image data. Therefore, the operator OP can perform the work safely.

[0121] In this embodiment, the image data captured by the central monitor D1Ea displays lines 1821 and 1822 connecting both ends of the bucket 6 to marks 1811 and 1812. Therefore, the operator OP can recognize the correspondence between marks 1811 and 1812 and the bucket 6. Whether or not lines 1821 and 1822 are displayed can be determined according to the embodiment; for example, the remote controller R40 may not display lines 1821 and 1822, and may only display marks 1811 and 1812.

[0122] In this embodiment, when the remote controller R40 is displaying mesh data on the central monitor D1Ea, it is possible to modify at least a portion of the mesh data (information representing the shape of the ground) that is the target of work by the work machine 100, according to the operation input from the operating device R42.

[0123] For example, if the operator (OP) recognizes that the mesh data displayed on the central monitor D1Ea differs from the shape of the ground, they will perform an operation to modify a portion of the displayed mesh data.

[0124] The remote controller R40 sends instructions to the controller 30 to modify a portion of the mesh data according to the operation.

[0125] The remote controller R40 can accept any operation to modify the mesh data. For example, the remote controller R40 can accept an operation to revert to the shape of the previous mesh data. When the remote controller R40 accepts an operation to revert to the previous shape, it sends an instruction to the work machine 100 to revert to the terrain data before the work machine 100 performed one operation such as excavation.

[0126] Another example is that the remote controller R40 can accept an operation to initialize any area of ​​the mesh data. When the remote controller R40 receives an operation to specify an area of ​​the mesh data to be initialized, it sends an instruction to the work machine 100 to delete the terrain data contained in that area.

[0127] Then, the terrain data processing unit 304 of the controller 30 updates the terrain data stored in the terrain data storage unit ST1 according to the received instructions. Therefore, the operator OP can modify the shape of the terrain displayed on the display device D1E, thereby improving the accuracy of the displayed terrain shape.

[0128] The processing procedure executed by the remote control system SYS according to this embodiment will now be described. Figure 8 is a sequence diagram showing the overall processing flow in the remote control system SYS according to this embodiment.

[0129] The remote controller R40 in the remote control room RC receives the operation performed by the operating device R42 from the operating sensor R43 (S1911).

[0130] The transmission control unit 404 controls the transmission of the operation signal generated by the signal generation unit 403 based on the received operation to the work machine 100 (S1912).

[0131] The actuator drive unit 302 of the work machine 100 controls the operation of the attachment AT of the work machine 100 based on the operation signal received by the receiving control unit 401 (S1901).

[0132] The acquisition unit 303 acquires the change in the posture of the bucket 6 (coordinates and orientation indicating the current position) (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 from various detection devices provided on the work machine 100 (S1902). The detection results acquired from the various detection devices include, for example, the detection results from each of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B, and angle information from the angle sensors S1, S2, and S3. The acquisition unit 303 then identifies the posture of the bucket 6 based on the angle information from the angle sensors S1, S2, and S3. Furthermore, based on the posture of the bucket 6 and the detection results from each of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B, the acquisition unit 303 can identify the part of the bucket 6 where the reaction force is generated (e.g., the toe or bottom surface) and the direction of the reaction force occurring at that part (e.g., the toe or bottom surface). The controller 30 can determine whether the bucket 6 is in contact with the ground, and if so, it can recognize the part of the bucket 6 that is in contact with the ground.

[0133] The terrain data processing unit 304 identifies the current shape of the ground being worked on based on the change in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6, and updates the terrain data indicating the current shape of the ground being worked on in the terrain data storage unit ST1 (S1903).

[0134] The image generation unit 305 generates mesh data representing the surface shape of the ground based on the terrain data stored in the terrain data storage unit ST1 (S1904).

[0135] The acquisition unit 303 acquires captured image data from the imaging device S6 and the indoor imaging device S10 (S1905).

[0136] The image generation unit 305 superimposes the generated mesh data onto the image data captured by the indoor imaging device S10 (S1906). The image generation unit 305 may also superimpose marks indicating the position of the bucket 6 onto the image data.

[0137] The transmission control unit 306 transmits the captured image data with the mesh data superimposed to it to the remote control room RC (S1907).

[0138] The output control unit 402 of the remote controller R40 displays the received image data with the mesh data superimposed on it on the display device D1E (S1913).

[0139] In this embodiment, the remote controller R40 has been described in an example where mesh data is used as the display method for indicating the shape of the ground on which the work machine 100 operates. However, this embodiment does not limit the display method for indicating the shape of the ground to mesh data. For example, the remote controller R40 may represent concentric circles at predetermined intervals with the work machine 100 as the center, and the shape of the ground around the work machine 100 may be represented by the irregularities in these concentric circles.

[0140] Thus, the controller 30 and remote controller R40 according to this embodiment, by having the above-described configuration, acquire information representing the shape of the ground that the work machine 100 is working on, without using a distance measuring sensor for detecting three-dimensional shapes, such as a LiDAR (an example of a three-dimensional shape detection device) capable of detecting the three-dimensional shape of objects around the work machine 100 body. Therefore, the controller 30 and remote controller R40 according to this embodiment can achieve cost reduction.

[0141] (Modification 1 of the first embodiment) In the first embodiment, an example was described in which the controller 30 of the work machine 100 generates image data with superimposed mesh data. However, the first embodiment is not limited to the mode in which the controller 30 generates image data with superimposed mesh data. Therefore, in a modification of the first embodiment, a mode in which the remote controller R40 generates image data with superimposed mesh data will be described.

[0142] Therefore, the remote controller R40 in this modified example is provided with a configuration for generating image data with mesh data superimposed on it (for example, a terrain data processing unit 304 and an image generation unit 305). Furthermore, the terrain data storage unit ST1 is stored in the storage device provided in the remote controller R40.

[0143] The processing procedure performed by the remote control system SYS according to this modified example will be described. Figure 9 is a sequence diagram showing the overall processing flow in the remote control system SYS according to this modified example.

[0144] The remote controller R40 in the remote control room RC receives operations from the operation sensor R43 and from the operation device R42 (S2011).

[0145] The transmission control unit 404 controls the transmission of the operation signal generated by the signal generation unit 403 based on the received operation to the work machine 100 (S2012).

[0146] The actuator drive unit 302 of the work machine 100 controls the operation of the attachment AT of the work machine 100 based on the operation signal received by the receiving control unit 401 (S2001).

[0147] The acquisition unit 303 acquires changes in the posture of the bucket 6 (coordinates and orientation indicating the current position) and the direction of the reaction force acting on the bucket 6 from various detection devices provided on the work machine 100 (S2002). The detection results acquired from the various detection devices include, for example, the detection results from each of the cylinder pressure sensors (examples of pressure detection devices) S7R, S7B, S8R, S8B, S9R, and S9B, and angle information from angle sensors (examples of posture detection devices) S1, S2, and S3. The acquisition unit 303 then identifies the posture of the bucket 6 based on the angle information from the angle sensors S1, S2, and S3. Furthermore, based on the posture of the bucket 6 and the detection results from each of the cylinder pressure sensors S7R, S7B, S8R, S8B, S9R, and S9B, the acquisition unit 303 can identify the part of the bucket 6 where the reaction force is generated (e.g., the toe or bottom surface) and the direction of the reaction force occurring at that part (e.g., the toe or bottom surface). The controller 30 can determine whether the bucket 6 is in contact with the ground, and if so, it can recognize the part of the bucket 6 that is in contact with the ground.

[0148] The transmission control unit 306 transmits information to the remote controller R40 indicating the change in the attitude of the bucket 6 (coordinates and orientation indicating its current position) (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 (S2003).

[0149] The terrain data processing unit 304 of the remote controller R40 identifies the current shape of the ground being worked on based on the received changes in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6, and updates the terrain data storage unit ST1 with terrain data indicating the current shape of the ground being worked on (S2013).

[0150] Then, the image generation unit 305 of the remote controller R40 generates mesh data representing the shape of the ground surface based on the terrain data stored in the terrain data storage unit ST1 (S2014).

[0151] Meanwhile, the acquisition unit 303 of the controller 30 acquires captured image data from the imaging device S6 and the indoor imaging device S10 (S2004).

[0152] The transmission control unit 306 of the controller 30 transmits the captured image data from the imaging device S6 and the indoor imaging device S10 to the remote controller R40 (S2005).

[0153] The image generation unit 305 of the remote controller R40 superimposes the generated mesh data onto the image data captured by the indoor imaging device S10 (S2015). The image generation unit 305 may also superimpose marks indicating the position of bucket 6 onto the image data.

[0154] The output control unit 402 of the remote controller R40 displays the captured image data with the mesh data superimposed on it on the display device D1E (S2016).

[0155] (Modification 2 of the first embodiment) The embodiments and modifications described above illustrate examples in which terrain data is stored in relative coordinates based on a predetermined position of the work machine 100. However, the embodiments and modifications described above are not limited to the configuration in which terrain data is stored in relative coordinates based on a predetermined position of the work machine 100.

[0156] Therefore, in this modified example, an example in which the work machine 100 is equipped with a positioning device will be described. The positioning device measures the position of the upper rotating body 3. The positioning device PS is, for example, a GNSS (Global Navigation Satellite System) compass and detects the position and orientation of the upper rotating body 3. The detection signals corresponding to the position and orientation of the upper rotating body 3 are taken up by the controller 30. The function of detecting the orientation of the upper rotating body 3 may be realized by an orientation sensor attached to the upper rotating body 3. In this embodiment, the positioning device PS measures the current position of the work machine 100 in a reference coordinate system that can determine its position worldwide.

[0157] A reference coordinate system is, for example, the World Geodetic System, which can determine a location on Earth. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with its origin at the Earth's center of mass, the X-axis pointing in the direction of the intersection of the Greenwich Meridian and the equator, the Y-axis pointing in the direction of 90 degrees east longitude, and the Z-axis pointing in the direction of the North Pole.

[0158] The controller 30 then stores the terrain data, which is shown in the reference coordinate system, in the terrain data storage unit ST1.

[0159] In this modified version, the controller 30 can generate image data with mesh data superimposed on it using the terrain data stored in the terrain data storage unit ST1 yesterday (the previous work day) when starting work at the start of the workday. In this modified version, it is easy to transfer terrain data.

[0160] In this modified example, the controller 30 controls the operation of the attachment AT so that the tip or bottom surface (an example of a predetermined part) of the bucket 6 is in contact with the ground to be worked on, without excavating, before the operator OP operates the work machine 100.

[0161] The controller 30 then acquires terrain data representing the shape of the ground being worked on from the detection results of the angle sensors S1 to S3, and stores the acquired terrain data in the terrain data storage unit ST1.

[0162] When the remote control room RC starts work with the work machine 100, the controller 30 transmits the image data, which has mesh data based on the terrain data stored in the terrain data storage unit ST1 superimposed on it, to the remote controller R40.

[0163] The remote controller R40 displays on the display device D1E the image data, which has the ground shape before work begins superimposed as mesh data. Therefore, the operator OP can recognize the ground shape at the start of work. Furthermore, the operator OP can start work considering the current ground shape, thereby improving safety and work efficiency.

[0164] The modified work machine 100 is not limited to a configuration that includes a positioning device. For example, when the work machine 100 moves to a predetermined position in the work area, the controller 30 sets the position information in the world coordinate system associated with that predetermined position as the current position of the work machine 100. Subsequently, the controller 30 identifies the current position in the world coordinate system based on the direction of travel and distance traveled of the work machine 100. Then, when work is performed with the attachment AT, the controller 30 identifies terrain data in the world coordinate system based on the identified current position and stores the identified terrain data in the terrain data storage unit ST1.

[0165] Furthermore, the controller 30 is not limited to storing terrain data based on a world coordinate system. For example, the controller 30 uses a predetermined position as a reference position and determines the current position from the reference position based on the travel direction and distance traveled by the work machine 100. Subsequently, when work is performed with the attachment AT, the controller 30 determines the terrain data in a coordinate system relative to the reference position based on the determined current position and stores the determined terrain data in the terrain data storage unit ST1.

[0166] (Second embodiment) The above-described embodiment explains the case in which terrain data generated by the work of the work machine 100 is used in the remote control room RC operating the work machine 100. The second embodiment explains the case in which terrain data generated by the work of multiple work machines 100 is used interchangeably by multiple work machines 100.

[0167] Figure 10 is a schematic diagram showing an example of a remote control system SYS according to the second embodiment. It includes a plurality of work machines 100A, 100B, a remote control room RC, and a management server 700.

[0168] The management server 700, multiple work machines 100A and 100B, and the remote control room RC are connected via a communication line NW to enable the transmission and reception of data.

[0169] Multiple work machines 100A and 100B may have the same configuration as work machine 100 in the embodiment described above.

[0170] The management server 700 manages the terrain data generated by each of the multiple work machines 100A and 100B.

[0171] Figure 11 is a conceptual diagram showing the relationship between a plurality of work machines 100A, 100B and a management server 700 according to the second embodiment. Figure 11 shows mesh data 2201 generated by work machine 100A and mesh data 2202 generated by work machine 100B.

[0172] The controller 30 of the work machine 100A transmits the terrain data stored in the terrain data storage unit ST1 to the management server 700. Similarly, the controller 30 of the work machine 100B transmits the terrain data stored in the terrain data storage unit ST1 to the management server 700.

[0173] The management server 700 centrally manages terrain data transmitted from multiple work machines 100A and 100B located at the work site. The management server 700 then transmits mesh data generated from the centrally managed terrain data to each of the remote control rooms RC used to operate the multiple work machines 100A and 100B. When transmitting the mesh data, the management server 700 may also transmit information that identifies the correspondence between the work machine 100 and the mesh data.

[0174] The remote controller R40 in the remote control room RC receives mesh data generated from terrain data from the management server 700. Similarly, the remote controller R40 in the remote control room RC receives image data captured from either the work machine 100A or the work machine 100B.

[0175] Then, the remote controller R40 in the remote control room RC superimposes the received mesh data onto the received image data and displays it on the display device D1E.

[0176] Figure 12 shows an example of a screen displayed by the remote controller R40 of the remote control room RC for operating the work machine 100A according to this embodiment. In the example screen shown in Figure 12, the work machine 100B is displayed in area 1301.

[0177] In the example screen shown in Figure 12, in addition to the mesh data 1311 generated based on the excavation of work machine 100A, the mesh data 1312 generated based on the excavation of work machine 100B is also displayed.

[0178] The remote controller R40 may differentiate the color of the mesh data 1311 generated based on the excavation of the work machine 100A from the color of the mesh data 1312 generated based on the excavation of the other work machine 100B. Thus, the operator OP can recognize whether or not the terrain was formed by their own operation.

[0179] Furthermore, when the remote controller R40 receives an authentication request from the operator OP, it displays mesh data 1311 and mesh data 1312 in the same color. In other words, the operator OP performs the authentication request only after confirming that the mesh data generated by the excavation of the other work machine 100B is correct. In this way, the operator OP can be made to confirm the work performed by the other work machine 100, thereby ensuring that the shape of the ground is properly understood.

[0180] Figure 13 shows an example of a screen displayed by the remote controller R40 of the remote control room RC for operating the work machine 100B according to this embodiment. In the example screen shown in Figure 13, the work machine 100A is displayed in area 1401.

[0181] In the example screen shown in Figure 13, in addition to the mesh data 1411 generated based on the excavation of the work machine 100B, the mesh data 1412 generated based on the excavation of the work machine 100A is also displayed.

[0182] The remote controller R40 may differentiate the color of the mesh data 1411 generated based on the excavation of the work machine 100B from the color of the mesh data 1311 generated based on the excavation of the other work machine 100A. Therefore, the operator OP of work machine 100B can recognize whether or not the terrain was formed by their own operation. Confirmation operations and the like on work machine 100B are the same as those on work machine 100A and will not be explained further.

[0183] In the remote control system SYS according to this embodiment, terrain data of multiple work machines 100 can be shared. Therefore, each operator OP of the multiple work machines 100 can also understand the shape of the ground in areas where work is not being performed. Thus, the operator OP can recognize the progress of work at the work site.

[0184] In the remote control system SYS according to this embodiment, the management server 700 may accept confirmation operations from an administrator after it has collected terrain data. For example, after the management server 700 has collected terrain data, and before accepting confirmation operations from an administrator, each of the remote controllers R40 corresponding to the work machines displays the mesh data generated by the excavation of each of the multiple work machines in a different color. Then, after the administrator has confirmed that there are no problems with the collected terrain data, each of the remote controllers R40 displays the mesh data generated by the excavation of each of the multiple work machines in a common color. The administrator's confirmation operation may be performed at the end of the workday or during the work.

[0185] The processing procedure performed by the remote control system SYS according to this embodiment will now be described. Figure 14 is a sequence diagram showing the overall processing flow in the remote control system SYS according to this embodiment. In the processing procedure shown in Figure 14, the remote control room RC for operating the work machine 100B will not be described.

[0186] The remote controller R40 in the remote control room RC for operating the work machine 100A receives the operation performed by the operating device R42 from the operating sensor R43 (S2501).

[0187] The transmission control unit 404 controls the transmission of the operation signal generated by the signal generation unit 403 based on the received operation to the work machine 100 (S2502).

[0188] The actuator drive unit 302 of the work machine 100A controls the operation of the attachment AT of the work machine 100A based on the operation signal received by the receiving control unit 401 (S2511).

[0189] The acquisition unit 303 of the work machine 100A acquires the change in the attitude of the bucket 6 (coordinates and orientation indicating the current position) (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 from various detection devices provided on the work machine 100A (S2512).

[0190] The transmission control unit 306 of the work machine 100A transmits information to the management server 700 indicating the change in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 (S2513).

[0191] The management server 700 identifies the current ground shape based on the change in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 received from the work machine 100A, and updates the terrain data storage unit ST1 with terrain data indicating the current ground shape (S2531).

[0192] Meanwhile, the actuator drive unit 302 of the work machine 100B controls the operation of the attachment AT of the work machine 100B based on the operation signal received by the receiving control unit 401 from the remote control room RC (S2521).

[0193] The acquisition unit 303 of the work machine 100B acquires the change in the attitude of the bucket 6 (coordinates and orientation indicating the current position) (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 from various detection devices provided on the work machine 100B (S2522).

[0194] The transmission control unit 306 of the work machine 100B transmits information to the management server 700 indicating the change in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 (S2523).

[0195] The management server 700 identifies the current ground shape based on the change in the attitude of the bucket 6 (trajectory of the bucket 6) and the direction of the reaction force acting on the bucket 6 received from the work machine 100B, and updates the terrain data storage unit ST1 with terrain data indicating the current ground shape (S2532).

[0196] The management server 700 generates mesh data showing the shape of the terrain surface for display in order to operate the work machine 100 (S2533).

[0197] The management server 700 then transmits the generated mesh data to the remote control room RC for operating the work machine 100A (S2534). The management server 700 also generates mesh data for operating work machine 100B and transmits the generated mesh data to the remote control room RC for operating work machine 100B.

[0198] The acquisition unit 303 of the work machine 100A acquires captured image data from the imaging device S6 and the indoor imaging device S10 (S2514).

[0199] The transmission control unit 306 of the work machine 100A transmits the image data captured by the imaging device S6 and the indoor imaging device S10 to the remote controller R40 (S2515).

[0200] The image generation unit 305 of the remote controller R40 superimposes the received mesh data onto the image data captured by the indoor imaging device S10 (S2503). The image generation unit 305 may also superimpose marks indicating the position of bucket 6 onto the image data.

[0201] The output control unit 402 of the remote controller R40 displays the captured image data with the mesh data superimposed on it on the display device D1E (S2504).

[0202] In the remote control system SYS according to this embodiment, terrain data generated by the work of multiple work machines 100 can be mutually utilized to recognize the shape of the ground over a wider area. Therefore, when the operator OP moves the work machine 100, they can pay attention to the shape of the ground formed by the work of other work machines, thereby improving safety.

[0203] <effect> The controller 30 and remote controller R40 according to the above-described embodiment and modified version update terrain data based on changes in the attitude of the bucket 6 and display the shape of the ground using mesh data based on the terrain data, thereby allowing the operator to grasp the three-dimensionality without using a three-dimensional distance measuring sensor such as LiDAR. Therefore, the controller 30 and remote controller R40 according to the above-described embodiment and modified version reduce costs and alleviate the burden on the operator by allowing them to grasp the three-dimensionality.

[0204] When a three-dimensional distance sensor such as LiDAR is installed, areas that cannot be detected will occur due to the shadows of objects such as raised ground. In contrast, the controller 30 and remote controller R40 according to the above-described embodiment and modified version update the terrain data representing the shape of the ground where objects have not been removed, each time the work machine 100 performs excavation work, assuming that objects that were present in the area through which the bucket 6 passed have been removed. Therefore, the remote controller R40 can display mesh data based on the updated terrain data, thereby suppressing the inability to display the shape of the ground due to the shadows of objects, as would occur when a three-dimensional distance sensor is installed. In other words, the operator OP can appropriately recognize the shape of the ground and perform work according to that shape. Therefore, work efficiency can be improved.

[0205] Preferred embodiments and modifications of the present disclosure have been described above. However, the inventions of the present disclosure are not limited to the embodiments described above. Various modifications, substitutions, etc., can be applied to the embodiments described above without departing from the scope of the inventions of the present disclosure. Furthermore, each of the features described with reference to the embodiments described above may be combined as appropriate, as long as they do not conflict technically. [Explanation of Symbols]

[0206] 100 working machines 1. Lower running body 2. Swivel mechanism 3. Upper rotating body 4 Boom 5 Arms 6 buckets S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S6 imaging device S7R, S7B, S8R, S8B, S9R, S9B Cylinder Pressure Sensor S10 Indoor imaging device T1 Communication Device ST storage device ST1 Terrain Data Storage Unit 30 controllers 301 Receiving Control Unit 302 Actuator drive unit 303 Acquisition Department 304 Terrain Data Processing Unit 305 Image Generation Unit 306 Transmission Control Unit 31 Proportional valve RC Remote Control Room T2 Communication Device D1E display device R40 Remote Controller 401 Receiving Control Unit 402 Output Control Unit 403 Signal Generation Unit 404 Transmission Control Unit R42 operating device R43 Operation Sensor 700 Management Server

Claims

1. A work machine comprising: a lower traveling body; an upper rotating body mounted on the lower traveling body so as to be rotatable; an attachment having an end attachment connected to the upper rotating body 3; a posture detection device capable of determining the posture of the attachment; and an imaging device capable of imaging the area around the work machine body. Display device and A control device that acquires information representing the shape of the ground that is the work target of the work machine based on the change in the posture of the attachment identified by the detection result of the posture detection device, and displays the shape of the ground represented by the acquired information on the display device, A remote control system for industrial machinery, equipped with the following features.

2. When the control device performs work on the ground using the attachment, it acquires the information representing the shape of the ground, assuming that any objects that were present in the first region through which the end attachment passed have been removed from the first region. A remote control system for a work machine according to claim 1.

3. The control device, after the object in the first region has been removed by the end attachment, displays the surface of the second region, from which the object has not been removed, as a mesh shape. A remote control system for a work machine according to claim 2.

4. The device further includes a pressure detection device that indicates the pressure related to the aforementioned attachment, The control device acquires information representing the shape of the ground, which is the work target of the work machine, based on the change in the posture of the attachment identified by the posture detection device and the pressure related to the attachment detected by the pressure detection device. A remote control system for a work machine according to claim 1.

5. The control device suppresses the acquisition of information representing the shape of the ground based on the change in the posture of the attachment when the pressure detected by the pressure detection device is lower than a predetermined threshold. A remote control system for a work machine according to claim 4.

6. The control device displays the vertically downward position of the end attachment relative to the shape of the ground on the display device. A remote control system for a work machine according to claim 1.

7. The control device identifies the portion of the end attachment that is in contact with the ground in accordance with the work performed by the attachment, and acquires the information representing the shape of the ground based on the change in the position of the portion. A remote control system for a work machine according to claim 1.

8. The control device is capable of changing at least a portion of the information representing the shape of the ground, which is the work target of the work machine, in accordance with the operation input from the operating device. A remote control system for a work machine according to claim 1.

9. The control device operates the work machine so that a predetermined part of the attachment contacts the ground, acquires information representing the shape of the ground which is the work target of the work machine, and displays the shape of the ground represented by the acquired information on the display device. A remote control system for a work machine according to claim 1.

10. The control device acquires the information representing the shape of the ground without using a three-dimensional shape detection device capable of detecting the three-dimensional shape of objects around the work machine. A remote control system for a work machine according to claim 1.

Images