Systems and display devices for industrial machinery

Abstract

To provide a system for work machines that enables more efficient setup and configuration. [Solution] The work support system SYS includes a spatial recognition device S6 that recognizes the ground shape around the work machine 100, a client-side controller 60, and a third display device DS3. The client-side controller 60 causes the work machine 100 to perform a predetermined operation by moving the work machine 100 according to a preset procedure, and predicts the ground shape after the predetermined operation is completed based on the current ground shape recognized by the spatial recognition device S6 and the content of the predetermined operation to be performed by the work machine 100. The third display device DS3 displays a predicted ground shape image, which is an image representing the ground shape predicted by the client-side controller 60.

Description

Technical Field

[0001] The present disclosure relates to a system and a display device for a working machine.

Background Art

[0002] Conventionally, a support device for a working machine capable of generating a trajectory of a working part of the working machine according to an operator's input is known (see Patent Document 1). This support device is configured to generate a trajectory of the working part of the working machine based on the current ground shape around the working machine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, this support device is not configured such that the operator can generate in advance another trajectory representing the movement of the next working part that starts after moving the working part along one generated trajectory. That is, the generation of another trajectory is not performed until the movement of the working part along one trajectory is completed. This is because the ground shape at the end point is unknown until the movement of the working part along one trajectory is completed. Therefore, the operator needs to set the next trajectory each time the movement of the working part along the trajectory is completed, and there is a risk that the time required to complete the entire series of operations will be prolonged.

[0005] Therefore, it is desirable to provide a system for a working machine that can more efficiently realize the setup of the setup including the setting of the trajectory of the working part.

Means for Solving the Problems

[0006] A system for a work machine according to one embodiment of the present disclosure is a system for a work machine comprising a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and an attachment having a work area attached to the upper rotating body, the system comprising a spatial recognition device for recognizing the ground shape around the work machine, a control device, and a display device, wherein the control device is configured to cause the work machine to perform a predetermined operation by moving the work machine according to a preset procedure, and is configured to predict the ground shape after the predetermined operation is completed based on the current ground shape recognized by the spatial recognition device and the content of the predetermined operation performed by the work machine, and the display device is configured to display a predicted ground shape image which is an image representing the ground shape predicted by the control device. [Effects of the Invention]

[0007] The aforementioned systems for work machines enable more efficient setup. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing an example configuration of a work support system according to the present disclosure. [Figure 2] Figure 1 is a side view of the work machine that constitutes the work support system shown. [Figure 3] This figure shows an example of the configuration of the drive control system for the work machine shown in Figure 2. [Figure 4] This is a block diagram showing an example configuration of a work support system. [Figure 5] This flowchart shows an example of the workflow for work support processing. [Figure 6] This figure shows an example of a support selection screen. [Figure 7] This figure shows an example of a screen for assisting with excavation and soil removal operations. [Figure 8] This figure shows an example of a screen for assisting with excavation and loading operations. [Figure 9] This figure shows an example of a screen for assisting with slope shaping operations. [Figure 10] This diagram shows an example of a support screen for excavation and transfer work. [Figure 11] This figure shows an example of a trenching work support screen. [Figure 12] This figure shows an example of a screen for assisting with excavation and loading operations. [Figure 13] This figure shows an example of a screen for assisting with slope shaping work. [Figure 14] This diagram shows the relationship between the time required for setting up the procedure and the time required for the actual operation according to that procedure. [Figure 15] This diagram shows the relationship between the time required for setting up the plan and the time required for the actual work to be carried out according to that plan. [Figure 16] This is a flowchart illustrating an example of the course correction process. [Figure 17] This figure shows an example of an actual ground shape image, a predicted ground shape image, and a difference image. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. Furthermore, the embodiments described below are illustrative and not limiting to the invention, and not all features or combinations thereof described as embodiments are necessarily essential to the invention. In each drawing, identical or corresponding components are denoted by the same or corresponding reference numerals, and their descriptions may be omitted.

[0010] First, with reference to Figure 1, a work support system SYS, which is a system for work machines according to an embodiment of this disclosure, will be described. Figure 1 is a schematic diagram showing an example configuration of the work support system SYS.

[0011] The work support system SYS is a system that supports the work by the work machine 100. In the illustrated example, the work support system SYS mainly includes a work machine 100, a management device 200, and a support device 300. Each of the work machine 100, the management device 200, and the support device 300 is equipped with a communication device TD and is directly or indirectly connected to each other via an information communication network IN such as a mobile phone communication network, a satellite communication network, or a short-range wireless communication network. Each of the work machine 100, the management device 200, and the support device 300 constituting the work support system SYS may be one unit or a plurality of units. In the illustrated example, the work support system SYS includes one work machine 100, one management device 200, and one support device 300.

[0012] The work machine 100 has an upper revolving body 3 mounted thereon via a slewing mechanism 2 on a crawler-type lower traveling body 1 so as to be slewed. A boom 4 as a working element is attached to the upper revolving body 3. An arm 5 as a working element is attached to the tip of the boom 4, and a bucket 6 as a working element and an end attachment is attached to the tip of the arm 5. The boom 4 is driven by a boom cylinder 7, the arm 5 is driven by an arm cylinder 8, and the bucket 6 is driven by a bucket cylinder 9. The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment which is an example of an attachment.

[0013] A cabin 10 as a driver's cab is provided on the upper revolving body 3, and a power source such as an engine 11 is mounted thereon. Further, a shovel-side controller 30, a display device DS (first display device DS1), an input device ID (first input device ID1), a communication device TD (first communication device TD1), a positioning device PD, etc. are attached to the upper revolving body 3. Note that the power source may be an electric motor driven by a battery or an external power source.

[0014] The excavator-side controller 30 is configured to control the work machine 100. In the illustrated example, the excavator-side controller 30 is an example of a processing circuit and consists of a computer equipped with a CPU, RAM, NVRAM, and ROM. The excavator-side controller 30 reads the program corresponding to each functional element from ROM and loads it into RAM, causing the CPU to execute the corresponding processing. However, each functional element may be composed of hardware, or it may be composed of a combination of software and hardware.

[0015] The management device 200 is a device that manages the work performed by the work machine 100. In the illustrated example, the management device 200 is a server computer installed in a management center or the like, located away from the work site (work machine 100), and includes a display device DS (second display device DS2), an input device ID (second input device ID2), a communication device TD (second communication device TD2), and a server-side controller 50. The management device 200 may also be a portable computer (for example, a notebook PC, tablet PC, or a mobile terminal device such as a smartphone).

[0016] The server-side controller 50 is configured to control the management device 200. In the illustrated example, the server-side controller 50 is an example of a processing circuit and consists of a computer equipped with a CPU, RAM, NVRAM, and ROM. The server-side controller 50 reads programs corresponding to each functional element from ROM and loads them into RAM, causing the CPU to execute the corresponding processing. However, each functional element may be composed of hardware, or it may be composed of a combination of software and hardware.

[0017] The support device 300 is a device that assists in the work performed by the work machine 100. In the illustrated example, the support device 300 is a portable client computer (e.g., a notebook PC, tablet PC, or smartphone) carried by a worker working around the work machine 100, and includes a display device DS (third display device DS3), an input device ID (third input device ID3), a communication device TD (third communication device TD3), and a client-side controller 60. The support device 300 may also function as a server.

[0018] The display device DS is configured to display various types of information. In the illustrated example, the display device DS is a liquid crystal display, but it may also be an XR (augmented reality) goggle or the like.

[0019] Next, with reference to Figure 2, the details of the work machine 100 will be described. Figure 2 is a side view of an example of the work machine 100, which is a shovel (excavator). The work machine 100 may be a material handling machine or a lifting magnet machine. In the illustrated example, the upper rotating body 3 is rotatably mounted on the lower traveling body 1 of the work machine 100 via a slewing mechanism 2. A boom 4 is attached to the upper rotating body 3, an arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5. The end attachment may be a breaker, magnet, or grapple, etc.

[0020] In the illustrated example, the lower vehicle 1 is a crawler-type vehicle having a crawler 1C, including a left crawler 1CL and a right crawler (not visible in Figure 2). The crawler 1C is driven by a travel actuator DA. However, the lower vehicle 1 may also be a wheeled vehicle having four wheels. In this case, the four wheels may be configured to be steered and rotated independently.

[0021] The boom 4, arm 5, and bucket 6 each constitute an excavation attachment, which is an example of an attachment AT, and are driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9, which are hydraulic cylinders, which are an example of a work actuator WA. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6.

[0022] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor and can detect the boom angle, which is the rotation angle of the boom 4 relative to the upper slewing body 3. The boom angle is smallest when the boom 4 is lowered to its lowest position, and increases as the boom 4 is raised.

[0023] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the arm angle, which is the rotation angle of the arm 5 relative to the boom 4. The arm angle is smallest when the arm 5 is closed to its shortest extent, and increases as the arm 5 is opened.

[0024] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the bucket angle, which is the rotation angle of the bucket 6 relative to the arm 5. The bucket angle is smallest when the bucket 6 is closed to its fullest extent, and increases as the bucket 6 is opened.

[0025] The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder HC, or a rotary encoder that detects the rotation angle around the connecting pin. The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 constitute an attitude sensor AS that detects the attitude of the excavation attachment.

[0026] The upper rotating body 3 is equipped with a cabin 10 as the driver's cab, an engine 11, a microphone array A1, a positioning device PD, an aircraft tilt sensor S4, a rotational velocity sensor S5, a spatial recognition device S6, an orientation detection sensor S7, a rotation actuator SA, and a communication device TD (first communication device TD1), among others.

[0027] A shovel-side controller 30 is installed inside the cabin 10. The cabin 10 also contains a driver's seat, operating devices 26, and a display device DS (first display device DS1), etc. The shovel-side controller 30 is a control device that performs various calculations. For example, the shovel-side controller 30 is located inside the cabin 10 and controls the drive of the work machine 100. The functions of the shovel-side controller 30 may be realized by any hardware, software, or a combination thereof. For example, the shovel-side controller 30 is mainly composed of a microcomputer including a CPU, memory (volatile memory) such as RAM, non-volatile memory such as ROM, and various input / output interface devices. The shovel-side controller 30 may realize various functions by executing various programs installed in the non-volatile memory on the CPU, for example.

[0028] Engine 11 is an example of a power source for the work machine 100. In the illustrated example, engine 11 is a diesel engine and is mounted at the rear of the upper slewing body 3. The output shaft of engine 11 is connected to the input shafts of the main pump 14 and the pilot pump 15, respectively. Specifically, engine 11 rotates at a constant speed at a preset target rotational speed under direct or indirect control by the shovel-side controller 30, driving the main pump 14 and the pilot pump 15, etc. The power source for the work machine 100 may also be a battery-powered electric motor. That is, the work machine 100 may be a hybrid work machine or an electric work machine.

[0029] The machine tilt sensor S4 is configured to detect the tilt of the upper rotating body 3 with respect to a predetermined plane. In the illustrated example, the machine tilt sensor S4 is an acceleration sensor that detects the tilt angle of the upper rotating body 3 around the longitudinal axis and the tilt angle around the left-right axis with respect to the horizontal plane. The longitudinal axis and left-right axis of the upper rotating body 3 are, for example, orthogonal to each other and pass through a central point which is a point on the rotation axis PV of the work machine 100.

[0030] The rotational angular velocity sensor S5 is configured to detect the rotational angular velocity of the upper rotating body 3. In this embodiment, the rotational angular velocity sensor S5 is a gyro sensor. The rotational angular velocity sensor S5 may also be a resolver or a rotary encoder, etc. The rotational angular velocity sensor S5 may also be configured to detect the rotational speed. The rotational speed may also be calculated from the rotational angular velocity.

[0031] The spatial recognition device S6 is configured to acquire images of the area surrounding the work machine 100. In the illustrated example, the spatial recognition device S6 includes a front camera S6F that captures the space in front of the work machine 100, a left camera S6L that captures the space to the left of the work machine 100, a right camera S6R that captures the space to the right of the work machine 100, and a rear camera S6B that captures the space behind the work machine 100.

[0032] The spatial recognition device S6 is, for example, a monocular camera having an image sensor such as a CCD or CMOS, and may output the captured image to a display device DS (first display device DS1). In the illustrated example, the spatial recognition device S6 includes a rear camera S6B, a front camera S6F, a left camera S6L, and a right camera S6R. The front camera S6F is mounted, for example, on the roof of the cabin 10. The left camera S6L is mounted on the upper left end of the upper surface of the upper rotating body 3. The right camera S6R is mounted on the upper right end of the upper surface of the upper rotating body 3. The rear camera S6B is mounted on the upper rear end of the upper surface of the upper rotating body 3.

[0033] The spatial recognition device S6, located at the position described above, can photograph objects in the vicinity of the work machine 100. The spatial recognition device S6 may be a camera capable of recognizing the distance to the object being photographed (for example, an RGBD camera or a stereo camera). Alternatively, the spatial recognition device S6 may be a LiDAR.

[0034] The positioning device PD is configured to acquire information regarding the position of the work machine 100. In this embodiment, the positioning device PD is configured to measure the position and orientation of the work machine 100 in a reference coordinate system. Specifically, the positioning device PD is a GNSS (Global Navigation Satellite System) receiver incorporating an electronic compass, and measures the latitude, longitude, and altitude of the current position of the work machine 100, as well as the orientation of the work machine 100 (upper rotating body 3). In the illustrated example, the reference coordinate system is the World Geodetic System. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with its origin at the center of gravity of the Earth, the X-axis pointing in the direction of the intersection of the Greenwich Meridian and the equator, the Y-axis pointing in the direction of 90 degrees east longitude, and the Z-axis pointing in the direction of the North Pole.

[0035] The orientation detection sensor S7 is configured to detect information regarding the relative relationship between the orientation of the upper rotating body 3 and the orientation of the lower traveling body 1. For example, the orientation detection sensor S7 may consist of a combination of a geomagnetic sensor attached to the lower traveling body 1 and a geomagnetic sensor attached to the upper rotating body 3. Alternatively, the orientation detection sensor S7 may consist of a combination of a lower positioning device (GNSS receiver incorporating an electronic compass) attached to the lower traveling body 1 and a positioning device PD (GNSS receiver incorporating an electronic compass) attached to the upper rotating body 3. Alternatively, the orientation detection sensor S7 may be a combination of a geomagnetic sensor or positioning device PD (GNSS receiver incorporating an electronic compass) attached to the upper rotating body 3 and a rotary encoder or rotary position sensor. Alternatively, in a configuration in which the upper rotating body 3 is driven to rotate by a rotating motor generator, which is an example of a rotating actuator, the orientation detection sensor S7 may consist of a combination of a geomagnetic sensor or positioning device PD (GNSS receiver incorporating an electronic compass) attached to the upper rotating body 3 and a resolver.

[0036] Alternatively, the orientation detection sensor S7 may consist of a camera mounted on the upper slewing body 3. In this case, the orientation detection sensor S7 applies known image recognition processing to the image (input image) captured by the camera mounted on the upper slewing body 3 to detect the image of the lower traveling body 1, thereby identifying the longitudinal direction, which is the direction along the front-rear axis of the lower traveling body 1. The front-rear axis and left-right axis of the lower traveling body 1 are, for example, orthogonal to each other and pass through a center point, which is a point on the slewing axis PV of the work machine 100. The orientation detection sensor S7 then derives the angle formed between the front-rear axis of the upper slewing body 3 and the front-rear axis of the lower traveling body 1. The direction of the front-rear axis of the upper slewing body 3 is derived from the camera mounting position. In particular, if the lower traveling body 1 is a crawler-type traveling body, the crawler 1C protrudes from the upper slewing body 3, so the orientation detection sensor S7 can determine the longitudinal direction of the lower traveling body 1 by detecting the image of the crawler 1C. In this case, the orientation detection sensor S7 may be integrated into the shovel-side controller 30. Alternatively, the camera may be a spatial recognition device S6.

[0037] Alternatively, the orientation detection sensor S7 may be a combination of a positioning device PD (GNSS receiver incorporating an electronic compass) attached to the upper rotating body 3 and a rotation angular velocity sensor S5 configured to detect the rotation angle.

[0038] The communication device TD (first communication device TD1) is configured to control communication with equipment located outside the work machine 100. In this embodiment, the first communication device TD1 is configured to control communication between the first communication device TD1 and equipment located outside the work machine 100 via a wireless communication network. The first communication device TD1 may include, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), or a satellite communication module for connecting to a satellite communication network.

[0039] Furthermore, the first communication device TD1 may be configured to control, for example, wireless communication between an external GNSS surveying system and the work machine 100.

[0040] The microphone array A1 has multiple microphones and is configured to collect sounds generated around the work machine 100. In the illustrated example, the microphone array A1 consists of multiple microphones attached to the upper rotating body 3.

[0041] Figure 3 shows an example of the drive control system configuration for the work machine 100 shown in Figure 2. In Figure 3, the mechanical power transmission system is shown by double lines, the hydraulic fluid lines by thick solid lines, the pilot lines by dashed lines, and the electric drive and control system by dotted lines.

[0042] The drive system of the work machine 100 according to this embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 includes a travel hydraulic motor (left travel hydraulic motor 1L and right travel hydraulic motor 1R) as a travel actuator DA, a slewing hydraulic motor 2A as a slewing actuator SA, and a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9 as work actuators WA.

[0043] The regulator 13 is configured to control the discharge rate of the main pump 14. In the illustrated example, the regulator 13 adjusts the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the shovel-side controller 30.

[0044] The main pump 14 is mounted on the upper slewing body 3, similar to the engine 11, and supplies hydraulic fluid to the control valve unit 17 through the hydraulic fluid line. The main pump 14 is also driven by the engine 11. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump, and under the control of the excavator-side controller 30, the piston stroke length is adjusted by adjusting the tilt angle of the swash plate by the regulator 13, thereby controlling the discharge flow rate (discharge pressure).

[0045] The control valve unit 17 is a hydraulic control device that controls the hydraulic system in the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171 to 176 as spool valves. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control, for example, the flow rate of hydraulic fluid flowing from the main pump 14 to the hydraulic actuators and the flow rate of hydraulic fluid flowing from the hydraulic actuators to the hydraulic fluid tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-travel hydraulic motor 1L, a right-travel hydraulic motor 1R, and a slewing hydraulic motor 2A. More specifically, control valve 171 corresponds to the left-travel hydraulic motor 1L, control valve 172 corresponds to the right-travel hydraulic motor 1R, and control valve 173 corresponds to the slewing hydraulic motor 2A. Furthermore, control valve 174 corresponds to bucket cylinder 9, control valve 175 corresponds to boom cylinder 7, and control valve 176 corresponds to arm cylinder 8.

[0046] The pilot pump 15 is an example of a pilot pressure generating device and is configured to supply hydraulic fluid to hydraulic control equipment via a pilot line. In this embodiment, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pressure generating device may be implemented by the main pump 14. That is, the main pump 14 may have the function of supplying hydraulic fluid to the control valve unit 17 via a hydraulic fluid line, as well as the function of supplying hydraulic fluid to various hydraulic control equipment via a pilot line. In this case, the pilot pump 15 may be omitted.

[0047] The operating device 26 is a device used by the operator inside the cabin 10 to operate the actuators. The actuators include at least one of a hydraulic actuator and an electric actuator. In the illustrated example, the operating device 26 includes an operating lever, a travel lever, and a travel pedal. The operating levers include a left operating lever for slewing and arm operation, and a right operating lever for boom and bucket operation.

[0048] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In this embodiment, the discharge pressure sensor 28 outputs the detected value to the shovel-side controller 30.

[0049] The operation sensor 29 is configured to detect the operator's actions using the operating device 26. In this embodiment, the operation sensor 29 detects the operating direction and amount of the operating device 26 corresponding to each actuator and outputs the detected values ​​to the shovel-side controller 30. Specifically, the operation sensor 29 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle around the pivot axis of the operating lever. The operation sensor 29 may also be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor.

[0050] In the illustrated example, the excavator-side controller 30 controls the opening area of ​​the solenoid valve 31 according to the output of the operation sensor 29. The excavator-side controller 30 then applies the pressure from the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The hydraulic fluid pressure (pilot pressure) acting on each pilot port is, in principle, the pressure corresponding to the operating direction and amount of the operating device 26 corresponding to each hydraulic actuator. Thus, the operating device 26 is configured to apply the pressure from the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0051] The solenoid valve 31, which functions as a control valve for machine control, is located in an oil passage connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow area of ​​the oil passage. In the illustrated example, the solenoid valve 31 operates in response to control commands output by the shovel-side controller 30. Therefore, the shovel-side controller 30 can apply the pressure of the hydraulic fluid discharged by the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the solenoid valve 31, regardless of the operator's operation of the operating device 26, thereby achieving the desired pilot pressure. In the illustrated example, the shovel-side controller 30 is configured to provide feedback control of the pilot pressure based on the output of the pilot pressure sensor 32.

[0052] With this configuration, the excavator-side controller 30 can operate the hydraulic actuator corresponding to a specific operating device 26 not only when an operation is being performed on that specific operating device 26, but also when no operation is being performed on that specific operating device 26.

[0053] Furthermore, the shovel-side controller 30 is configured to perform various functions in addition to controlling the pilot pressure. For example, the shovel-side controller 30 can set a target rotational speed based on a work mode or the like, which is set in advance by a predetermined operation by the operator, and perform drive control to keep the engine 11 rotating at a constant speed.

[0054] Furthermore, the shovel-side controller 30 can, for example, output control commands to the regulator 13 as needed, thereby changing the discharge rate of the main pump 14.

[0055] Furthermore, the shovel-side controller 30 can perform control related to a machine guidance function, for example, to guide the manual operation of the work machine 100 by the operator via the operating device 26. The shovel-side controller 30 can also perform control related to a machine control function, for example, to automatically assist the manual operation of the attachment AT by the operator via the operating device 26. In the illustrated example, the machine guidance function semi-automatically operates the work machine 100 so that the working part (represented as a single point), such as the tip of the bucket 6, moves along a set trajectory, thereby guiding the manual operation of the work machine 100. In this case, the work machine 100 may be a remotely operated work machine. The machine control function automatically operates the work machine 100 so that the working part moves along a set trajectory. In this case, the work machine 100 may be an unmanned machine. Note that "trajectory" refers to the line in three-dimensional space followed by the working part (represented as a single point), such as the tip of the bucket 6.

[0056] Furthermore, some of the functions of the excavator-side controller 30 may be implemented by other controllers (control devices). In other words, the functions of the excavator-side controller 30 may be implemented by multiple controllers. For example, the machine guidance function and the machine control function may be implemented by a dedicated controller (control device).

[0057] Next, an example of the operation of the work support system SYS will be described with reference to Figures 4 to 7. Figure 4 is a block diagram showing an example of the configuration of the work support system SYS. Figure 5 is a flowchart showing an example of the flow of processing to support the work of the work machine 100 (hereinafter referred to as "work support processing"). Figures 6 and 7 are diagrams showing examples of screens displayed on the display device DS. Specifically, Figure 6 is a diagram showing an example of a support content selection screen displayed on the third display device DS3 in the support device 300, and Figure 7 is a diagram showing an excavation and soil removal operation support screen, which is an example of an operation (work) support screen displayed on the third display device DS3. The control devices that constitute the work support system SYS (at least one of the shovel-side controller 30, server-side controller 50, and client-side controller 60) execute this work support processing when predetermined support start conditions are met. The predetermined support start conditions are, for example, when a predetermined button is pressed.

[0058] Specifically, as shown in Figure 5, the client-side controller 60 of the support device 300 displays the support content selection screen (see Figure 6) on the third display device DS3 (step ST1). In the illustrated example, the operator of the support device 300 displays the support content selection screen on the third display device DS3 by touching a predetermined software button image using the third input device ID3 (touch panel).

[0059] The support selection screen is a screen displayed to allow the operator to select the support content. In the illustrated example, the support selection screen includes multiple software button images BT, as shown in Figure 6. Specifically, the software button images BT include a first button image BT1 for selecting excavation and soil removal, a second button image BT2 for selecting excavation and loading, a third button image BT3 for selecting slope shaping, a fourth button image BT4 for selecting shoveling and re-transfer work, a fifth button image BT5 for selecting trenching work, a sixth button image BT6 for selecting excavation and loading work, and a seventh button image BT7 for selecting slope shaping work.

[0060] Subsequently, the client-side controller 60 displays an operation (task) support screen (step ST2). In the illustrated example, the client-side controller 60 displays the corresponding operation (task) support screen when any of the first button image BT1 to the seventh button image BT7 is touched.

[0061] Here, "work" refers to various operations performed by the work machine 100, such as excavation and loading, slope shaping, plowing and reloading, or trenching. "Action" refers to a specific action performed by the work machine 100 to perform a particular operation, such as an excavation and loading action performed by the work machine 100 to perform an excavation and loading operation, or a slope shaping action performed by the work machine 100 to perform a slope shaping operation. In other words, a particular "work" is achieved by one or more specific "actions".

[0062] Specifically, when the first button image BT1 is touched on the support content selection screen, the client-side controller 60 displays an example of an operation (work) support screen, which is the excavation and soil removal operation support screen (Figure 7).

[0063] The excavation and soil removal operation support screen is a screen displayed to allow the operator to input information regarding support for excavation and soil removal operations. Excavation and soil removal operations typically consist of a combination of excavation, rotation, and soil removal. In the illustrated example, the excavation and soil removal operation support screen includes an actual ground shape image AG, a software button image BT, an operation image MV, a predicted ground shape image PG, and a text image TX, as shown in Figure 7. If a device other than a touch panel (e.g., a mouse or trackball) is used as the third input device ID3, the excavation and soil removal operation support screen may also include a cursor image.

[0064] The actual ground shape image AG is an image representing the actual shape of the ground around the work machine 100. In the illustrated example, the actual ground shape image AG is a contour map (contour diagram) image generated based on the output of the spatial recognition device S6 mounted on the work machine 100. Each pixel constituting the actual ground shape image AG has a pixel value that represents the height relative to a pre-set virtual horizontal plane, which is a reference plane. In Figure 7, for clarity, the contour lines are omitted, and a finer dot pattern is applied to higher positions relative to the reference plane, while a finer cross pattern is applied to lower positions relative to the reference plane. The same applies to the predicted ground shape image PG.

[0065] In the illustrated example, the client-side controller 60 receives the actual ground shape image AG generated by the shovel-side controller 30 based on the output of the spatial recognition device S6 via the third communication device TD3. Specifically, the actual ground shape image AG generated by the shovel-side controller 30 is transmitted to the management device 200 via the first communication device TD1, and also to the support device 300 via the second communication device TD2. The actual ground shape image AG generated by the shovel-side controller 30 may also be transmitted directly to the support device 300. Furthermore, the actual ground shape image AG may be generated by the server-side controller 50 or by the client-side controller 60.

[0066] Furthermore, in the excavation and soil removal operation support screen shown in Figure 7, the software button image BT includes the area setting button image BT11, the viewpoint switching button image BT12, the setting complete button image BT13, the work complete button image BT14, the autonomous operation button image BT15, and the MG button image BT16.

[0067] The area setting button image BT11 is a software button image used to set a specific area in the ground shape image (actual ground shape image AG and predicted ground shape image PG) as the work area. In the illustrated example, the operator can, for example, touch the area setting button image BT11, then touch and drag their finger to a first point on the ground shape image, and release their finger at a second point on the ground shape image, thereby setting a rectangular area with the line segment connecting the first and second points as the diagonal as the work area. However, the client-side controller 60 may be configured to allow the operator to set an area as the work area by any other method.

[0068] The viewpoint switching button image BT12 is a software button image used to change the appearance of the ground shape images (actual ground shape image AG and predicted ground shape image PG). In the illustrated example, the operator can switch between a contour map image and a cross-sectional view image of the ground shape image displayed to the left of the viewpoint switching button image BT12 each time they touch the button.

[0069] The "Setting Complete" button image BT13 is a software button image used to complete the settings for the work target. In the illustrated example, the operator can complete the setting of the track's starting point or the work target area by selecting a point or area on the ground shape image and then touching the "Setting Complete" button image BT13.

[0070] The "Work Complete" button image BT14 is a software button image used by the operator to notify the work support system SYS of the completion of a task. In the illustrated example, the operator can, for example, set the starting point of the trajectory on the ground shape image displayed to the left of the "Work Complete" button image BT14, and then touch the "Work Complete" button image BT14 to notify the work support system SYS that the work is complete when the work area reaches the end point of that trajectory.

[0071] The autonomous operation button image BT15 is a software button image used to initiate autonomous operation (operation by machine control function) of the work machine 100. In the illustrated example, the operator can, for example, after completing the trajectory settings, touch the autonomous operation button image BT15 to transmit trajectory information to the work machine 100 and initiate the machine control function of the work machine 100, which enables the movement of the work part along that trajectory.

[0072] The MG button image BT16 is a software button image used to initiate the semi-autonomous operation (operation using the machine guidance function) of the work machine 100. In the illustrated example, the operator can, for example, after completing the trajectory settings, touch the MG button image BT16 to transmit trajectory information to the work machine 100 and initiate the machine guidance function of the work machine 100, which enables the movement of the work part along that trajectory.

[0073] The motion image MV is an image used to explain to the operator in advance the movement of the work machine 100, which is realized by the work support system SYS. In the illustrated example, the motion image MV represents the movement of the work machine 100 realized by the work support system SYS by simultaneously displaying the state (position and orientation) of the work machine 100 at multiple points in time (three in the illustrated example). The motion image MV may be a still image as shown, a moving image such as an animation image, or a stop-motion image. In the illustrated example, the motion image MV includes the first motion image MV1 corresponding to the first excavation and soil removal operation, the second motion image MV2 corresponding to the second excavation and soil removal operation, and the third motion image MV3 corresponding to the third excavation and soil removal operation. In the illustrated example, the estimated amount of excavation (the amount of soil, etc., excavated in one excavation operation) is displayed on the lower right side of each of the three motion images MV.

[0074] The predicted ground shape image PG is an image representing the future shape of the ground around the work machine 100. In the illustrated example, the predicted ground shape image PG is a contour map image generated based on the set trajectory and the actual ground shape image AG. Specifically, the predicted ground shape image PG includes a first predicted ground shape image PG1 representing the ground shape after the first excavation and soil removal operation is completed, and a second predicted ground shape image PG2 representing the ground shape after the second excavation and soil removal operation is completed.

[0075] By viewing the excavation and soil removal operation support screen, the operator can recognize that when the first excavation and soil removal operation, represented by the first operation image MV1, is performed on the current ground shape represented by the actual ground shape image AG, the ground shape represented by the first predicted ground shape image PG1 will be obtained. Furthermore, by viewing the excavation and soil removal operation support screen, the operator can recognize that when the second excavation and soil removal operation, represented by the second operation image MV2, is performed on the ground shape represented by the first predicted ground shape image PG1, the ground shape represented by the second predicted ground shape image PG2 will be obtained.

[0076] Text image TX is an image containing text to convey various information to the operator. In the illustrated example, text image TX includes a first text image TX1 and a second text image TX2. The first text image TX1 represents the number of excavation and soil removal operations required to complete the work to achieve the target construction surface (the ground shape (three-dimensional shape) achieved by a series of excavation and soil removal operations). The second text image TX2 represents the (expected) time when the work to achieve the target construction surface will be completed. Note that the second text image TX2 may also be a text image representing the expected time required to complete the work to achieve the target construction surface.

[0077] Referring again to Figure 5, after displaying the operation (work) support screen, the client-side controller 60 determines whether or not the content of the operation (work) has been entered (step ST3). In the example shown in Figure 7, the client-side controller 60 determines that the content of the operation (work) has been entered when the part represented by the circle C1 (which is not actually displayed) in the actual ground shape image AG is selected (touched) as the starting point of the trajectory. Alternatively, the client-side controller 60 may determine that the content of the operation (work) has been entered when a part of the area in the actual ground shape image AG is set as the target of the excavation and soil removal operation. The client-side controller 60 may also calculate a provisional trajectory as an initial value based on the current ground shape and the support content when the support content is selected by the operator. In this case, the client-side controller 60 may start the machine control function using the provisional trajectory when the autonomous operation button image BT15 is touched without the operator selecting the starting point of the trajectory, etc. The same applies when the machine guidance function is started when the MG button image BT16 is touched. In other words, the operator can change the provisional track to the track to be actually used by taking interventions such as selecting the starting point of the track. Alternatively, the operator may set the track by tracing a desired path on the actual ground shape image AG with their finger. In this case, the client-side controller 60 may calculate the track based on the path traced by the operator. Or, the operator may set a desired area as the excavation target area by tracing an arbitrary area on the actual ground shape image AG with their finger. In this case, the client-side controller 60 may calculate the track based on the set excavation target area.

[0078] If it is determined that the details of the operation (task) have been entered (YES in step ST3), the client-side controller 60 calculates the trajectory (step ST4).

[0079] In the illustrated example, the client-side controller 60 calculates the trajectory for the first excavation and soil removal operation based on the position (three-dimensional coordinates) of the starting point of the trajectory entered by the operator and the current ground shape (three-dimensional shape). In this case, the client-side controller 60 may use information such as the maximum excavation amount (maximum amount of soil, etc., to be excavated in one excavation operation) or the current state of the work machine 100 (position or posture, etc.). Alternatively, the client-side controller 60 may calculate the trajectory for the first excavation and soil removal operation based on the position (three-dimensional coordinates) of the starting point of the trajectory entered by the operator, the current ground shape (three-dimensional shape), and the position (three-dimensional coordinates) of the ending point of the trajectory entered by the operator. Alternatively, the client-side controller 60 may calculate the trajectory along a path set by the operator tracing with their finger, or calculate the trajectory so that the area selected by the operator tracing with their finger can be excavated.

[0080] Alternatively, the client-side controller 60 may calculate the trajectory of the excavation and soil removal operation based on the position of the starting point of the trajectory (three-dimensional coordinates) input by the operator, the current ground shape (three-dimensional shape), and the target construction surface (three-dimensional shape). For example, the client-side controller 60 may calculate the trajectory of the excavation and soil removal operation to achieve the target construction surface with the fewest number of excavation and soil removal operations. Alternatively, the client-side controller 60 may calculate the trajectory of each excavation and soil removal operation so as to minimize the variation in the amount of soil excavated (excavation volume) by each excavation and soil removal operation.

[0081] Subsequently, the client-side controller 60 estimates the ground shape after the operation (work) is completed (step ST5). Specifically, the client-side controller 60 estimates the ground shape formed by moving the work area along the trajectory calculated in step ST4 to the end of that trajectory as the predicted ground shape (first predicted ground shape). Typically, the client-side controller 60 estimates the predicted ground shape (first predicted ground shape) by removing the soil and other debris in the space through which the bucket 6, moving along the trajectory calculated in step ST4, passes from the current ground shape. This estimation may also be realized by a predictor, which is a functional collection of elements that constitute the client-side controller 60. The predictor can be realized by hardware, software, or a combination thereof. Alternatively, the client-side controller 60 may estimate the predicted ground shape using a trained model obtained by machine learning the movement of soil and debris predicted by particle simulation.

[0082] Subsequently, the client-side controller 60 displays the estimation results (step ST6). In the illustrated example, the client-side controller 60 displays the first predicted ground shape image PG1, which is a contour map image representing the estimated predicted ground shape (first predicted ground shape), and the first motion image MV1, which is an image showing the working machine 100 moving the work area along the calculated trajectory. Note that the circle C2 (which is not actually displayed) on the first predicted ground shape image PG1 shown in Figure 7 represents the position selected by the operator as the starting point of the trajectory for the second excavation and soil removal operation.

[0083] Subsequently, the client-side controller 60 determines whether the conditions for starting an operation (task) have been met (step ST7). The conditions for starting an operation (task) include, for example, that the autonomous operation button image BT15 has been touched, or that the MG button image BT16 has been touched. In other words, the client-side controller 60 determines that the conditions for starting an operation (task) have been met when the autonomous operation button image BT15 or the MG button image BT16 has been touched.

[0084] If it is determined that the conditions for starting the operation (task) are not met (NO in step ST7), the client-side controller 60 returns to step ST3 and determines whether or not the content of the operation (task) has been entered (step ST3).

[0085] If, at this stage, it is determined that the details of the operation (task) have not been entered (NO in step ST3), the client-side controller 60 executes step ST7 without executing steps ST4 to ST6. In other words, the client-side controller 60 waits until the details of the operation (task) are entered or the start conditions for the operation (task) are met.

[0086] Furthermore, if the system determines at this stage that the details of the operation (work) have been entered (YES in step ST3), the client-side controller 60 calculates, for example, the trajectory of the second excavation and soil removal operation (step ST4), and estimates the ground shape after the completion of the second excavation and soil removal operation as the predicted ground shape (second predicted ground shape) (step ST5). The client-side controller 60 then displays the second predicted ground shape image PG2, which is a contour map image representing the estimated predicted ground shape (second predicted ground shape), and the second operation image MV2, which is an image showing the working machine 100 moving the work area along the calculated trajectory. The same procedure is followed for the third and subsequent excavation and soil removal operations. Note that the circle C3 (which is not actually displayed) on the second predicted ground shape image PG2 shown in Figure 7 represents the position selected by the operator as the starting point of the trajectory for the third excavation and soil removal operation. Furthermore, if at this stage it is determined that the content of the first operation (task) that has already been entered has been re-entered (changed), the client-side controller 60 may recalculate the trajectory of the first excavation and soil removal operation and re-estimate the ground shape after the completion of the first excavation and soil removal operation as the predicted ground shape (second predicted ground shape).

[0087] Then, if it is determined that the conditions for starting the operation (work) have been met (YES in step ST7), the client-side controller 60 terminates the current work support process. If the autonomous operation button image BT15 or MG button image BT16 is touched and it is determined that the conditions for starting the operation (work) have been met, the client-side controller 60 transmits information about one or more calculated trajectories to the work machine 100. In the illustrated example, the client-side controller 60 transmits information about one or more calculated trajectories to the work machine 100 through the management device 200.

[0088] Upon receiving information regarding one or more tracks, the shovel-side controller 30 of the work machine 100 uses that track information to initiate control related to the machine guidance function or the machine control function. Specifically, the shovel-side controller 30 initiates control related to the machine control function when the autonomous operation button image BT15 is touched, and initiates control related to the machine guidance function when the MG button image BT16 is touched.

[0089] Next, with reference to Figures 8 to 13, another example of the operation (work) support screen will be described. Figures 8 to 13 are diagrams showing examples of screens displayed on the display device DS. Specifically, Figure 8 shows an excavation and loading operation support screen, which is another example of the operation (work) support screen displayed on the third display device DS3. Figure 9 shows a slope shaping operation support screen, which is yet another example of the operation (work) support screen displayed on the third display device DS3. Figure 10 shows a shoveling and transshipment operation support screen, which is yet another example of the operation (work) support screen displayed on the third display device DS3. Figure 11 shows a trenching operation support screen, which is yet another example of the operation (work) support screen displayed on the third display device DS3. Figure 12 shows an excavation and loading operation support screen, which is yet another example of the operation (work) support screen displayed on the third display device DS3. Figure 13 shows a slope shaping operation support screen, which is yet another example of the operation (work) support screen displayed on the third display device DS3.

[0090] Specifically, the excavation and loading operation support screen shown in Figure 8 is a screen displayed to allow the operator to input details regarding support for the excavation and loading operation. The excavation and loading operation is typically a combination of excavation, boom raising and slewing, soil removal, and boom lowering and slewing. In the illustrated example, the trajectory of each excavation and loading operation is set so that the work machine 100 excavates at a point selected by the operator and removes the excavated soil to the ground on the left side (from the perspective of the work machine 100). Specifically, the part represented by circle C4 (not actually displayed) in the actual ground shape image AG represents the position selected by the operator as the starting point of the trajectory for the first excavation and loading operation. Similarly, the part represented by circle C5 (not actually displayed) in the first predicted ground shape image PG1 represents the position selected by the operator as the starting point of the trajectory for the second excavation and loading operation. In addition, the part represented by circle C6 (not actually displayed) in the second predicted ground shape image PG2 represents the position selected by the operator as the starting point of the trajectory for the third excavation and loading operation.

[0091] The slope shaping operation support screen shown in Figure 9 is a screen displayed to allow the operator to input details regarding support for slope shaping operations. Slope shaping operations typically involve using a slope bucket, which is an example of a bucket 6, to flatten the surface of the slope. In the illustrated example, the trajectory of each slope shaping operation is set so that the work machine 100 excavates the lower part of the uphill slope to a depth that reaches the toe of the slope, and discharges the excavated soil onto the ground to the left rear (from the perspective of the work machine 100). Note that the toe of the slope is currently underground and not exposed. Specifically, the part represented by circle C7 (not actually displayed) in the actual ground shape image AG represents the position selected by the operator as the starting point of the trajectory for the first slope shaping operation. Similarly, the part represented by circle C8 (not actually displayed) in the first predicted ground shape image PG1 represents the position selected by the operator as the starting point of the trajectory for the second slope shaping operation. Furthermore, the circle C9 (which is not actually displayed) in the second predicted ground shape image PG2 represents the position selected by the operator as the starting point of the trajectory for the third slope shaping operation.

[0092] The excavation and transfer operation support screen shown in Figure 10 is a screen displayed to allow the operator to input details of the support for the excavation and transfer operation. The excavation and transfer operation is the operation to remove unwanted soil and sand around the work machine 100 and is achieved by one or more excavation and transfer operations. The excavation and transfer operation is typically an operation to level the ground in a predetermined range around the work machine 100 and is a combination of horizontal pulling operation (excavation operation), rotation operation, and soil removal operation. In the illustrated example, the trajectory of the first excavation and transfer operation is set so that the work machine 100 raises the ground in the range R1 (excavation target range) selected by the operator to a predetermined depth, levels its bottom surface, and moves the soil and sand excavated from the excavation target range to another range R2 (temporary storage range) selected by the operator. Furthermore, the track for the second excavation and transfer operation is set so that the work machine 100 moves the soil deposited in the temporary deposit area to another area R3 (final deposit area) selected by the operator. Note that the three circular arrows in Figure 10 (which are not actually displayed) indicate that the operator has selected areas R1, R2, and R3 respectively by tracing a circular motion with their finger on a part of the image (touch panel).

[0093] The trenching operation support screen shown in Figure 11 is a screen displayed to allow the operator to input information regarding support for the trenching operation. Trenching is the operation of digging a trench in the ground and is achieved by one or more trenching operations. A trenching operation is typically an operation to form a trench of a predetermined width and depth, and is a combination of excavation, swiveling, and soil removal operations. In the illustrated example, the trajectory of the first trenching operation is set so that the work machine 100 digs the ground in the area selected by the operator (the trenching target area) to a predetermined depth, levels the bottom surface, and removes the excavated soil from the trenching target area onto the ground to the right (as seen from the work machine 100). The two straight arrows AR1 in Figure 11 (which are not actually displayed) indicate that the operator has selected the trenching target area by tracing a part of the image (touch panel) twice in a straight line with their finger. The two straight arrows AR1 also correspond to the edges on both sides of the trench.

[0094] The excavation and loading operation support screen shown in Figure 12 is a screen displayed to allow the operator to input details regarding support for the excavation and loading operation. The excavation and loading operation is achieved by one or more excavation and loading operations. In the illustrated example, the trajectory for the first excavation and loading operation is set so that the work machine 100 excavates soil from the embankment at a point selected by the operator and temporarily discharges the excavated soil into a trench to the left of the embankment (from the perspective of the work machine 100). The trajectory for the second excavation and loading operation is set so that the work machine 100 re-excavates the soil that was temporarily discharged into the trench and discharges the re-excavated soil onto the ground to the left of the trench (from the perspective of the work machine 100). Note that the straight arrow AR2 in Figure 12 (which is not actually displayed) indicates that the operator is selecting the excavation location (location of the embankment) and the soil discharge location (location of the trench) by tracing a straight line with their finger on a part of the actual ground shape image AG. In other words, the starting point of the straight arrow AR2 corresponds to the excavation location (the location of the embankment), and the ending point of the straight arrow AR2 represents the soil removal location (the location of the ditch). Also, the straight arrow AR3 in Figure 12 (which is not actually displayed) represents the operator selecting the excavation location (the location of the ditch) and the soil removal location by tracing a straight line with their finger over a portion of the first predicted ground shape image PG1. In other words, the starting point of the straight arrow AR3 corresponds to the excavation location (the location of the ditch), and the ending point of the straight arrow AR3 represents the soil removal location.

[0095] The slope shaping work support screen shown in Figure 13 is a screen displayed to allow the operator to input details of the support for slope shaping work. Slope shaping work is achieved by one or more slope shaping operations. In the illustrated example, the trajectory of the first slope shaping operation is set so that the work machine 100 excavates the lower part of the uphill slope to a depth that reaches the toe of the slope, and then discharges the excavated soil onto the ground to the left rear (from the perspective of the work machine 100). The trajectory of the second slope shaping operation is set so that the work machine 100 moves in the direction selected by the operator (the direction along the slope), then excavates the lower part of the uphill slope to a depth that reaches the toe of the slope, and then discharges the excavated soil onto the ground to the left rear (from the perspective of the work machine 100). Note that the straight arrow AR4 in Figure 13 (which is not actually displayed) indicates that the operator is selecting the direction of the slope extension by tracing a part of the actual ground shape image AG with their finger in a straight line. Similarly, the straight arrow AR5 in Figure 13 (which is not actually displayed) indicates that the operator is selecting the direction of the slope extension by tracing a portion of the first predicted ground shape image PG1 in a straight line with their finger.

[0096] In this way, the work machine 100 operates automatically or semi-automatically according to the setup set by the operator on various operation (work) support screens. "Setup" refers to the sequence or procedure of operations or work performed by the work machine 100. In the illustrated example, the setup setting includes setting the trajectory that the work part will follow. However, the setup setting may not include the trajectory setting, but may include the setting of the work area. In this case, the client-side controller 60 may calculate (generate) the trajectory based on the set work area. Furthermore, the work machine 100 may be configured to operate completely autonomously without intervention from the operator of the work machine 100.

[0097] Furthermore, the operator of the support device 300 can set the steps for the next operation (task) while checking the predicted ground shape after at least one incomplete operation (task) has been completed. Therefore, the operator can divide the entire operation performed by the work machine 100 into multiple stages and set the steps for the entire operation while recognizing the ground shape when each stage is completed in advance. In other words, before starting the operation, the operator can set all the steps until the entire operation is completed while recognizing the ground shape when each stage is completed in advance. Consequently, the work support system SYS can improve the efficiency of setting up steps compared to when the operator sets the steps for the next operation (task) each time a stage is completed, and this results in a reduction in the time required to complete the entire operation.

[0098] Next, with reference to Figures 14 and 15, the specific effects of executing the work support process will be explained. Figure 14 is a diagram showing the relationship between the number of setups (minor setups) and the time required when setting up a setup (minor setup) for a specific operation by the work machine 100 and then actually operating the work machine 100. Specifically, the upper part of Figure 14 shows the time elapsed when setting up a setup (minor setup) and the actual operation by the work machine 100 are repeated sequentially and alternately three times, and the lower part of Figure 14 shows the time elapsed when the second and third setups (minor setups) and the first actual operation by the work machine 100 are performed simultaneously. Figure 15 is a diagram showing the relationship between the number of setups (major setups) and the time required when setting up a setup (major setup) for a specific operation by the work machine 100 and then actually operating the work machine 100. Specifically, the upper diagram of Figure 15 shows the time progression when the setup (major setup) and the actual work performed by the machine 100 are repeated sequentially and alternately twice, while the lower diagram of Figure 15 shows the time progression when the second setup (major setup) and the first actual work performed by the machine 100 are performed simultaneously. In the illustrated examples, the setup refers to the setting of the track. Furthermore, minor setup refers to the setup related to "actions," and major setup refers to the setup related to "work" that includes multiple "actions."

[0099] More specifically, in the example shown in Figure 14, in the case of "no work support function" where work support processing is not performed, the operator of the support device 300 first sets up the first setup (minor setup), and when the setting up the first setup (minor setup) is completed, starts the operation of the work machine 100 for the first time (operation of the work part along the set trajectory). Then, when the operation of the work machine 100 for the first time is completed, that is, when the execution time of the first operation has elapsed, the operator of the support device 300 sets up the second setup (minor setup), and when the setting up the second setup (minor setup) is completed, starts the operation of the work machine 100 for the second time. Similarly thereafter, when the operation of the work machine 100 for the second time is completed, that is, when the execution time of the second operation has elapsed, the operator of the support device 300 sets up the third setup (minor setup), and when the setting up the third setup (minor setup) is completed, starts the operation of the work machine 100 for the third time. In this way, the operator of the support device 300 sets up the setup (minor setup) for each operation of the work machine 100 based on the actual ground shape at that time, immediately before starting each operation, and then starts each operation. Therefore, the time required to complete the entire work is the sum of the time required to set up the minor setup (minor setup) for each operation and the execution time for each operation.

[0100] On the other hand, if the "work support function is enabled" which performs work support processing, the operator of the support device 300 first sets up the first setup (minor setup). Then, as soon as the operator completes the setup of the first setup (minor setup), they can start the operation of the first work machine 100 (operation of the work part along the set trajectory) and simultaneously start setting up the second setup (minor setup). Furthermore, even if the operation of the first work machine 100 is still ongoing, the operator can start setting up the third setup (minor setup) as soon as the setup of the second setup (minor setup) is completed. In the illustrated example, the operation of the second work machine 100 starts immediately as soon as the operation of the first work machine 100 is completed, because the setup of the second setup (minor setup) has already been completed before the operation of the first work machine 100 is completed. Similarly, the operation of the third work machine 100 starts immediately as soon as the operation of the second work machine 100 is completed, because the setup of the third setup (minor setup) has already been completed before the operation of the second work machine 100 is completed. In this way, the operator of the support device 300 can start setting up a second setup (minor setup) immediately after completing the first setup (minor setup), regardless of whether the work machine 100 is in operation or not, and can then start setting up a third setup (minor setup) immediately after completing the second setup (minor setup). Therefore, the time required to complete the entire operation is reduced by up to two minor setup times compared to the case without the work support function.

[0101] The example shown in Figure 15 is the same as the example shown in Figure 14, except that "action" is replaced with "task" and the number of times the "task" is performed is two. Note that a "task" consists of one or more "actions".

[0102] Specifically, in the example shown in Figure 15, in the case of "no work support function" where work support processing is not performed, the operator of the support device 300 first sets up the first setup (major setup), and starts the first operation of the work machine 100 (movement of the work part along the set trajectory) when the first setup (major setup) is completed. Then, when the first operation of the work machine 100 is completed, that is, when the execution time of the first operation has elapsed, the operator of the support device 300 sets up the second setup (major setup), and starts the second operation of the work machine 100 when the setup (major setup) is completed. In this way, the operator of the support device 300 sets up the setup (major setup) for each operation based on the actual ground shape at that time, just before the start of each operation of the work machine 100, and starts each operation. Therefore, the time required to complete the entire operation is the sum of the time required for setting up each setup (major setup) and the execution time of each operation.

[0103] On the other hand, when the "work support function is enabled" to perform work support processing, the operator of the support device 300 first sets up the first setup (major setup). Then, as soon as the operator completes the first setup (major setup), they can start the first operation of the work machine 100 (movement of the work part along the set trajectory) and simultaneously start setting up the second setup (major setup). In the illustrated example, the second operation of the work machine 100 starts immediately after the first operation of the work machine 100 is completed, because the second setup (major setup) has already been completed before the first operation of the work machine 100 is completed. In this way, the operator of the support device 300 can start setting up the second setup (major setup) immediately after the first setup (major setup) is completed, regardless of whether the work machine 100 is in operation or not. Therefore, the time required to complete the entire operation is shortened by up to the time of one major setup compared to the case without the "work support function".

[0104] Next, with reference to Figure 16, the process by which the control device corrects the trajectory (hereinafter referred to as the "trajectory correction process") will be explained. Figure 16 is a flowchart showing an example of the flow of the trajectory correction process. In the illustrated example, the client-side controller 60 executes this trajectory correction process each time an operation (work) by the work machine 100 is completed. The client-side controller 60 may also execute this trajectory correction process even in the middle of an operation (work) by the work machine 100, if it can recognize the actual ground shape and the predicted ground shape at that time. Furthermore, this trajectory correction process may be executed by the shovel-side controller 30 or the server-side controller 50, or it may be executed in a distributed manner by at least two of the shovel-side controller 30, the server-side controller 50, and the client-side controller 60.

[0105] First, the client-side controller 60 determines whether the actual ground shape and the predicted ground shape are different (step ST11). In the illustrated example, the client-side controller 60 determines whether the actual ground shape and the predicted ground shape are different based on the actual ground shape image generated based on the output of the spatial recognition device S6 at the present time when the first excavation and soil removal operation is completed, and the predicted ground shape image predicted at a past time when the first excavation and soil removal operation is completed. Specifically, the client-side controller 60 generates a difference image (contour map image) between the actual ground shape image and the predicted ground shape image, for example. More specifically, the client-side controller 60 translates, rotates, enlarges, or shrinks the actual ground shape image and the predicted ground shape image relatively so that the sum of the absolute values ​​(height difference) of each pixel constituting the difference image is smallest, and then superimposes them. That is, the client-side controller 60 associates each pixel constituting the actual ground shape image with each pixel constituting the predicted ground shape image.

[0106] Based on this, the client-side controller 60 determines that the actual ground shape and the predicted ground shape are different if the sum of the absolute values ​​(height differences) of each pixel constituting the difference image exceeds a predetermined threshold. The difference image represents the unevenness of the actual ground shape image relative to the predicted ground shape, and each pixel constituting the difference image has a pixel value that represents the magnitude of the unevenness (height difference). Figure 17 shows an example of an actual ground shape image, a predicted ground shape image, and a difference image. In Figure 17, for clarity, contour lines are omitted, and finer dot patterns are applied to locations where the height difference is greater than zero (locations with larger convexity), while finer cross patterns are applied to locations where the height difference is less than zero (locations with larger concavity).

[0107] Alternatively, the client-side controller 60 may determine that the actual ground shape and the predicted ground shape are different if the number of pixels (area) in the difference image whose absolute value exceeds a predetermined value exceeds a predetermined number. Alternatively, the client-side controller 60 may determine whether the actual ground shape and the predicted ground shape are different based on any determination criterion using the maximum, median, or average value of the pixel values.

[0108] If the client-side controller 60 determines that the actual ground shape and the predicted ground shape are different (YES in step ST11), it corrects the trajectory of the next (subsequent) operation (task) (step ST12). In the illustrated example, for example, if the client-side controller 60 determines that the current actual ground shape is different from the predicted ground shape estimated before the first excavation and soil removal operation is completed, it corrects or resets the trajectory for the second excavation and soil removal operation. The correction (resetting) of the trajectory for the second excavation and soil removal operation is performed, for example, so that the actual ground shape (which does not currently exist) at the time the second excavation and soil removal operation is completed matches the predicted ground shape (which has already been estimated) as closely as possible. However, the client-side controller 60 may also correct (reset) the trajectories for one or more excavation and soil removal operations from the second onward so that the actual ground shape (which does not currently exist) at the time the third and subsequent excavation and soil removal operations are completed match the predicted ground shape (which has already been estimated) as closely as possible. In this case, the actual ground shape (which does not exist at that time) at the end of the second excavation and soil removal operation may differ from the predicted ground shape (which has already been estimated). Alternatively, the client-side controller 60 may modify (reset) the trajectory for one or more excavation and soil removal operations from the third operation onward without modifying the trajectory for the second excavation and soil removal operation.

[0109] Subsequently, the client-side controller 60 transmits information about the corrected (reconfigured) trajectory to the work machine 100, and can start the machine control function of the work machine 100 to realize the movement of the work part along that trajectory.

[0110] As described above, the work support system SYS according to the embodiment of this disclosure is a system for a work machine 100, which has a lower traveling body 1, an upper rotating body 3 that is rotatably mounted on the lower traveling body 1, and an attachment AT having a work area WP that is attached to the upper rotating body 3, as shown in Figure 2. Specifically, as shown in Figure 4, the work support system SYS has a spatial recognition device S6 that recognizes the ground shape around the work machine 100, a control device (client-side controller 60), and a display device DS (third display device DS3). The control device is configured to cause the work machine 100 to perform predetermined operations by moving the work machine 100 automatically or semi-automatically according to a preset procedure. The control device is also configured to predict the ground shape after the predetermined operation is completed, based on the current ground shape recognized by the spatial recognition device S6 and the content of the predetermined operation to be performed by the work machine 100. The default operations include, for example, excavation and soil removal, excavation and loading, slope shaping, plowing and reloading, or trenching. Furthermore, as shown in Figure 7, the display device DS is configured to display a predicted ground shape image PG, which is an image representing the ground shape predicted by the control device.

[0111] This work support system SYS predicts the ground shape after a predetermined operation is completed and displays it on the display device DS before the predetermined operation is finished. This allows the operator to visually confirm the predicted ground shape image PG before the predetermined operation is completed. Therefore, this work support system SYS allows the operator to easily set up the next predetermined operation while viewing the predicted ground shape image PG. Consequently, this work support system SYS has the effect of improving the efficiency of setup by the operator.

[0112] Furthermore, the control device (client-side controller 60) may be configured to set up the setup for the next default operation while the default operation is being executed. Specifically, the control device may be configured to start setting up the setup for the next default operation before the default operation is completed, and to complete setting up the setup for the next default operation before the default operation is completed. The control device may also be configured to start setting up the setup for the default operation after the next one before the default operation is completed, and to complete setting up the setup for the default operation after the next one before the default operation is completed. The same applies to default operations even further in the future. For example, as shown in the lower diagram of Figure 14, the control device may be configured to set up the setup (minor setup) for the second default operation and the setup (minor setup) for the third default operation while the first default operation is being executed. In this case, the control device may start executing the second default operation while setting up the setup (minor setup) for the third default operation, or it may start setting up the setup (minor setup) for the third default operation after starting the execution of the second default operation. Alternatively, the control device may be configured to set up the setup (main setup) for the second default operation while the first default operation is being executed, as shown in the lower diagram of Figure 15. A default operation is an operation that is achieved by one or more default operations.

[0113] This configuration allows for more efficient setup compared to a system where the operator sets up the next default operation after each default operation is completed, and consequently, it reduces the time required to complete the entire operation.

[0114] Furthermore, the control device (client-side controller 60) may be configured to automatically or semi-automatically move the work machine 100 so that the work area WP moves along a pre-set trajectory, thereby causing the work machine 100 to perform a predetermined operation. In this case, the control device may be configured to recognize the difference between the actual ground shape and the predicted ground shape, and to correct the trajectory used in the predetermined operation to be performed thereafter based on the recognized difference. The actual ground shape is the ground shape after the work is performed as recognized by the spatial recognition device S6 after the predetermined operation is completed. The predicted ground shape is the ground shape after the predetermined operation is completed, as predicted by the control device before the predetermined operation is completed. In the illustrated example, as shown in Figure 17, the client-side controller 60 recognizes the difference between the actual ground shape image and the predicted ground shape image as a difference image. The client-side controller 60 is configured to correct or reset the trajectory used in the predetermined operation to be performed thereafter based on the information derived from the difference image (for example, the sum of each pixel value).

[0115] This configuration allows for adjustments to the trajectory of subsequent pre-defined operations if, for example, the actual ground shape differs from the predicted ground shape after the first pre-defined operation is completed. This has the effect of preventing the difference between the actual ground shape and the predicted ground shape from increasing with each completed pre-defined operation. In other words, this configuration has the effect of preventing the final realized actual ground shape from differing significantly from the ground shape of the target construction surface. Therefore, this configuration has the additional effect of preventing the difference between the final realized actual ground shape and the ground shape of the target construction surface from becoming too large compared to the case where the steps are set sequentially, as shown in the upper diagram of Figure 14 or the upper diagram of Figure 15.

[0116] Furthermore, the control device (client-side controller 60) may be configured to predict the first ground shape (ground shape represented by the first predicted ground shape image PG1) after the completion of the first predetermined operation (first excavation and soil removal operation), which is one of a series of predetermined operations (excavation and soil removal operations), before the first predetermined operation is completed, as shown in Figure 7. The control device may also be configured to predict the second ground shape (ground shape represented by the second predicted ground shape image PG2) after the completion of the second predetermined operation (second excavation and soil removal operation), which is another of a series of predetermined operations. In this case, the display device DS (third display device DS3) may be configured to display the first predicted ground shape image PG1, which is an image representing the first ground shape predicted by the control device, and the second predicted ground shape image PG2, which is an image representing the second ground shape predicted by the control device.

[0117] This configuration allows operators to set up procedures for actions that will occur two or more times in advance. Therefore, this configuration enables more efficient setup, and consequently, reduces the time required to complete the entire series of tasks, including the time required for setup.

[0118] Furthermore, the control device (client-side controller 60) may be configured to derive a numerical value representing the magnitude of the difference between the actual ground shape and the predicted ground shape, and to modify or reset the trajectory used in the predetermined operation to be executed thereafter if that numerical value is greater than or equal to a predetermined value. In the illustrated example, as shown in Figure 17, the client-side controller 60 derives the sum of the absolute values ​​of the values ​​(height difference) of each pixel constituting the difference image as a numerical value representing the magnitude of the difference between the actual ground shape and the predicted ground shape. The client-side controller 60 then modifies or resets the trajectory used in the predetermined operation to be executed thereafter if that sum is greater than or equal to a predetermined value. The control device may also derive a numerical value representing the magnitude of the difference between the actual ground shape and the predicted ground shape and update the predictor or trained model based on that numerical value. The predictor is a device (part of the control device) that predicts the ground shape after the predetermined operation is completed, based on the current ground shape recognized by the spatial recognition device S6 and the content of the predetermined operation to be executed by the work machine 100. This allows the control device to improve the accuracy of estimating the predicted ground shape.

[0119] This configuration has the effect of clearly (appropriately) determining whether the actual ground shape and the predicted ground shape are different. Therefore, this configuration has the effect of preventing inappropriate trajectory corrections from being performed based on ambiguous (inappropriate) determination results.

[0120] Furthermore, the display device DS (third display device DS3) may be configured to display an operation image MV, which is an image representing the progression of the default operation, before the default operation is completed, as shown in Figure 7.

[0121] This configuration has the effect of clearly showing the operator how the default operations will be performed. Therefore, the operator can confirm in advance that the work machine 100 will perform the intended operation. This configuration also has the effect of clearly showing the operator the operating range of the work machine 100 (the range that the attachment AT can reach). Therefore, the operator can ensure the safety of the area around the work machine 100 in advance, such as by prompting people who are within or near the operating range to move away from the operating range. Alternatively, the operator can draw the attention of people around the work machine 100 and urge them not to enter or approach the operating range.

[0122] Furthermore, the display device DS (third display device DS3) may be configured to display the predicted ground shape image PG and the motion image MV in association, as shown in Figure 7.

[0123] This configuration has the effect of clearly showing the operator how the default actions will be performed and what kind of ground shape will be achieved as a result of those actions. Therefore, the operator can confirm in advance that the work machine 100 will perform the intended actions and what kind of ground shape will be achieved as a result of those actions.

[0124] Furthermore, the display device DS (third display device DS3) according to the embodiment of this disclosure is configured to cause the work machine 100 to perform a predetermined operation by automatically or semi-automatically moving the work machine 100 according to a preset procedure. The display device DS is also configured to predict the ground shape after the predetermined operation is completed, based on the current ground shape recognized by the spatial recognition device S6 and the content of the predetermined operation to be performed by the work machine 100. In addition, as shown in Figure 7, the display device DS is configured to display a predicted ground shape image PG, which is an image representing the predicted ground shape.

[0125] This display device DS predicts and displays the ground shape after the predetermined operation is completed, allowing the operator to view the predicted ground shape image PG before the predetermined operation is finished. Therefore, this display device DS allows the operator to easily set up the next predetermined operation while viewing the predicted ground shape image PG. Consequently, this display device DS has the effect of improving the efficiency of setup by the operator.

[0126] Alternatively, the display device DS (third display device DS3) may cause the work machine 100 to perform a predetermined operation by moving the work machine 100 so that the work part WP moves along a pre-set trajectory. In this case, the display device DS may recognize the difference between the ground shape after the work is completed, as recognized by the spatial recognition device S6 after the predetermined operation is completed, and the ground shape after the predetermined operation is completed, as predicted before the predetermined operation is completed. Based on the recognized difference, the display device DS may then modify or reset the trajectory used in the predetermined operation to be performed thereafter.

[0127] This display device DS can, for example, correct or reset the trajectory for subsequent default operations if the actual ground shape and the predicted ground shape differ after the first default operation is completed. Therefore, this display device DS has the effect of suppressing the increasing difference between the actual ground shape and the predicted ground shape with each completion of a default operation. In other words, this display device DS has the effect of suppressing a large difference between the final realized actual ground shape and the ground shape of the target construction surface. Therefore, this display device DS has the additional effect of suppressing a large difference between the final realized actual ground shape and the ground shape of the target construction surface compared to the case where the setup is set sequentially, as shown in the upper diagram of Figure 14 or the upper diagram of Figure 15.

[0128] Preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the embodiments described above, nor is it limited to the embodiments described later. Various modifications or substitutions can be applied to the embodiments described above or later without departing from the scope of the present invention. Furthermore, features described separately can be combined as long as no technical inconsistencies arise.

[0129] For example, in the above embodiment, the spatial recognition device S6 is attached to the upper rotating body 3 of the work machine 100. However, the spatial recognition device S6 may also be attached to an aircraft such as a multicopter flying above the work machine 100, or to a steel tower or the like at the work site where the work machine 100 is working, or a combination of these.

[0130] Furthermore, in the above-described embodiment, the support content selection screen and the operation (work) support screen are displayed on the third display device DS3 provided in the support device 300. However, the support content selection screen and the operation (work) support screen may also be displayed on the first display device DS1 provided in the cabin 10 of the work machine 100. In this case, the operator sets the setup via the first input device ID1. Alternatively, the support content selection screen and the operation (work) support screen may also be displayed on the second display device DS2 provided in the management device 200. In this case, the operator sets the setup via the second input device ID2.

[0131] Furthermore, in the above-described embodiment, the work support processing is performed by the client-side controller 60 of the support device 300. However, the work support processing may also be performed by the shovel-side controller 30 of the work machine 100, or by the server-side controller 50 of the management device 200. Alternatively, the work support processing may be performed in a distributed manner by at least two of the shovel-side controller 30, the server-side controller 50, and the client-side controller 60. [Explanation of Symbols]

[0132] 1. Lower travel body 1C. Crawler 1CL. Left crawler 1L. Left travel hydraulic motor 1R. Right travel hydraulic motor 2. Swivel mechanism 2A. Swivel hydraulic motor 3. Upper slewing body 4. Boom 5. Arm 6. Bucket 7. Boom cylinder 8. Arm cylinder 9. Bucket cylinder 10. Cabin 11. Engine 13. Regulator 14. Main pump 15. Pilot pump 17. Control valve unit 26. Operating device 28. Discharge pressure sensor 29. Operation sensor 30. Excavator side controller 31. Solenoid valve 32. Pilot pressure sensor 50. Server side controller 60. Client side controller 100. Working machine 171-176... Control valve 200... Management device 300... Support device A1... Microphone array AS... Attitude sensor AT... Attachment DA... Travel actuator DS... Display device DS1... First display device DS2... Second display device DS3... Third display device ID... Input device ID1... First input device ID2... Second input device ID3... Third input device IN... Information and communication network PD... Positioning device PV... Swivel axis S1... Boom angle sensor S2... Arm angle sensor S3... Bucket angle sensor S4... Machine tilt sensor S5... Swivel angular velocity sensor S6... Spatial recognition device S6B... Rear camera S6F... Front camera S6L... Left camera S6R... Right camera S7... Orientation detection sensor SA... Swivel actuator SYS... Work support system TD... Communication device TD1... First communication device TD2... Second communication device TD3... Third communication device WA... Work actuator

Claims

1. A system for a work machine comprising a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and an attachment having a working part attached to the upper rotating body, A spatial recognition device that recognizes the ground shape around the aforementioned work machine, Control device and It has a display device, The control device is The system is configured to cause the machine to perform a predetermined operation by moving it according to a pre-set procedure, and, The spatial recognition device is configured to predict the ground shape after the predetermined operation is completed, based on the current ground shape recognized by the spatial recognition device and the content of the predetermined operation performed by the work machine. The display device is configured to display a predicted ground shape image, which is an image representing the ground shape predicted by the control device. A system for industrial machinery.

2. The control device sets the procedure for the next default operation while the default operation is being executed. The system for a work machine according to claim 1.

3. The control device is The system is configured to cause the work machine to perform the predetermined operation by moving the work machine so that the work part moves along a predetermined trajectory, The system is configured to recognize the difference between the ground shape after the operation is completed as recognized by the spatial recognition device after the operation is completed and the ground shape after the operation is completed as predicted by the control device before the operation is completed, and to correct the trajectory used in the subsequent operation based on the recognized difference. The system for a work machine according to claim 1.

4. The control device is configured to predict the first ground shape after the completion of the first predetermined operation, which is one of a series of predetermined operations, before the first predetermined operation is completed, and to predict the second ground shape after the completion of the second predetermined operation, which is another one of a series of predetermined operations. The display device is configured to display a first predicted ground shape image, which is an image representing the first ground shape predicted by the control device, and a second predicted ground shape image, which is an image representing the second ground shape predicted by the control device. The system for a work machine according to claim 1.

5. The control device is configured to derive a numerical value representing the magnitude of the difference, and to correct the trajectory used in a predetermined operation to be performed thereafter if the numerical value is greater than or equal to a predetermined value. The system for a work machine according to claim 3.

6. The display device is configured to display an operation image, which is an image representing the progression of the predetermined operation, before the predetermined operation is completed. The system for a work machine according to claim 1.

7. The display device displays the predicted ground shape image and the motion image in association. The system for a work machine according to claim 6.

8. A display device comprising a system for a work machine, the system comprising a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and an attachment having a working part attached to the upper rotating body, The system is configured to cause the work machine to perform a predetermined operation by moving it according to a pre-set procedure. The spatial recognition device, which recognizes the ground shape around the aforementioned work machine, is configured to predict the ground shape after the predetermined operation is completed, based on the current ground shape recognized by the spatial recognition device and the content of the predetermined operation performed by the aforementioned work machine, and It is configured to display a predicted ground shape image, which is an image representing the predicted ground shape. Display device.

9. The system is configured to cause the work machine to perform the predetermined operation by moving the work machine so that the work part moves along a predetermined trajectory, The spatial recognition device is configured to recognize the difference between the ground shape after the operation is completed as recognized by the spatial recognition device after the operation is completed and the ground shape after the operation is completed as predicted before the operation is completed, and to correct the trajectory used in the subsequent operation based on the recognized difference. The display device according to claim 8.

Images