Work machine, information processing device, and program

Abstract

Provided is a technique with which it is possible to ascertain the shape of a work object of a work machine. A shovel 100 according to an embodiment of the present disclosure comprises a controller 30, and sensors S1-S6 that acquire data relating to the trajectory of a bucket 6 of the shovel 100. The controller 30 acquires data relating to the trajectory of the bucket 6 in accordance with the execution of an excavation operation or a soil removal operation by the shovel 100, and estimates the shape of a work target after the execution of the excavation operation or the soil removal operation on the basis of the acquired data.

Description

Work machine, information processing device, and program

[0001] The present disclosure relates to work machines and the like.

[0002] BACKGROUND ART Conventionally, there are known work machines such as excavators that are operated by an operator or that are autonomously operated (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-188432

[0004] Incidentally, for example, an operator riding an shovel may have a blind spot for part of the ground to be worked on due to unevenness in the soil and sand, shovel attachments, etc. Furthermore, when an shovel is remotely operated, the operator must operate the shovel while looking at images from a camera mounted on the shovel, and depending on the state of the camera image, it may be difficult to grasp the shape of the ground to be worked on. Therefore, the shape of the work target may be recognized based on data from a camera, a ranging sensor (distance sensor), etc., and the shape of the work target may be displayed on a display mounted in the cabin of the work machine or on a display for remote operation.

[0005] Furthermore, for example, when an excavator is operated autonomously, the trajectory of the working part (bucket) of the excavator may be determined depending on the shape of the soil and sand on the ground as the work target. In this case, the shape of the work target is recognized based on data from a camera, a distance measurement sensor, etc., and the trajectory of the working part of the work machine is determined based on the shape of the work target.

[0006] However, for example, when using a camera or ranging sensor mounted on an excavator, there is a possibility of occlusion, in which the shape of part of the work object cannot be measured due to obstructions from the shovel's attachments or unevenness in the soil. This may result in, for example, a hole appearing in part of the shape of the work object displayed on the display, creating an undesirable situation from the perspective of supporting the operator. Furthermore, for example, the inability to measure part of the shape of the work object may result in a decrease in accuracy in generating the trajectory of the work part for autonomous driving. Furthermore, for example, if a camera or ranging sensor is not mounted on an excavator, it is not possible to employ a function for displaying the shape of the work object on the display or an autonomous driving function for the excavator.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide a technology that can grasp the shape of a work target of a work machine.

[0008] In order to achieve the above object, one embodiment of the present disclosure provides a work machine including a processing device that acquires data representing the state of the work machine, which data relates to changes in the shape of a work object, in accordance with the operation of the work machine, and estimates the shape of the work object after the operation of the work machine based on the acquired data.

[0009] Furthermore, in another embodiment of the present disclosure, there is provided an information processing device that acquires data representing the state of a work machine in accordance with the operation of the work machine, the data relating to changes in the shape of a work object of the work machine, and estimates the shape of the work object after the operation of the work machine based on the acquired data.

[0010] Furthermore, in yet another embodiment of the present disclosure, a program is provided that causes an information processing device to execute the steps of: acquiring data representing the state of the work machine in accordance with the operation of the work machine, the data relating to changes in the shape of a work object of the work machine; and estimating the shape of the work object after operation of the work machine based on the acquired data.

[0011] In yet another embodiment of the present disclosure, a program is provided that causes an assistance device to execute the following steps: acquire data representing the state of the work machine in accordance with the operation of the work machine, the data relating to changes in the shape of a work object of the work machine; estimate the shape of the work object after the work machine operates based on the acquired data; and display the shape of the work object on a display unit based on the result of the estimation of the shape of the work object.

[0012] According to the above-described embodiment, it is possible to grasp the shape of the work target of the work machine.

[0013] 1 is a diagram illustrating an example of an operation support system. FIG. 1 is a top view illustrating an example of a shovel. FIG. 2 is a diagram illustrating an example of a configuration related to remote operation of a shovel. FIG. 3 is a diagram illustrating an example of a hardware configuration of a shovel. FIG. 4 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 5 is a functional block diagram illustrating a first example of a functional configuration of the operation support system. FIG. 6 is a functional block diagram illustrating a second example of a functional configuration of the operation support system. FIG. 7 is a functional block diagram illustrating a third example of a functional configuration of the operation support system. FIG. 8 is a diagram illustrating a method for estimating the shape of a work object. FIG. 9 is a diagram illustrating an example of measured data and estimated data of a reaction force from the work object to a working part (bucket) during an excavation operation of the shovel. FIG. 10 is a diagram illustrating another example of measured data and estimated data of a reaction force from the work object to a working part (bucket) during an excavation operation of the shovel. FIG. 11 is a diagram illustrating yet another example of measured data and estimated data of a reaction force from the work object to a working part (bucket) during an excavation operation of the shovel. FIG. 12 is a diagram illustrating an example of a relationship between the accuracy of data representing the shape of the work object and a predetermined operation of the shovel. FIG. 13 is a diagram illustrating another example of a relationship between the accuracy of data representing the shape of the work object and a predetermined operation of the shovel. FIG. 14 is a diagram illustrating a first example of a process related to the use of data representing the shape of the work object. FIG. 1 is a diagram showing a third example of processing relating to the use of data representing the shape of a work object. FIG. 2 is a diagram showing a fourth example of processing relating to the use of data representing the shape of a work object. FIG. 3 is a flowchart showing a first example of processing for acquiring data representing the shape of a work object. FIG. 4 is a diagram showing an example of an observation target area. FIG. 5 is a diagram showing an example of an influence area. FIG. 6 is a flowchart showing a second example of processing for acquiring data representing the shape of a work object. FIG. 7 is a flowchart showing a third example of processing for acquiring data representing the shape of a work object.

[0014] Hereinafter, an embodiment will be described with reference to the drawings.

[0015] [Outline of Operation Support System] An outline of the operation support system SYS according to this embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0016] Fig. 1 is a diagram showing an example of an operation support system SYS. In Fig. 1, a left side view of a shovel 100 is shown. Fig. 2 is a top view showing an example of the shovel 100. Fig. 3 is a diagram showing an example of a configuration related to remote operation of the shovel 100. Hereinafter, the direction in which the attachment AT extends when viewed from above the shovel 100 (the upward direction in Fig. 2) will be defined as "front," and directions on the shovel 100 or directions seen from the shovel 100 may be described.

[0017] As shown in FIG. 1 , the operation support system SYS includes an excavator 100 , an information processing device 200 , and a sensor group 300 .

[0018] The operation support system SYS uses the information processing device 200 to cooperate with the shovel 100 and provide support regarding the operation of the shovel 100.

[0019] The operation support system SYS may include one or more excavators 100.

[0020] The excavator 100 is a work machine that receives support for operation in the operation support system SYS.

[0021] As shown in FIGS. 1 and 2 , the excavator 100 includes a lower traveling body 1 , an upper rotating body 3 , an attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cabin 10 .

[0022] The undercarriage 1 uses crawlers 1C to travel the excavator 100. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CL is hydraulically driven by a traveling hydraulic motor 1MR. This allows the undercarriage 1 to travel independently.

[0023] The upper rotating body 3 is mounted on the lower traveling body 1 so as to be rotatable (freely rotatable) via the rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by a rotating hydraulic motor 2M.

[0024] The boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down about a rotation axis that extends in the left-right direction. The arm 5 is attached to the tip of the boom 4 so as to be able to rotate about a rotation axis that also extends in the left-right direction. The bucket 6 is attached to the tip of the arm 5 so as to be able to rotate about a rotation axis that also extends in the left-right direction.

[0025] The bucket 6 is an example of an end attachment, and is used for, for example, excavation work, slope work, ground leveling work, and the like.

[0026] The bucket 6 is attached to the tip of the arm 5 in a manner that allows it to be appropriately replaced depending on the work content of the shovel 100. In other words, instead of the bucket 6, a bucket of a different type from the bucket 6, such as a relatively large bucket, a slope bucket, or a dredging bucket, may be attached to the tip of the arm 5. Also, a type of end attachment other than a bucket, such as a mixer, breaker, or crusher, may be attached to the tip of the arm 5. Also, a spare attachment such as a quick coupling or a tiltrotator may be provided between the arm 5 and the end attachment.

[0027] The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0028] The cabin 10 is a control room where an operator sits and operates the excavator 100. The cabin 10 is mounted on the front left side of the upper rotating body 3, for example.

[0029] The excavator 100 is equipped with a communication device 60 and can communicate with the information processing device 200 via a predetermined communication line NW.

[0030] The communication line NW may include, for example, a local area network (LAN) at a work site. The communication line NW may also include a wide area network (WAN). Examples of wide area networks include a mobile communication network terminated at a base station, a satellite communication network using a communication satellite, and the Internet. The communication line NW may also include, for example, a short-distance communication line based on a wireless communication standard such as Wi-Fi or Bluetooth (registered trademark).

[0031] For example, the excavator 100 operates driven elements such as the lower running body 1 (i.e., a pair of left and right crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6 in response to operations by an operator seated in the cabin 10.

[0032] Furthermore, instead of or in addition to being configured to be operable by an operator inside the cabin 10, the shovel 100 may be configured to be remotely operable from outside the shovel 100. When the shovel 100 is remotely operated, the interior of the cabin 10 may be unmanned. Furthermore, when the shovel 100 is exclusively for remote operation, the cabin 10 may be omitted. The following description will be given on the assumption that the operation of the operator includes at least one of operation of the operating device 26 by the operator inside the cabin 10 and remote operation by an external operator.

[0033] 3 , remote operation includes a mode in which the shovel 100 is operated by an operation input related to an actuator of the shovel 100, which is performed by a remote operation support device 400 that can communicate with the shovel 100 via a communication line NW. The remote operation support device 400 may be provided separately from the information processing device 200, or may be the information processing device 200.

[0034] The remote operation support device 400 is provided, for example, in a management center or the like that externally manages the work of the shovel 100. The remote operation support device 400 may also be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the work status of the shovel 100 from the vicinity of the shovel 100.

[0035] The shovel 100 may transmit to the remote operation support device 400, for example, via a communication device 60 (described later), an image showing the surroundings including the front of the shovel 100, based on an image captured by an imaging device mounted on the shovel 100 (hereinafter, a "peripheral image"). The shovel 100 may also transmit the captured image output by the imaging device to the remote operation support device 400 via the communication device 60, and the remote operation support device 400 may process the captured image received from the shovel 100 to generate a peripheral image. The remote operation support device 400 may then display the peripheral image showing the surroundings including the front of the shovel 100 on its own display device. Various information images (information screens) displayed on the output device 50 (display device 50A) inside the cabin 10 of the shovel 100 may also be displayed on the display device of the remote operation support device 400. This allows the operator using the remote operation support device 400 to remotely operate the shovel 100 while checking, for example, the contents of an image, information screen, or the like showing the surroundings of the shovel 100 displayed on the display device. The excavator 100 may then operate actuators in response to a remote control signal indicating the content of the remote control, which is received by the communication device 60 from the remote control support device 400, to drive driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6.

[0036] Remote control may also include, for example, a mode in which the shovel 100 is operated by an external voice input, gesture input, or the like to the shovel 100 by a person (e.g., a worker) around the shovel 100. Specifically, the shovel 100 recognizes voices uttered by surrounding workers and gestures made by the workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., an imaging device) mounted on the shovel 100. Then, the shovel 100 may operate actuators in accordance with the content of the recognized voices, gestures, or the like to drive driven elements such as the lower traveling body 1 (left and right crawlers 1C), upper rotating body 3, boom 4, arm 5, and bucket 6.

[0037] Furthermore, the excavator 100 may automatically operate the actuators regardless of the operation by the operator. This allows the excavator 100 to realize a function of automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, i.e., a so-called "automatic driving function" or "machine control (MC) function."

[0038] The automatic driving function includes, for example, a semi-automatic driving function (operation assistance type MC function). The semi-automatic driving function is a function that automatically operates driven elements (actuators) other than the driven element (actuator) to be operated in response to an operator's operation. The automatic driving function may also include a fully automatic driving function (fully automatic MC function). The fully automatic driving function is a function that automatically operates at least some of the multiple driven elements (hydraulic actuators) without operator operation. When the fully automatic driving function is enabled in the shovel 100, the interior of the cabin 10 may be unmanned. Furthermore, if the shovel 100 is exclusively for fully automatic operation, the cabin 10 may be omitted. Furthermore, the semi-automatic driving function, the fully automatic driving function, etc. include, for example, a rule-based automatic driving function. The rule-based automatic driving function is an automatic driving function in which the operation content of the driven element (actuator) to be operated automatically is determined according to predefined rules. Furthermore, the semi-automatic driving function, the fully automatic driving function, etc. may also include an autonomous driving function. The autonomous driving function is an automatic driving function in which the excavator 100 autonomously makes various decisions and determines the operation content of the driven element (hydraulic actuator) that is the target of automatic driving based on the results of those decisions.

[0039] Furthermore, the work of the shovel 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 400 may be provided. The remote monitoring support device is, for example, the information processing device 200. This allows a monitor, who is a user of the remote monitoring support device, to monitor the status of the work of the shovel 100 while checking a peripheral image displayed on a display device of the remote monitoring support device. Furthermore, for example, if the monitor determines it is necessary from a safety standpoint, the monitor can intervene in the operation by the operator or automatic operation of the shovel 100 and bring the shovel 100 to an emergency stop by making a predetermined input using an input device of the remote monitoring support device.

[0040] The information processing device 200 communicates with the shovel 100 to cooperate with each other and provide support regarding the operation of the shovel 100.

[0041] The information processing device 200 is, for example, a server device or a management terminal device installed in a management office within the work site of the shovel 100, or in a management center that manages the operating status of the shovel 100 and is located in a location different from the work site of the shovel 100. The server device may be an on-premises server, a cloud server, or an edge server. The management terminal device may be, for example, a fixed terminal device such as a desktop personal computer (PC), or a portable terminal device (mobile terminal) such as a tablet terminal, a smartphone, or a laptop PC. In the latter case, a worker at the work site, a supervisor who oversees the work, or a manager who manages the work site can move around the work site carrying the portable information processing device 200. In the latter case, an operator can, for example, bring the portable information processing device 200 into the cabin of the shovel 100.

[0042] The information processing device 200, for example, acquires data related to the operating state from the shovel 100. This enables the information processing device 200 to grasp the operating state of the shovel 100 and monitor the presence or absence of abnormalities in the shovel 100. Furthermore, the information processing device 200 can display data related to the operating state of the shovel 100 via a display device 208 (described later), for example, to allow a user to check the data. Furthermore, the information processing device 200 can, for example, train a learning model to learn the operating state of the shovel 100, and generate a trained model for supporting the operation of the shovel 100.

[0043] Furthermore, the information processing device 200 may transmit to the shovel 100 various data such as programs and reference data used in the processing of the controller 30, etc. In this way, the shovel 100 can perform various processes related to the operation of the shovel 100 using the various data downloaded from the information processing device 200.

[0044] The sensor group 300 is provided at the work site of the shovel 100.

[0045] For example, when the operation support system SYS includes a plurality of shovels 100, a sensor group 300 is provided for each shovel 100. Furthermore, when a plurality of shovels 100 included in the operation support system SYS perform work at the same work site, one sensor group 300 may be shared by the plurality of shovels 100.

[0046] The sensor group 300 includes sensors 300-1 to 300-M (M: an integer of 2 or greater). The sensors 300-1 to 300-M measure the state of objects at the work site around the shovel 100 and acquire measurement data relating to the state. The objects at the work site include the work target of the shovel 100. The work target is, for example, the earth and sand in the work area around the shovel 100. Furthermore, the objects at the work site include, in addition to the work target around the shovel 100 (earth and sand in the work area), other shovels around the shovel 100, work machines such as bulldozers, and work vehicles such as trucks for transporting earth and sand. The state of an object includes the shape and characteristics of the object.

[0047] The sensors 300-1 to 300-M include, for example, ranging sensors (distance sensors). Ranging sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensors, infrared sensors, and the like. The sensors 300-1 to 300-M may also include, for example, a 3D camera capable of acquiring data related to distance (depth) in addition to two-dimensional images, such as a stereo camera or a TOF (Time Of Flight) camera. The sensors 300-1 to 300-M may also include a mixture of ranging sensors and 3D cameras. This allows the sensor group 300 to acquire measurement data representing the shape of objects at the work site around the shovel 100. Hereinafter, sensors capable of acquiring measurement data representing the shape of objects, such as ranging sensors and 3D cameras, may be referred to as "shape sensors" for convenience.

[0048] Furthermore, the sensors 300-1 to 300-M may include a multi-wavelength spectroscopic camera. Examples of multi-wavelength spectroscopic cameras include a multispectral camera and a hyperspectral camera. This allows the sensor group 300 to acquire measurement data that represents the characteristics of objects at the work site around the shovel 100, such as the hardness and moisture content of soil and sand. Hereinafter, for convenience, a sensor that can acquire measurement data that represents the characteristics of an object, such as a multi-wavelength spectroscopic camera, may be referred to as a "characteristic sensor."

[0049] For example, sensors 300-1 to 300-M include multiple shape sensors. The multiple shape sensors may be installed in different locations on the work site around shovel 100, and the sensing range of each may overlap the sensing range of at least one other shape sensor. This may allow, for example, even if occlusion occurs and measurement data representing the shape of an object in a portion of the sensing range cannot be obtained with measurement data from one shape sensor, other shape sensors may be able to obtain measurement data representing the shape of the object in that range. This allows sensor group 300 to more reliably obtain measurement data representing the shape of objects on the work site around shovel 100.

[0050] Furthermore, sensors 300-1 to 300-M may include multiple characteristic sensors. The multiple characteristic sensors may be installed at different locations on the work site around shovel 100, and each sensorable range may overlap with at least one other characteristic sensor. This allows, for example, even if occlusion occurs in the measurement data of one characteristic sensor, making it impossible to acquire measurement data representing the characteristics of a portion of an object within the sensing range, other shape sensors may still be able to acquire measurement data representing the characteristics of the object within that range. Therefore, sensor group 300 can more reliably acquire measurement data representing the characteristics of objects on the work site around shovel 100.

[0051] Furthermore, sensors 300-1 to 300-M may include a sensor having both the function of a shape sensor and the function of a characteristic sensor (hereinafter referred to as an "integrated sensor"). In this case, sensors 300-1 to 300-M may include a plurality of integrated sensors. The plurality of characteristic sensors may be provided at different locations in the work site around shovel 100, and each sensorable range may overlap with at least one other characteristic sensor.

[0052] The sensor group 300 may simply include only one shape sensor or one characteristic sensor. Furthermore, instead of the sensor group 300, the operation support system SYS may simply include only one sensor that can acquire measurement data related to the state of objects at the work site around the shovel 100.

[0053] Sensors 300-1 to 300-M may be fixed to the work site around shovel 100, or may be mounted on a mobile object that can move within the work site around shovel 100. Mobile objects include, for example, work machines and work vehicles that move within the work site. Furthermore, mobile objects that can move within the work site may include, for example, flying objects such as drones that fly above the work site.

[0054] The outputs (measurement data) of the sensors 300-1 to 300-M are taken into the information processing device 200 via the communication line NW. The outputs of the sensors 300-1 to 300-M are taken into the information processing device 200 directly, for example, via the communication line NW. Furthermore, the outputs of the sensors 300-1 to 300-M may be once taken into the shovel 100 via the communication line NW, and then taken into the information processing device 200 via the shovel 100. Furthermore, when the sensors 300-1 to 300-M are mounted on a predetermined device such as the above-mentioned mobile body, the outputs of the sensors 300-1 to 300-M may be once taken into the predetermined device, and then taken from that device into the information processing device 200.

[0055] [Hardware Configuration of Operation Support System] The hardware configuration of the operation support system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS. 1 to 3.

[0056] The hardware configuration of the remote operation support device 400 may be the same as that of the information processing device 200. Therefore, illustration and description of the hardware configuration of the remote operation support device 400 will be omitted.

[0057] <Hardware Configuration of Shovel> FIG. 4 is a diagram illustrating an example of the hardware configuration of the shovel 100. As shown in FIG.

[0058] In FIG. 4, the paths through which mechanical power is transmitted are indicated by double lines, the paths through which high-pressure hydraulic oil that drives the hydraulic actuator flows are indicated by solid lines, the paths through which pilot pressure is transmitted are indicated by dashed lines, and the paths through which electrical signals are transmitted are indicated by dotted lines.

[0059] The shovel 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0060] 4 , the hydraulic drive system of the excavator 100 includes hydraulic actuators HA that hydraulically drive each of the driven elements, such as the lower traveling structure 1 (left and right crawlers 1C), upper rotating structure 3, boom 4, arm 5, and bucket 6, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0061] The hydraulic actuator HA includes traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

[0062] Note that the hydraulic actuators HA of the shovel 100 may be partially or entirely replaced with electric actuators. In other words, the shovel 100 may be a hybrid shovel or an electric shovel.

[0063] The engine 11 is the prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, on the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target rotation speed under direct or indirect control by a controller 30 (described later), for example, to drive the main pump 14 and the pilot pump 15.

[0064] It should be noted that instead of or in addition to the engine 11, another prime mover (for example, an electric motor) may be mounted on the excavator 100.

[0065] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

[0066] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0067] The control valve 17 drives the hydraulic actuators HA in response to an operator's operation of the operating device 26, the details of remote operation, or an operation command corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (directional control valves) that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0068] <Operation System> As shown in FIG. 4 , the operation system of the excavator 100 includes the pilot pump 15 , the operation device 26 , the hydraulic control valve 31 , the shuttle valve 32 , and the hydraulic control valve 33 .

[0069] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

[0070] The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic devices as pilot pressure.

[0071] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements, thereby enabling the operator to operate the driven elements that are driven by the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0072] For example, as shown in FIG. 4 , the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32 and is connected to the control valve 17 via a pilot line 27 connected to the outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0073] Alternatively, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0074] Furthermore, the control valves (directional control valves) that are built into the control valve 17 and drive the hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17 (i.e., to the electromagnetic solenoid type control valve).

[0075] As described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output control commands to the electric actuators or a driver that drives the electric actuators, depending on the operation content of the operation device 26 and the remote operation content specified by the remote operation signal. Furthermore, when the excavator 100 is remotely operated, the operation device 26 may be omitted.

[0076] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via a shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, for example, the controller 30 can supply pilot pressure according to an operation command corresponding to the automatic driving function from the hydraulic control valve 31 to the control valve 17, thereby realizing operation of the excavator 100 using the automatic driving function.

[0077] Furthermore, the controller 30 may control the hydraulic control valve 31 to realize remote operation of the shovel 100. Specifically, the controller 30 outputs, via the communication device 60, a control signal corresponding to the content of remote operation specified in a remote operation signal received from the remote operation assistance device 400 to the hydraulic control valve 31. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of remote operation to the control valve 17, thereby realizing operation of the shovel 100 based on remote operation by the operator.

[0078] In addition, if the operating device 26 is electric, the controller 30 can supply pilot pressure corresponding to the operation content (operation signal) of the operating device 26 directly to the control valve 17 from the hydraulic control valve 31, thereby realizing operation of the shovel 100 based on the operation of the operator.

[0079] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having a higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, a shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to a secondary pilot line 27A of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator HA that is the target of operation of the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6) regardless of the state of operation of the operating device 26 by the operator, thereby realizing an automatic operation function or a remote operation function.

[0080] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the automatic operation function and remote control function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0081] <User Interface System> As shown in FIG. 4 , the user interface system of the shovel 100 includes the operation device 26 , an output device 50 , and an input device 52 .

[0082] The output device 50 outputs various information to the user of the shovel 100 (e.g., the operator of the cabin 10 or an external remote operator) and people in the vicinity of the shovel 100 (e.g., workers or drivers of work vehicles).

[0083] For example, the output device 50 includes a lighting device or a display device 50A (see FIG. 7 ) that outputs various types of information visually. The lighting device is, for example, a warning light (indicator lamp). The display device 50A is, for example, a liquid crystal display or an organic EL (electroluminescence) display. For example, as shown in FIG. 2 , the lighting device or the display device 50A may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting device or the display device 50A may be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the excavator 100.

[0084] The output device 50 may also include a sound output device that outputs various types of information auditorily. Examples of sound output devices include a buzzer, a speaker, etc. The sound output device may be provided, for example, inside or outside the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the excavator 100.

[0085] The output device 50 may also include a device that outputs various information in a tactile manner, such as by vibrating the cockpit.

[0086] The input device 52 receives various inputs from the user of the excavator 100, and signals corresponding to the received inputs are taken into the controller 30. For example, as shown in Fig. 2, the input device 52 is provided inside the cabin 10 and receives inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3 and receive inputs from a worker or the like in the vicinity of the excavator 100.

[0087] For example, the input device 52 includes an operation input device that accepts input by mechanical operation from the user. The operation input device may include a touch panel mounted on the display device 50A, a touch pad installed around the display device 50A, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc.

[0088] The input device 52 may also include an audio input device that accepts audio input from the user. The audio input device includes, for example, a microphone.

[0089] The input device 52 may also include a gesture input device that accepts gesture inputs from the user. The gesture input device includes, for example, an imaging device that captures an image of a gesture made by the user.

[0090] The input device 52 may also include a biometric input device that accepts biometric input from the user, such as input of biometric information such as the user's fingerprint or iris.

[0091] <Communication System> As shown in FIG. 4 , the communication system of the shovel 100 according to this embodiment includes a communication device 60 .

[0092] The communication device 60 is connected to an external communication line NW and communicates with devices provided separately from the shovel 100. The devices provided separately from the shovel 100 may include devices external to the shovel 100 as well as portable terminal devices (mobile terminals) brought into the cabin 10 by the user of the shovel 100. The communication device 60 may be, for example, a 4G (4 th Generation) and 5G (5 th The communication device 60 may include a mobile communication module conforming to a standard such as the IEEE 802.11 Generation. The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module. If there are multiple connectable communication lines NW, the communication device 60 may include multiple communication devices in accordance with the types of the communication lines NW.

[0093] For example, the communication device 60 communicates with external devices such as the information processing device 200 and the remote operation support device 400 at the work site through a local communication line established at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line established at the work site or a local network using Wi-Fi 6.

[0094] The communication device 60 may also communicate with the information processing device 200 and the remote operation support device 400 outside the work site via a wide area communication line including the work site, that is, a wide area network.

[0095] <Control System> As shown in Fig. 4, the control system of the shovel 100 includes a controller 30. The control system of the shovel 100 according to this embodiment also includes an operating pressure sensor 29, a sensor 40, and sensors S1 to S9.

[0096] The controller 30 performs various controls related to the shovel 100 .

[0097] The functions of the controller 30 may be realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 3, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus BS1.

[0098] The auxiliary storage device 30A is a non-volatile storage means that stores the programs to be installed as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

[0099] For example, when a program start instruction is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0100] The CPU 30C executes, for example, a program loaded into the memory device 30B, and realizes various functions of the controller 30 according to instructions of the program.

[0101] The interface device 30D functions as, for example, a communication interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of communication interfaces in accordance with the types of communication lines to be connected.

[0102] The interface device 30D also functions as an external interface for reading data from a recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool connected to a connector installed inside the cabin 10 via a detachable cable. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may also be downloaded from another computer (for example, the information processing device 200) external to the excavator 100 via the communication device 60 and installed in the auxiliary storage device 30A.

[0103] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the shovel 100.

[0104] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure by the operating pressure sensor 29, which corresponds to the operating state of each driven element (hydraulic actuator HA) in the operating device 26, is input to the controller 30.

[0105] If the operating device 26 is an electric type, the operating pressure sensor 29 is omitted because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26.

[0106] The sensor 40 acquires measurement data relating to the shape of objects around the shovel 100, for example.

[0107] For example, the sensor 40 is a shape sensor, such as a distance measuring sensor or a 3D camera, that can acquire measurement data that represents the shape of an object around the shovel 100. The sensor 40 may also be an integrated sensor that has the function of a property sensor, such as a multi-wavelength spectroscopic camera, that can acquire measurement data that represents the properties of an object around the shovel 100, in addition to the function of a shape sensor.

[0108] For example, as shown in FIG. 2 , the sensor 40 includes sensors 40F, 40B, 40L, and 40R. Sensor 40F measures the state (shape and characteristics) of an object in front of the upper rotating body 3. Sensor 40B measures the state of an object on the upper rotating body 3. Sensor 40L measures the state of an object to the left of the upper rotating body 3. Sensor 40R measures the state of an object to the right of the upper rotating body 3. In this way, the sensor 40 can measure the state of objects around the entire circumference of the excavator 100, i.e., over a 360-degree angular range, when viewed from above the excavator 100. Hereinafter, sensors 40F, 40B, 40L, and 40R may be collectively or individually referred to as "sensor 40X."

[0109] The output data of the sensor 40 (sensor 40X) (i.e., measurement data relating to the state of objects around the shovel 100) is taken into the controller 30 via a one-to-one communication line or an in-vehicle network. This allows the controller 30 to grasp the state of the shapes, characteristics, etc. of objects around the shovel 100, for example, based on the output data of the sensor 40X.

[0110] It should be noted that some or all of the sensors 40B, 40L, and 40R may be omitted.

[0111] The sensor S1 is attached to the boom 4 and measures the attitude of the boom 4. The sensor S1 outputs measurement data representing the attitude of the boom 4. The attitude of the boom 4 is, for example, the attitude angle around the rotation axis of the base end of the boom 4, which corresponds to the connection point between the boom 4 and the upper rotating body 3 (hereinafter referred to as the "boom angle"). The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may apply to the sensors S2 to S4 below. The sensor S1 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may apply to the sensors S2 and S3 below. The output of the sensor S1 (measurement data representing the attitude of the boom 4) is input to the controller 30. This allows the controller 30 to grasp the attitude of the boom 4.

[0112] The sensor S2 is attached to the arm 5 and measures the posture of the arm 5. The sensor S2 outputs measurement data representing the posture of the arm 5. The posture of the arm 5 is, for example, the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The output of the sensor S2 (measurement data representing the posture of the arm 5) is input to the controller 30. This enables the controller 30 to grasp the posture of the arm 5.

[0113] The sensor S3 is attached to the bucket 6 and measures the attitude of the bucket 6. The sensor S3 outputs measurement data that indicates the attitude of the bucket 6. The attitude of the bucket 6 is, for example, the attitude angle (hereinafter referred to as the "arm angle") around the rotation axis of the base end of the bucket 6 that corresponds to the connection part with the arm 5. The output of the sensor S3 (measurement data that indicates the attitude of the bucket 6) is input to the controller 30. This enables the controller 30 to grasp the attitude of the bucket 6.

[0114] The sensor S4 measures the attitude state of the machine body (e.g., the upper rotating body 3) of the shovel 100. The sensor S4 outputs measurement data representing the attitude state of the machine body of the shovel 100. The attitude state of the machine body of the shovel 100 is, for example, the inclination state of the machine body with respect to a predetermined reference plane (e.g., a horizontal plane). For example, the sensor S4 is attached to the upper rotating body 3 and measures the inclination angles of the shovel 100 about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft inclination angle" and the "lateral inclination angle"). The output of the sensor S4 (measurement data representing the attitude state of the machine body of the shovel 100) is taken into the controller 30. This allows the controller 30 to grasp the attitude state (inclination state) of the machine body (upper rotating body 3).

[0115] The sensor S5 is attached to the upper rotating body 3 and measures the rotation state of the upper rotating body 3. The sensor S5 outputs measurement data representing the rotation state of the upper rotating body 3. The sensor S5 measures, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the sensor S5 (measurement data representing the rotation state of the upper rotating body 3) is input to the controller 30. This allows the controller 30 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0116] The controller 30 can grasp (estimate) the position of the tip (bucket 6) of the attachment AT based on the outputs of the sensors S1 to S5.

[0117] If the sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocities around three axes, the rotation state (e.g., rotation angular velocity) of the upper rotating body 3 may be detected based on the detection signal of the sensor S4. In this case, the sensor S5 may be omitted.

[0118] The sensor S6 measures the position of the shovel 100. The sensor S6 may measure the position in world (global) coordinates, or may measure the position in local coordinates at the work site. In the former case, the sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, the sensor S6 is a transceiver that communicates with equipment that serves as a reference for the position at the work site and is capable of outputting a signal corresponding to the position relative to the reference. The output of the sensor S6 is taken into the controller 30.

[0119] Sensor S7 measures the pressure (cylinder pressure) in the oil chamber of boom cylinder 7. Sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of boom cylinder 7, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side. The output of sensor S7 (measurement data of the cylinder pressure of boom cylinder 7) is taken into controller 30.

[0120] The sensor S8 measures the pressure (cylinder pressure) in the oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of the arm cylinder 8, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side of the arm cylinder 8. The output of the sensor S8 (measurement data of the cylinder pressure of the arm cylinder 8) is taken into the controller 30.

[0121] The sensor S9 measures the pressure (cylinder pressure) in the oil chamber of the bucket cylinder 9. The sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of the bucket cylinder 9, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side of the bucket cylinder 9. The output of the sensor S9 (measurement data of the cylinder pressure of the bucket cylinder 9) is taken into the controller 30.

[0122] The controller 30 can grasp the load state acting on the attachment AT based on the outputs of the sensors S7 to S9. The load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from the work target (earth and sand on the ground) and the weight of the earth and sand contained in the bucket 6.

[0123] In addition to the sensors S1 to S9, the shovel 100 may be equipped with other sensors capable of grasping the state of the shovel 100. For example, the shovel 100 may be equipped with an orientation sensor capable of detecting its own orientation. The orientation sensor is, for example, an electronic compass including a geomagnetic sensor.

[0124] <Hardware Configuration of Information Processing Apparatus> FIG. 5 is a diagram showing an example of the hardware configuration of the information processing apparatus 200. As shown in FIG.

[0125] The functions of the information processing device 200 are realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed calculation device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. These are connected by a bus BS2.

[0126] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. Examples of the recording medium 201A include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. This allows the information processing device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions.

[0127] The information processing device 200 may acquire various data and programs used in processing from an external device via the communication interface 206 .

[0128] The auxiliary storage device 202 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state disk (SSD), a flash memory, etc.

[0129] When an instruction to start a program is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a dynamic random access memory (DRAM) or an SRAM.

[0130] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the information processing device 200 in accordance with the programs.

[0131] The high-speed arithmetic unit 205 performs arithmetic processing at a relatively high speed in cooperation with the CPU 204. The high-speed arithmetic unit 205 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0132] The high-speed calculation unit 205 may be omitted depending on the required calculation processing speed.

[0133] The communication interface 206 is used as an interface for communicatively connecting with an external device. This allows the information processing device 200 to communicate with an external device such as the excavator 100 through the communication interface 206. The communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected device and the like.

[0134] The input device 207 receives various inputs from a user. The input device 207 includes a remote control operation device for remotely operating the excavator 100.

[0135] The input device 207 includes, for example, an input device that accepts mechanical operation input from a user (hereinafter referred to as an "operation input device"). The operation device for remote operation may be an operation input device. The operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0136] The input device 207 may also include a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0137] The input device 207 may also include a gesture input device capable of receiving gesture input from a user. The gesture input device may include, for example, a camera capable of capturing images of the user's gestures.

[0138] The input device 207 may also include a biometric input device capable of accepting biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about a user's fingerprint or iris.

[0139] The display device 208 displays an information screen or an operation screen for the user of the information processing device 200. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0140] The sound output device 209 conveys various pieces of information by sound to the user of the information processing device 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0141] [Functional Configuration of Operation Support System] The functional configuration of the operation support system SYS will be described with reference to Figures 6 to 14 in addition to Figures 1 to 5. Specifically, the functional configuration related to generation of a trajectory of a working portion of the excavator 100 in the operation support system SYS will be described.

[0142] <First Example> FIG. 6 is a functional block diagram showing a first example of the functional configuration of the operation support system SYS.

[0143] Hereinafter, the term "the trajectory of the working part of the shovel 100" will be used to include both the path that the working part of the shovel 100 has already traveled (i.e., the trajectory) and the path that it may travel in the future. The working part corresponds to the tip of the AT attachment used to make changes to the work target. Specifically, the working part is the bucket 6.

[0144] The shovel 100 includes an assistance device 150. In this example, the assistance device 150 provides assistance to the shovel 100 operating with an autonomous driving function in carrying out work.

[0145] As shown in FIG. 6, the support device 150 includes a controller 30, a hydraulic control valve 31, a sensor 40, and sensors S1 to S9.

[0146] The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302 .

[0147] When the operation support system SYS includes a plurality of shovels 100, there may be an shovel 100 in which the controller 30 includes only the operation log providing unit 301 and the work support unit 302, and another shovel 100 in which the controller 30 includes only the latter. In this case, the former shovel 100 has only the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the work support function of the latter shovel 100. The same may be true for second and third examples described below.

[0148] The information processing device 200 includes, as functional units, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

[0149] The operation log providing unit 301 is a functional unit for acquiring an operation log during a predetermined operation of the excavator 100 and providing the log to the information processing device 200 .

[0150] The predetermined operations include, for example, digging operations, boom-raising and swinging operations, boom-lowering and swinging operations, soil removal operations, broom operations, etc., which are used during excavation work. The predetermined operations may also include digging operations, soil removal operations, sweeping operations, leveling operations, compaction operations, broom operations, etc., which are used during ground leveling work. The predetermined operations may also include cutting operations, compaction operations, etc., which are used during slope work. The sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping soil and sand forward with the back of the bucket 6. In the sweeping operation, for example, the attachment AT lowers the boom 4 and opens the arm 5. The leveling operation is, for example, an operation in which the attachment AT is operated to move the tip of the bucket 6 approximately horizontally along the ground toward the user, thereby leveling out unevenness in the ground (surface of the terrain). In the horizontal towing operation, for example, the attachment AT raises the boom 4 and closes the arm 5. The rolling operation is, for example, an operation in which the attachment AT is operated to press the ground with the back surface of the bucket 6. The rolling operation may also be an operation in which the bucket 6 is moved up and down and the back surface of the bucket 6 is struck against the ground to press the ground. The rolling operation may also be an operation in which the bucket 6 is pushed forward along the ground, the back surface of the bucket 6 sweeps soil and sand to a predetermined position forward, and then the ground at the predetermined position is pressed against the ground with the back surface of the bucket 6. In the rolling operation, for example, the attachment AT lowers the boom 4 while pressing the ground. The broom operation is, for example, an operation in which the upper rotating body 3 is operated to rotate the bucket 6 left and right while keeping it along the ground. The broom operation may also be an operation in which the attachment AT and the upper rotating body 3 are operated to push the bucket 6 forward while alternately rotating left and right while keeping it along the ground. In the broom operation, for example, the upper rotating body 3 alternately rotates left and right. In addition to the alternating left and right rotation of the upper rotating body 3, the attachment AT may lower the boom 4 and open the arm 5, as in the sweeping operation.

[0151] The operation log of the shovel 100 is time-series data representing the operating state of the shovel 100. For example, the operation log of the shovel 100 includes time-series data representing the operation content of the operator. The time-series data representing the operation content of the operator is, for example, time-series output data of the operating pressure sensor 29 corresponding to the hydraulic pilot type operating device 26 or time-series output data (operation signal data) of the operating device 26 corresponding to the electric type operating device 26. Furthermore, the operation log of the shovel 100 may be time-series output data of the sensors S1 to S5 or time-series data representing the posture state of the shovel 100 acquired from the output data of the sensors S1 to S5.

[0152] For example, the operation log providing unit 301 acquires an operation log when the shovel 100 is operated by an operator who has a long history of operating the shovel 100 and is relatively experienced (hereinafter, for convenience, referred to as an "expert"), and provides the operation log to the information processing device 200. As a result, as will be described later, it is possible to generate a learned model LM2 that can reproduce the operation of the shovel 100 operated by an expert, by machine learning based on the operation log of the shovel 100.

[0153] Furthermore, as will be described later, the learned model LM2 may be omitted, and only the learned model LM1 may be generated. In this case, the operation log of the shovel 100 provided to the information processing device 200 may include an operation log when an operator other than an expert operates the shovel 100, or may include an operation log corresponding to the operation of the shovel 100 by the automatic driving function. Furthermore, the operation log of the shovel 100 for the learned model LM1 and the operation log of the shovel 100 for the learned model LM2 may be acquired separately.

[0154] The operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmission unit 301C.

[0155] The operation log recording unit 301A acquires an operation log during a predetermined operation of the shovel 100 and records the operation log in the operation log storage unit 301B. For example, every time a predetermined operation of the shovel 100 is executed, the operation log recording unit 301A records an operation log during the operation in the operation log storage unit 301B.

[0156] The operation log storage unit 301B stores an operation log of the shovel 100. For example, the operation log storage unit 301B stores, for each predetermined operation performed by the shovel 100, an operation log and data on the time (date and time) when the predetermined operation was performed, linked to each other. The data on the time when the predetermined operation was performed includes data on both the start and end times of the predetermined operation of the shovel 100. Furthermore, when a plurality of predetermined operations are specified, the operation log storage unit 301B stores, for each predetermined operation performed by the shovel 100, an operation log, data on the time when the predetermined operation was performed, and data on identification information of the performed predetermined operation, linked to each other. Hereinafter, data linked to the operation log of the shovel 100 may be referred to as "associated data" for convenience. For example, the operation log storage unit 301B accumulates record data indicating the correspondence between the operation log and the associated data for each predetermined operation performed by the shovel 100, thereby building a database of operation logs for the execution of the predetermined operations of the shovel 100.

[0157] The operation log stored in the operation log storage unit 301B that has already been transmitted to the information processing device 200 by the operation log transmission unit 301C (described later) may be deleted afterward.

[0158] The operation log transmission unit 301C transmits the operation log stored in the operation log storage unit 301B when the shovel 100 performs a predetermined operation and the associated data linked to the operation log to the information processing device 200 via the communication device 60. The operation log transmission unit 301C may also transmit record data indicating the correspondence between the operation log of the shovel 100 and the associated data for each predetermined operation performed by the shovel 100 to the information processing device 200.

[0159] For example, the operation log transmission unit 301C transmits to the information processing device 200 the operation log and associated data of the shovel 100 that have not yet been transmitted and that are stored in the operation log storage unit 301B, in response to a transmission request for the operation log of the shovel 100 received from the information processing device 200. Furthermore, the operation log transmission unit 301C may automatically transmit to the information processing device 200 the operation log and associated data of the shovel 100 that have not yet been transmitted and that are stored in the operation log storage unit 301B, at a predetermined timing. The predetermined timing is, for example, when the shovel 100 stops operating (when the key switch is turned off) or when it starts operating (when the key switch is turned on), etc.

[0160] The log acquisition unit 2001 acquires a log when the excavator 100 performs a predetermined operation.

[0161] The log when the shovel 100 performs a predetermined operation includes an operation log when the shovel 100 performs the predetermined operation and a status log of the work object. The status log of the work object includes data representing the status of the work object before and after the shovel 100 performs the predetermined operation. The status of the work object includes the shape of the work object (i.e., the topographical shape of the ground area of ​​the work object) and the characteristics of the soil and sand on the work object. The operation log when the shovel 100 performs a predetermined operation is uploaded from the shovel 100. The status log of the work object when the shovel 100 performs a predetermined operation is acquired based on the measurement data uploaded from the sensor group 300 and the associated data uploaded from the shovel 100 (data on the time the predetermined operation was performed).

[0162] The simulator unit 2002 performs a computer simulation of a predetermined operation of the shovel 100 using a virtual model of the shovel 100 and the work object (earth and sand).

[0163] For example, the distinct element method (DEM) is used to model the soil on the ground as a collection of minute particles. As a result, the simulator unit 2002 can virtually reproduce the overall behavior of the soil as a collection and the reaction force from the soil by having the virtual model of the shovel 100 perform a predetermined operation such as an excavation operation and analyzing the movement of each minute particle.

[0164] The simulator unit 2002 acquires data on the trajectory of the working part of the shovel 100 and data on the state of the work object (earth and sand) before and after the execution of the predetermined operation as logs when the shovel 100 executes a predetermined operation through computer simulation. The former data corresponds to an operation log when the shovel 100 executes a predetermined operation through computer simulation, and the latter data corresponds to a state log of the work object when the shovel 100 executes a predetermined operation through computer simulation.

[0165] The simulator unit 2002 performs computer simulations of numerous patterns relating to predetermined operations of the shovel 100, using various conditions of the work object (earth and sand) and various trajectories of the working parts of the shovel 100. In this way, the simulator unit 2002 can accumulate in the log storage unit 2003 logs when the shovel 100 performs predetermined operations through computer simulations under mutually different conditions.

[0166] The log storage unit 2003 stores, in an accumulated form, logs acquired by the log acquisition unit 2001 and the simulator unit 2002 when the shovel 100 performs a predetermined operation. For example, the log storage unit 2003 stores an operation log for each predetermined operation actually performed by the shovel 100 or performed by computer simulation, a status log of the work target, and associated data in a linked form. In the log storage unit 2003, the log acquired by the log acquisition unit 2001 and the log acquired by the simulator unit 2002 may be stored in a distinguishable manner, or may be stored mixed together in an indistinguishable manner.

[0167] The teacher data generation unit 2004 generates teacher data for machine learning based on the log stored in the log storage unit 2003 when the shovel 100 performs a predetermined operation, and outputs a teacher data set that is a collection of a large amount of teacher data. The teacher data generation unit 2004 may automatically generate teacher data by batch processing, or may generate teacher data in response to input from the user of the information processing device 200. The teacher data generation unit 2004 includes teacher data generation units 2004A and 2004B.

[0168] The teacher data generation unit 2004A generates a teacher data set for generating the trained model LM1. The trained model LM1 infers the shape of the work object after a predetermined operation of the shovel 100 is performed based on predetermined input data. The shape of the work object means, for example, the topographical shape of the ground surface of the work object, and more specifically, the shape of the soil and sand, such as undulations, that appear on the ground surface of the work object.

[0169] The input data corresponding to the trained model LM1 includes data representing the state of the work object before the shovel 100 performs a predetermined operation, and data on the trajectory (track) of the work part when the shovel 100 performs the predetermined operation. The state of the work object includes, for example, the shape of the work object. The state of the work object may also include the characteristics of the soil and sand that are the work object. For example, the data representing the characteristics of the soil and sand includes data on the angle of repose of the soil and sand. This allows the trained model LM1 to infer a more appropriate shape of the work object by taking the angle of repose of the soil and sand into account. The input data corresponding to the trained model LM1 may also include data representing an excavation reaction force when the shovel 100 performs a predetermined operation. The excavation reaction force is a reaction force acting from the ground against the working part of the shovel 100 (specifically, the bucket 6). This allows the trained model LM1 to estimate an appropriate soil and sand shape by taking the excavation reaction force into account, even in cases where, for example, the bucket 6 hits a rock or the like underground, resulting in failure to excavate the soil and sand.

[0170] The training data is a combination of input data of a type specified for the trained model LM1 and data (correct answer data) representing the correct inference result corresponding to the input data. The correct answer data is data representing the state of the work object after the shovel 100 has performed a predetermined operation corresponding to the input data included in the training data. Furthermore, if multiple types of predetermined operations are specified, a trained model LM1 may be generated for each type of predetermined operation. In this case, the training data generation unit 2004A generates a training data set for each type of predetermined operation.

[0171] The teacher dataset for generating the trained model LM1 is generated, for example, from both the log acquired by the log acquisition unit 2001 and the log output from the simulator unit 2002. The teacher dataset for generating the trained model LM1 may be generated from only the log acquired by the log acquisition unit 2001 and the log output from the simulator unit 2002. In this case, the simulator unit 2002 may be omitted. The teacher dataset for generating the trained model LM1 may be generated from only the log acquired by the log acquisition unit 2001 and the log output from the simulator unit 2002. In this case, the operation log providing unit 301 of the sensor group 300 and the excavator 100 may be omitted. The teacher dataset for generating the trained model LM1 may include a base teacher dataset and a teacher dataset for final adjustment (fine tuning). In this case, since a large amount of data is required, the base teacher dataset may be generated based on the log output from the simulator unit 2002, and the teacher dataset for final adjustment may be generated based on the log acquired by the log acquisition unit 2001. Hereinafter, the contents regarding the method for generating the trained model LM1 may be similarly applied to the trained models LM2 to LM4 described below, and may be incorporated into the method for generating the trained models LM2 to LM4.

[0172] The training data generation unit 2004B generates training data for generating the trained model LM2. The trained model LM2 infers a target trajectory of the working part during a predetermined operation of the shovel 100 based on predetermined input data.

[0173] The input data corresponding to the trained model LM2 includes, for example, data representing the state of the work object around the shovel 100. The data representing the state of the work object includes, for example, data representing the shape of the work object, as described above. The data representing the state of the work object may also include data representing the characteristics of the soil and gravel of the work object, as described above. The input data corresponding to the trained model LM2 may also include data representing the target shape of the work object. Furthermore, when the specified operation of the shovel 100 is an earth-discharging operation, the input data corresponding to the trained model LM2 may also include data representing the weight or volume of the soil and gravel contained in the bucket 6 before earth-discharging.

[0174] The training data is a combination of input data of a type specified for the trained model LM2 and data (correct answer data) representing a correct inference result corresponding to the input data. Specifically, the input data included in the training data includes data representing the state of the work object before the shovel 100 performs a predetermined operation. The input data included in the training data may also include data representing a target shape corresponding to the work object. If the predetermined operation of the shovel 100 is an earth-discharging operation, the input data included in the training data includes data representing the weight or volume of earth and sand contained in the bucket 6 before earth-discharging. The correct answer data included in the training data includes data representing the trajectory of the work part when the shovel 100 performs a predetermined operation through operation by an expert, assuming the state of the work object corresponding to the input data. In other words, the training data generation unit 2004B generates a training data set based on the log acquired by the log acquisition unit 2001 when the shovel 100 performs a predetermined operation through operation by an expert. If multiple types of predetermined operations are specified, a trained model LM2 may be generated for each type of predetermined operation. In this case, the teacher data generation unit 2004B generates a teacher data set for each type of predetermined motion.

[0175] The machine learning unit 2005 generates trained models LM1 and LM2 by performing machine learning on the base learning model based on the teacher data set generated by the teacher data generation unit 2004. The trained model (base learning model) includes, for example, a neural network such as a deep neural network (DNN).

[0176] The machine learning unit 2005 includes machine learning units 2005A and 2005B.

[0177] The machine learning unit 2005A performs machine learning on the base learning model M1 based on the teacher data set output from the teacher data generation unit 2004A. As a result, the machine learning unit 2005A can generate a learned model LM1 that can output (infer) the state of the work object after the shovel 100 performs the predetermined operation, using inputs such as data on the state of the work object before the shovel 100 performs a predetermined operation and data on the trajectory of the work part when the shovel 100 performs the predetermined operation. For example, the machine learning unit 2005A can optimize the learning model M1 using an error backpropagation algorithm based on the error between the output data of the learning model M1 for the input data included in the teacher data and the correct data, thereby generating the learned model LM1. The same may be true for generating the learned models LM2 to LM4 described below. The machine learning unit 2005A may also correct the learned model LM1 by additionally learning it so as to reduce the error between the inference results of the learned model LM1 and the actual measurement results of the sensor 40. In this case, the inference results from the learned model LM1 and the data of the actual measurement results from the sensor 40 are uploaded from the shovel 100 to the information processing device 200 and used for additional learning.

[0178] The machine learning unit 2005B causes the base learning model M2 to perform machine learning based on the teacher data set output from the teacher data generation unit 2004 B. As a result, the machine learning unit 2005B can generate a learned model LM2 that is capable of outputting (inferring) a target trajectory of a working part in a predetermined operation of the shovel 100, using data on the state of the work target around the shovel 100 as input.

[0179] Furthermore, the machine learning unit 2005B may generate the learned model LM2 by applying reinforcement learning instead of supervised learning to the learning model M2. In this case, the teacher data generation unit 2004B is omitted. For example, the machine learning unit 2005B trains the learning model M1 by machine learning based on logs acquired by the log acquisition unit 2001 and the simulator unit 2002 so as to maximize a predetermined reward related to work efficiency. At this time, the machine learning unit 2005B works in conjunction with the simulator unit 2002 and causes the simulator unit 2002 to try out many operation patterns, thereby enabling the learning model M1 to be trained more efficiently by machine learning.

[0180] The trained model storage unit 2006 stores the trained models LM1 and LM2 output by the machine learning unit 2005. Furthermore, when the machine learning unit 2005A re-learns or additionally learns the trained model LM1, the trained model LM1 in the trained model storage unit 2006 is updated. The same applies when the trained model LM2 is re-learned or additionally learned by the machine learning unit 2005B. Furthermore, when the trained model LM1 is updated, the trained model LM1 before the update may be stored in the trained model storage unit 2006 or another storage unit in a reusable form. The same may be true for the trained model LM2 before the update when the trained model LM2 is updated. This allows, for example, when there is a problem with the updated trained model LM1 or the trained model LM2, to restore and reuse the trained model LM1 before the update or the trained model LM2 before the update.

[0181] The distribution unit 2007 distributes data of the learned models LM1 and LM2 to the excavator 100.

[0182] For example, when the machine learning unit 2005A generates or updates the trained model LM1, the distribution unit 2007 distributes the most recently generated or updated trained model LM1 to the shovel 100. In addition, the distribution unit 2007 may distribute the latest trained model LM1 in the trained model storage unit 2006 to the shovel 100 in response to a signal received from the shovel 100 requesting distribution of the trained model LM1. The same may be true for the trained model LM2.

[0183] The work support unit 302 is a functional unit for providing work support to the excavator 100 operating using an autonomous driving function.

[0184] The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, a target trajectory generation unit 302C, and a motion control unit 302D.

[0185] The trained model storage unit 302A stores trained models LM1 and LM2 that are distributed from the information processing device 200 and received via the communication device 60.

[0186] The work object shape acquisition unit 302B acquires data representing the shape of the work object of the shovel 100, based on the trajectory (locus) of the work part when the shovel 100 performs a predetermined operation. Specifically, the work object shape acquisition unit 302B may acquire data representing the shape of the work object of the shovel 100 by estimating changes in the terrain shape (shape of the soil and sand) from before the execution of the predetermined operation, based on the trajectory of the work part when the shovel 100 performs the predetermined operation. The work object shape acquisition unit 302B may also acquire data representing the shape of the work object of the shovel 100, taking into account measurement data of the properties of the soil and sand on the work object obtained by sensor 40 (property sensor) and measurement data from sensors S7 to S9 (i.e., data related to the reaction force from the soil and sand on the work object to the work part).

[0187] In this case, the work object shape acquisition unit 302B may interpolate the shape of the soil or sand at locations that cannot be measured by the sensor 40 based on the trajectory of the working part when the shovel 100 performs a predetermined operation, assuming measurement data of the shape of the work object obtained by the sensor 40. Alternatively, the work object shape acquisition unit 302B may acquire data representing the shape of the work object without using the sensor 40, while estimating changes in the shape of the soil or sand based on the trajectory of the working part each time the shovel 100 performs a predetermined operation, based on the initial state of the shape of the work object of the shovel 100. In this case, the sensor 40 may be omitted. The initial state of the shape of the work object of the shovel 100 is acquired, for example, by being distributed from outside the shovel 100 via the communication device 60 or by being input by the user via the input device 52. Alternatively, the initial state (initial shape) of the shape of the work object of the shovel 100 may be fixed, for example, as a flat surface at the same height as the ground on which the crawlers of the shovel 100 are placed.

[0188] For example, the work object shape acquisition unit 302B uses the learned model LM1 to estimate the shape of the soil after the shovel 100 performs a predetermined operation, based on the shape of the soil before the shovel 100 performs the predetermined operation and the trajectory of the working part during the shovel 100's performance. The work object shape acquisition unit 302B may then acquire data representing the current state of the work object by integrating the above estimation result with measurement data of the soil shape after the shovel 100 performs the predetermined operation, obtained by the sensor 40. Specifically, the work object shape acquisition unit 302B performs data integration by interpolating the data of the above estimation result for locations in the observation target area around the shovel 100 where the sensor 40 was unable to measure the soil shape. The observation target area refers to the range around the shovel 100 from which data representing the shape of the work object of the shovel 100 is acquired by the work object shape acquisition unit 302B. As a result, even if the sensor 40 is unable to acquire measurement data of the work object in a part of the observation area due to occlusion or the like, the controller 30 can acquire data representing the shape of the work object in the observation area, including the shape of the work object in that part.

[0189] The shape of the work object before the execution of the predetermined operation of the shovel 100 corresponds to, for example, the previous output by the work object shape acquisition unit 302B. The trajectory of the work part during the predetermined operation of the shovel 100 is acquired based on, for example, the outputs of the sensors S1 to S6.

[0190] If an obstacle is detected within the observation area around the shovel 100 based on the output of the sensor 40, the work object shape acquisition unit 302B may exclude the area where the obstacle exists from the area targeted for estimating the shape of the work object. Examples of obstacles include moving objects such as construction machines and work vehicles, and features such as utility poles and fences. Furthermore, if multiple shovels 100 are operating at the same work site, the multiple shovels 100 may share data representing the trajectories (tracks) of the work parts when performing predetermined operations. This allows the work object shape acquisition unit 302B (trained model LM1) to take into account changes in the shape of the work object due to predetermined operations of other shovels 100. Therefore, the work object shape acquisition unit 302B (trained model LM1) can more appropriately estimate the shape of the work object around the shovel 100. Some or all of the functions of the work object shape acquisition unit 302B may be transferred to an external device (e.g., the information processing device 200) with relatively high processing power. As a result, even if the controller 30's own processing power is insufficient, the controller 30 can acquire data representing the shape of the work object after the shovel 100 has performed the specified operation, taking into account the trajectory of the work part when the shovel 100 is performing the specified operation.

[0191] The target trajectory generating unit 302C generates a target trajectory for the specified operation of the shovel 100 based on the estimation result of the work object shape acquiring unit 302B (the state of the work object after the execution of the specified operation of the shovel 100).

[0192] For example, the target trajectory generation unit 302C generates a target trajectory of the work part during a specified operation of the shovel 100 using the learned model LM2 based on the shape of the work object acquired by the work object shape acquisition unit 302B.

[0193] The target trajectory generating unit 302C may generate a target trajectory for the working part of the shovel 100 that is tailored to the state (prediction result) of the work object around the shovel 100 by applying any known method instead of the learned model LM2. In this case, the teacher data generating unit 2004C and the machine learning unit 2005B may be omitted. For example, the target trajectory generating unit 302C may generate data of the target trajectory for the working part of the shovel 100 by using model predictive control (MPC) based on the shape of the work object acquired by the work object shape acquiring unit 302B. The target trajectory generating unit 302C may also generate data of the target trajectory for the working part of the shovel 100 by optimizing a predetermined reference trajectory for the working part of the shovel 100 based on the shape of the work object acquired by the work object shape acquiring unit 302B.

[0194] The operation control unit 302D causes the shovel 100 to perform a predetermined operation so that a predetermined part of the shovel 100 moves along the target trajectory generated by the target trajectory generating unit 302C. Specifically, the operation control unit 302D controls the hydraulic control valve 31 while grasping the position of the working part from the outputs of the sensors S1 to S5, etc., thereby causing the shovel 100 to perform a predetermined operation so that the working part of the shovel 100 moves along the target trajectory. This allows the shovel 100 to autonomously proceed with work while performing a predetermined operation in accordance with the shape of the work object.

[0195] As described above, in this example, the controller 30 acquires data representing the state of the shovel 100 and related to the shape of the work target in accordance with the predetermined operation of the shovel 100. The controller 30 then estimates the shape of the work target (ground surface) of the shovel 100 based on the acquired data. Specifically, the controller 30 acquires data representing the trajectory (track) of the working part of the shovel 100 when the predetermined operation of the shovel 100 is being performed, and estimates the shape of the work target (ground surface) after the predetermined operation of the shovel 100 is performed based on the acquired data. As a result, even in a situation where the sensor 40 cannot measure the soil shape at a certain location due to occlusion or the like, the controller 30 can interpolate the soil shape at that location by taking into account changes in the soil shape associated with the movement of the working part of the shovel 100. Furthermore, the shape of the work target of the shovel 100 can be estimated based on the history of changes in the soil shape associated with the movement of the working part of the shovel 100, even without using the sensor 40. Therefore, the controller 30 can more appropriately grasp the shape of the soil and sand that is the work target of the shovel 100 within the observation target area, and as a result, can more appropriately generate the target trajectory of the work portion of the shovel 100. Therefore, the controller 30 can more appropriately cause the shovel 100 to perform autonomous operation.

[0196] Note that some or all of the functions of the work object shape acquisition unit 302B, the target trajectory generation unit 302C, and the operation control unit 302D may be transferred to the information processing device 200. This makes it possible to reduce the processing load on the shovel 100 for processing related to generation of the target trajectory of the working part of the shovel 100 and processing related to control of the operation of the shovel 100. The same may be true for a third example described below.

[0197] Second Example FIG. 7 is a functional block diagram showing a second example of the functional configuration of the operation support system SYS.

[0198] In the following, the same or corresponding components as those in the first example described above will be denoted by the same reference numerals, and the description will focus on the differences from the first example described above.

[0199] As in the first example described above, the shovel 100 includes the support device 150. In this example, the support device 150 supports a user who operates the shovel 100 to perform work or who monitors the work of the shovel 100.

[0200] 7, in this example, the support device 150 includes a controller 30, a sensor 40, a display device 50A, and sensors S1 to S9. In addition, when the shovel 100 is remotely operated, the support device 150 may include a communication device 60.

[0201] The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302, similar to the first example described above.

[0202] As in the first example described above, the operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmission unit 301C.

[0203] As functional units, the information processing device 200 includes, similarly to the first example described above, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

[0204] In this example, the information processing device 200 differs from the first example described above in that the teacher data generation unit 2004B, the machine learning unit 2005B, and the trained model LM2 are omitted.

[0205] The work support unit 302 is a functional unit for providing support to a user who operates the shovel 100 to perform work or who monitors the work of the shovel 100 .

[0206] The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, and a display processing unit 302E. In other words, the work support unit 302 differs from the first example described above in that the target trajectory generation unit 302C and the operation control unit 302D are omitted and a display processing unit 302E is added.

[0207] The display processing unit 302E causes the display device 50A to display a screen relating to work support for a user who operates the shovel 100 to perform work or who monitors the work of the shovel 100. The display processing unit 302E may also transmit data corresponding to a similar screen to the remote operation support device 400 or the remote monitoring support device via the communication device 60, and cause the data to be displayed on the remote operation support device 400 or the remote monitoring support device.

[0208] For example, the display processing unit 302E displays an image representing the shape of the work target (ground) of the shovel 100 on the display device 50A based on the output of the work target shape acquisition unit 302B (data on the shape of the work target after the shovel 100 has performed a predetermined operation). The display processing unit 302E may also transmit image data representing the shape of the work target of the shovel 100 to the remote operation support device 400 or the remote monitoring support device via the communication device 60, and display the image on the remote operation support device 400 or the remote monitoring device. This allows the user to operate the shovel 100 or monitor the work of the shovel 100 while understanding the shape of the work target of the shovel 100 through the image representing the shape of the work target of the shovel 100 displayed on the display device 50A or the like. The display device 50A or the like can update the image representing the shape of the work target of the shovel 100 in accordance with the execution of a predetermined operation of the shovel 100. Therefore, the user can more appropriately operate the shovel 100 and monitor the work being performed by the shovel 100 while understanding the shape of the work target of the shovel 100 in real time.

[0209] In this way, in this example, the controller 30 more appropriately grasps the shape of the soil that is the target of work by the shovel 100 within the observation area, and as a result, the user can more appropriately grasp the shape of the soil that is the target of work through the display of an image representing the shape of the soil. Therefore, the user can more appropriately operate the shovel 100 and more appropriately monitor the work of the shovel 100 while grasping the shape of the soil that is the target of work.

[0210] When the shovel 100 is remotely operated, some or all of the functions of the learned model storage unit 302A, the work object shape acquisition unit 302B, and the display processing unit 302E may be provided in the remote operation assistance device 400. Furthermore, the function of the work object shape acquisition unit 302B may be transferred to the information processing device 200. This reduces the processing load on the shovel 100.

[0211] <Third Example> FIG. 8 is a functional block diagram showing a third example of the functional configuration of the operation support system SYS. FIG. 9 is a diagram illustrating a method for estimating the shape of a work object. Specifically, FIG. 9 is a diagram illustrating a method for estimating the shape of a work object after repeating a predetermined operation of the shovel 100 multiple times (n times) starting from the initial shape of the work object. FIGS. 10 to 12 are diagrams illustrating an example, another example, and yet another example of measurement data and estimated data of the reaction force from the work object to the working part (bucket 6) during the excavation operation of the shovel 100. FIGS. 13 and 14 are diagrams illustrating an example and another example of the relationship between the accuracy of the estimation result of the shape of the work object by the shovel 100 and the predetermined operation of the shovel 100.

[0212] 13 and 14, the accuracy of the estimation result of the shape of the work target of the shovel 100 is defined as a value between the lowest "0" and the highest "1." Also, in FIG. 13, the speed multiplier means the multiplier of the speed of a predetermined operation of the shovel 100, based on normal times, i.e., a state in which the operation of the shovel 100 is not restricted. Also, in FIG. 14, the digging depth multiplier means the multiplier of the digging depth of the shovel 100, based on normal times, i.e., a state in which the operation of the shovel 100 is not restricted. The speed multiplier and the digging depth multiplier are defined as "1" when the operation of the shovel 100 is not restricted, and are expressed as positive values ​​smaller than "1" when the operation of the shovel 100 is restricted.

[0213] In the following, the same or corresponding components as those in the first and second examples will be denoted by the same reference numerals, and the description will focus on the differences from the first and second examples.

[0214] Similar to the first and second examples described above, the shovel 100 includes the assistance device 150. In this example, similar to the first example described above, the assistance device 150 provides assistance to the shovel 100 operating using the autonomous driving function in carrying out work.

[0215] 8, the support device 150 includes the controller 30, the hydraulic control valve 31, and the sensors S1 to S9, similar to the first and second examples described above. In this example, unlike the first and second examples described above, the excavator 100 is not equipped with the sensor 40, and the support device 150 does not include the sensor 40.

[0216] The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302, similar to the first and second examples described above.

[0217] As in the first and second examples, the operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmission unit 301C.

[0218] As functional units, the information processing device 200 includes, similarly to the first and second examples described above, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

[0219] The information processing device 200 in this example differs from the first example described above in that teacher data generation units 2004C and 2004D and machine learning units 2005C and 2005D are added, and that learned models LM3 and LM4 are stored in the learned model storage unit 2006. This example also differs from the first example described above in that the predetermined motion of the shovel 100 to be processed is limited to motion in which the soil to be processed comes into contact with the working part. The predetermined motion of the shovel 100 to be processed in this example includes, for example, an excavation motion. The predetermined motion of the shovel 100 to be processed in this example may also include a sweeping motion, a compaction motion, or a broom motion.

[0220] The teacher data generation unit 2004 includes teacher data generation units 2004A to 2004D.

[0221] The training data generation unit 2004C generates a training data set for generating the trained model LM3. The trained model LM3 infers the reaction force acting from the work object to the work part based on predetermined input data.

[0222] The input data corresponding to the trained model LM3 includes data representing the shape of the work target before the execution of a predetermined operation of the shovel 100, and data on the trajectory (track) of the work part when the predetermined operation is executed by the shovel 100. The input data corresponding to the trained model LM3 may also include data representing the characteristics of the soil and sand that is the work target of the shovel 100.

[0223] The training data is a combination of input data of a type specified for the trained model LM3 and data (correct answer data) representing the correct inference result corresponding to the input data. Specifically, the input data included in the training data includes data representing the shape of the work object before the shovel 100 performs a predetermined operation, and data representing the trajectory (track) of the work part when the shovel 100 performs the predetermined operation. The input data included in the training data may also include data representing the characteristics of the soil and sand in the work object when the shovel 100 performs the predetermined operation. The correct answer data included in the training data includes time-series data representing the reaction force acting from the work object to the work part when the shovel 100 performs the predetermined operation. When multiple types of predetermined operations are specified, a trained model LM3 may be generated for each type of predetermined operation. In this case, the training data generation unit 2004C generates a training data set for each type of predetermined operation.

[0224] The teacher data generation unit 2004D generates a teacher data set for generating a trained model LM4. The trained model LM4 infers the accuracy of the estimation result of the shape of the work object of the shovel 100. The estimation result of the shape of the work object of the shovel 100 includes the estimation result of the shape of the work object of the shovel 100 by the work object shape acquisition unit 302B. The estimation result of the shape of the work object of the shovel 100 may also include the initial shape of the work object recognized by the work object shape acquisition unit 302B. This is because, in this example, the shovel 100 is not equipped with a sensor 40, and the initial shape of the work object cannot be observed by the shovel 100. The accuracy of the estimation result of the shape of the work object of the shovel 100 refers to the degree of certainty of the estimation result of the shape of the work object of the shovel 100. The accuracy of the estimation result of the shape of the work object by the shovel 100 is specified so that it becomes higher as the difference between the estimation result of the shape of the work object by the shovel 100 and the actual shape of the work object becomes smaller, and becomes lower as the difference becomes larger. Furthermore, a relatively high accuracy of the estimation result of the shape of the work object by the shovel 100 indicates, in other words, that the uncertainty of the estimation result is relatively low, and a relatively low accuracy of the estimation result indicates, in other words, that the uncertainty of the estimation result is relatively high. Therefore, the accuracy of the estimation result of the shape of the work object by the shovel 100 also means the degree of uncertainty of the estimation result of the shape of the work object by the shovel 100. Furthermore, the accuracy of the estimation result of the shape of the work object by the shovel 100 is specified, for example, for each of a plurality of small regions obtained by dividing the observation target area of ​​the shape of the work object into multiple regions. Therefore, the accuracy of the estimation result of the shape of the work object of the shovel 100 is expressed, for example, as a vector defined by the accuracy of the estimation result of the shape of each small region of the work object of the shovel 100.

[0225] For example, the learned model LM4 infers the accuracy of the corrected shape of the work object of the shovel 100 under the assumption that the estimated result of the shape of the work object before the shovel 100 performs a predetermined operation is corrected using data obtained when the shovel 100 subsequently performs the predetermined operation. For example, as described below, the estimated result of the shape of the work object before the shovel 100 performs a predetermined operation is corrected based on the error between the estimated result and the measurement result of the reaction force acting from the work object to the work part when the shovel 100 performs the predetermined operation. The shape of the work object before the shovel 100 performs a predetermined operation is, for example, the shape of the work object immediately before one predetermined operation of the shovel 100 is performed. Furthermore, the shape of the work object before the shovel 100 performs a predetermined operation may be the shape of the work object of the shovel 100 before the start of repeated predetermined operations of the shovel 100 (i.e., the initial shape).

[0226] In other words, the learned model LM4 may infer the accuracy of the estimated result of the shape of the work object after correction and before the shovel 100 performs a specified operation for input data including data representing the estimated result of the shape of the work object after correction and before the shovel 100 performs a specified operation.

[0227] The input data of the trained model LM4 includes data representing the estimated result of the shape of the work object after correction before the execution of the predetermined operation of the shovel 100, and data obtained when the predetermined operation of the shovel 100 is performed. The data obtained when the operation of the shovel 100 is performed includes, for example, data representing the trajectory of the working part when the predetermined operation of the shovel 100 is performed. Furthermore, the data obtained when the predetermined operation of the shovel 100 is performed may include data representing the reaction force acting from the work object on the working part when the predetermined operation of the shovel 100 is performed. Furthermore, the input data of the trained model LM4 may include data representing the characteristics of the soil and sand that is the work object of the shovel 100, similar to the trained model LM1, etc.

[0228] The training data is a combination of input data of the trained model LM4 and data (correct answer data) representing the correct inference result corresponding to the input data. The training data includes, as the correct answer data, data (correct answer data) representing the correct answer of the accuracy of the shape of the work target before the execution of a predetermined operation of the shovel 100, corresponding to the input data.

[0229] The machine learning unit 2005 includes machine learning units 2005A to 2005D.

[0230] The machine learning unit 2005C generates a trained model LM3 by performing machine learning on the base learning model M3 based on the teacher data set output from the teacher data generation unit 2004C. The machine learning unit 2005C may also correct the trained model LM3 by additionally training it so as to reduce the error between the output (inference result) of the trained model LM3 and the actual reaction force corresponding to the shape of the work object.

[0231] The machine learning unit 2005D performs machine learning on the base learning model M4 based on the teacher data set output from the teacher data generation unit 2004D to generate a trained model LM4. The machine learning unit 2005D may also correct the trained model LM4 by additionally training it so as to reduce the error between the output (inference result) of the trained model LM4 and the actual accuracy.

[0232] The trained model storage unit 2006 stores the trained models LM1 to LM4 output to the machine learning unit 2005. Furthermore, when the machine learning unit 2005A re-learns or performs additional learning on the trained model LM1, the trained model LM1 in the trained model storage unit 2006 is updated. The same applies when the trained models LM2 to LM4 are re-learned or additionally learned by each of the machine learning units 2005B to 2005D. Furthermore, the trained models LM1 to LM4 before updating may be stored in the trained model storage unit 2006 or another storage unit in a reusable form. This allows, for example, if there is a problem with the trained models LM1 to LM4 after updating, to restore and reuse the trained models LM1 to LM4 before updating.

[0233] The distribution unit 2007 distributes data of the learned models LM1 to LM4 to the excavator 100.

[0234] The distribution method of the trained models LM3 and LM4 may be similar to the above-mentioned distribution method of the trained models LM1 and LM2.

[0235] As in the first example described above, the work support unit 302 is a functional unit that provides work support to the excavator 100 operating using the autonomous driving function.

[0236] The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, a target trajectory generation unit 302C, a movement control unit 302D, a reaction force estimation unit 302F, and an accuracy estimation unit 302G.

[0237] The trained model storage unit 302A stores trained models LM1 to LM4 that are distributed from the information processing device 200 and received via the communication device 60.

[0238] The reaction force estimation unit 302F estimates the reaction force acting on the working part during a specified operation of the shovel 100 based on the state of the work object before the shovel 100 performs the specified operation and the trajectory of the working part when the shovel 100 performs the specified operation.

[0239] For example, the reaction force estimation unit 302F uses the learned model LM3 to estimate the reaction force acting on the working part during a specified operation of the shovel 100, based on the shape of the work object before the shovel 100 performs a specified operation and the trajectory of the working part when the shovel 100 performs the specified operation.

[0240] The work object shape acquisition unit 302B estimates the shape of the work object being worked on by the shovel 100 based on the difference between the reaction force estimation result, obtained by the reaction force estimation unit 302F, of the reaction force acting on the working part during a predetermined operation of the shovel 100 and the actual reaction force measurement result, and acquires data representing that shape. Specifically, the work object shape acquisition unit 302B corrects the estimated shape of the work object before the shovel 100 performs the predetermined operation based on the difference (estimation error) between the reaction force estimation result acting on the working part during a predetermined operation of the shovel 100 and the actual reaction force measurement result. The reaction force measurement data of the shovel 100 is calculated, for example, from the measurement data of sensors S7 to S9. The work object shape acquisition unit 302B then estimates the shape of the work object after the shovel 100 performs the predetermined operation based on the corrected estimation result of the shape of the work object before the shovel 100 performs the predetermined operation and the trajectory of the working part during the shovel 100 performs the predetermined operation.

[0241] For example, as shown in FIG. 9 , the work object shape acquisition unit 302B uses data representing the initial shape of the work object acquired in advance as a starting point and estimates the shape of the work object after each execution of a predetermined operation of the shovel 100 using a learned model LM1. Hereinafter, the initial shape corresponding to the data representing the initial shape of the work object of the shovel 100 may be conveniently referred to as a "tentative initial shape" to distinguish it from the actual initial shape. Furthermore, the reaction force estimation unit 302F estimates the reaction force on the work part of the shovel 100 using a learned model LM3 based on data representing the estimated shape of the work object before each execution of a predetermined operation of the shovel 100 (estimated soil shape) and data representing the trajectory (track) of the work part each time. In this case, in the case of the first execution of the predetermined operation of the shovel 100, the data representing the estimated shape of the work object before the execution of the predetermined operation of the shovel 100 is data representing the initial shape of the work object. The trajectory of the work part each time may be data representing a target trajectory generated by the target trajectory generating unit 302C, or may be measurement data of the trajectory based on the outputs of the sensors S1 to S3.

[0242] In addition, the work object shape acquisition unit 302B calculates the estimated error of the reaction force on the work part when the shovel 100 performs a specified operation each time, based on the data of the reaction force estimation result (estimated reaction force) by the reaction force estimation unit 302F and the data of the actual reaction force measurement result.

[0243] Then, the work object shape acquisition unit 302B corrects the data representing the initial shape of the work object of the shovel 100 based on the estimated error of the reaction force on the work part when the shovel 100 performs each predetermined operation. Specifically, the work object shape acquisition unit 302B applies a known mathematical programming algorithm to correct the tentative initial shape of the work object so as to minimize the estimated error of the reaction force.

[0244] For example, the accuracy of the provisional initial shape of the soil to be worked on may be low, and the height position may be set lower than the actual initial shape. As a result, as shown in FIG. 10 , the working part of the shovel 100 may not come into contact with the soil to be worked on at all during a predetermined operation, resulting in a missed swing. Alternatively, as shown in FIG. 11 , the working part of the shovel 100 may come into contact with the soil to be worked on only part of the entire movement section where it should come into contact, resulting in a missed swing. Furthermore, the measured reaction force is smaller than the estimated reaction force throughout the entire predetermined operation of the shovel 100, and the estimation error obtained by subtracting the measured reaction force from the estimated reaction force is a positive and relatively large value. In this case, the work object shape acquisition unit 302B corrects the ground height position of the work object to be higher for at least a small area of ​​the observation target area of ​​the work object of the shovel 100 through which the working part passed during the predetermined operation of the shovel 100.

[0245] Conversely, the accuracy of the provisional initial shape of the soil and sand to be worked on may be low, resulting in the elevation being set higher than the actual elevation of the initial shape. As a result, as shown in FIG. 12 , the working part may move on a trajectory that passes relatively deep below the ground surface of the work object during a predetermined operation of the shovel 100. Furthermore, the measured reaction force is greater than the estimated reaction force from the beginning of the predetermined operation of the shovel 100. In this example, the measured reaction force reaches the allowable upper limit, and the shovel 100 stops the predetermined operation and performs an avoidance operation to move the working part away from the work object (see the dashed line in the figure). In this case, the work object shape acquisition unit 302B corrects the ground elevation of the work object to a lower position for at least a small area of ​​the observation area of ​​the work object through which the working part passed during the predetermined operation of the shovel 100 (excluding the small area passed through by the avoidance operation).

[0246] The work object shape acquisition unit 302B starts from data representing the initial shape of the work object after correction and sequentially estimates the shape of the work object after each predetermined operation of the shovel 100, based on data representing the trajectory (track) of the work part during each predetermined operation of the shovel 100. In this way, the work object shape acquisition unit 302B can estimate the shape of the work object after the nth predetermined operation of the shovel 100 is performed, and acquire data representing the latest shape of the work object.

[0247] The work object shape acquisition unit 302B corrects the data representing the initial shape of the work object each time a predetermined operation of the shovel 100 is completed, estimates the latest shape of the work object based on the data representing the corrected shape of the work object, and acquires data representing that shape. This allows the work object shape acquisition unit 302B to sequentially correct the provisional initial shape starting from the provisional initial shape of the work object each time the shovel 100 performs a predetermined operation, even if there is a discrepancy between the provisional initial shape and the actual initial shape at the start of work. Therefore, the work object shape acquisition unit 302B sequentially reduces the error between the data representing the initial shape of the work object of the shovel 100 and the data representing the actual initial shape, and as a result, can estimate the latest shape of the work object of the shovel 100 with relatively high accuracy. Therefore, for example, even if measurement data representing the initial shape cannot be acquired externally or if the measurement data representing the initial shape acquired externally has relatively low accuracy for some reason, the shovel 100 can use data representing the shape of the work object with relatively high accuracy.

[0248] The accuracy estimation unit 302G estimates the accuracy of the estimation result of the shape of the work object of the shovel 100. The estimation result of the shape of the work object of the shovel 100 may be the estimation result of the latest shape of the work object estimated by the work object shape acquisition unit 302B, or may be the estimation result of the shape of the work object before the shovel 100 performs a predetermined operation. Furthermore, the estimation result of the shape of the work object of the shovel 100 may be a tentative initial shape of the work object of the shovel 100. For example, the accuracy estimation unit 302G estimates the accuracy of the corrected shape of the work object before the shovel 100 performs a predetermined operation, based on data representing the trajectory of the working part when the shovel 100 performs the predetermined operation and data representing the reaction force acting on the working part. This is because, as described above, the shape of the work object before the shovel 100 performs a predetermined operation is corrected based on data representing the trajectory of the working part when the shovel 100 performs the predetermined operation and data representing the reaction force acting on the working part. In this case, the accuracy of the estimation result of the shape of the work object before the shovel 100 performs the predetermined operation is, for example, the accuracy of the provisional initial shape of the work object after correction. Specifically, the accuracy estimation unit 302G uses the learned model LM4 to estimate the accuracy of the estimation result of the corrected shape of the work object before the shovel 100 performs the predetermined operation. The accuracy estimation unit 302G may also estimate the accuracy of the estimation result of the corrected shape of the work object before the shovel 100 performs the predetermined operation on a rule-based basis. For example, the accuracy estimation unit 302G determines, for each small area of ​​the observation target area of ​​the work object, whether the working part contacted the work object (ground) when the shovel 100 performed the predetermined operation. Then, for the small area of ​​the target, if the working part contacted the work object, the accuracy estimation unit 302G may increase the accuracy by a relatively large amount from the current value, and if not, may increase the accuracy by a relatively small amount or maintain it at the current value. This is because the shape of the work object before the shovel 100 performs the predetermined operation is likely to be corrected around the small area that the work part comes into contact with when the shovel 100 performs the predetermined operation.

[0249] The operation control unit 302D causes the shovel 100 to perform a predetermined operation so that the working part of the shovel 100 moves along the target trajectory generated by the target trajectory generating unit 302C. At this time, the operation control unit 302D may restrict the operation of the shovel 100 based on the accuracy of the estimation result of the shape of the work target of the shovel 100 estimated by the accuracy estimation unit 302G. This is because, when the accuracy of the estimation result of the shape of the work target of the shovel 100 is relatively low, a mismatch between the estimation result of the shape of the work target of the shovel 100 and the target trajectory may cause a reaction force acting on the working part to become an issue, and a large impact may be applied to the shovel 100. Specifically, the operation control unit 302D may calculate the average accuracy (average accuracy) of small areas on the target trajectory when viewed from above the shovel 100. The operation control unit 302D may increase the degree of restriction on the operation of the shovel 100 when the average accuracy is relatively low, and may decrease the degree of restriction on the operation of the shovel 100 when the average accuracy is relatively high.

[0250] For example, as shown in FIG. 13 , the operation control unit 302D changes the speed multiplier corresponding to the speed of a predetermined operation of the shovel 100 based on the average accuracy. In this example, the speed multiplier is set to 1 when the average accuracy exceeds a threshold TH1 (<1), and is set to decrease linearly with a decrease in the average accuracy when the average accuracy is equal to or less than the threshold TH1. As a result, when the average accuracy is equal to or less than the threshold TH1, the operation control unit 302D limits the operation speed of the shovel 100, allowing the shovel 100 to perform a predetermined operation at a speed slower than when there is no operation limit. Therefore, even if the working portion of the shovel 100 unintentionally comes into contact with the work object due to a mismatch between the estimated shape of the work object and the target trajectory, the impact acting on the shovel 100 can be kept relatively small.

[0251] 14 , the operation control unit 302D changes the excavation depth multiplier corresponding to the excavation operation of the shovel 100 based on the average accuracy. In this example, the excavation depth multiplier is set to 1 when the average accuracy exceeds a threshold value TH2 (<1) and is set to decrease linearly with the decrease in the average accuracy when the average accuracy is equal to or less than the threshold value TH2. As a result, when the average accuracy is equal to or less than the threshold value TH2, the operation control unit 302D limits the excavation depth during the excavation operation of the shovel 100, thereby limiting the range in which the working part can move toward the work target (ground side) compared to when no operation limit is in place. Therefore, even if the accuracy of the estimation result of the shape of the work target by the shovel 100 is low and the actual ground level is higher than expected, it is possible to prevent a situation in which the working part penetrates very deep into the ground and a relatively large impact is applied to the shovel 100. For example, the operation control unit 302D limits the excavation depth by correcting the target trajectory generated by the target trajectory generation unit 302C. In addition, the target trajectory generation unit 302C may generate a target trajectory for an excavation depth according to the average accuracy, and the operation control unit 302D may execute a predetermined operation of the shovel 100 based on the target trajectory, thereby resulting in the operation control unit 302D limiting the excavation depth.

[0252] Furthermore, for a predetermined operation of the shovel 100 other than the excavation operation, the operation control unit 302D may limit the vertical movement range of the working part during the predetermined operation of the shovel 100 based on the average accuracy.

[0253] As described above, in this example, the controller 30 acquires data representing the state of the shovel 100 and relating to the shape of the work target in accordance with the predetermined operation of the shovel 100. The controller 30 then estimates the shape of the soil and sand that is the work target (ground surface) of the shovel 100 based on the acquired data. Specifically, the controller 30 acquires data representing the trajectory (track) of the shovel 100 when performing the predetermined operation, and estimates the reaction force acting on the working part when the shovel 100 performs the predetermined operation based on that data. The controller 30 also acquires measurement data of the reaction force acting on the working part when the shovel 100 performs the predetermined operation, and corrects the shape of the work target before the shovel 100 performs the predetermined operation based on the difference (estimation error) between the reaction force measurement data and the estimated data. The controller 30 then estimates the shape of the work target after the shovel 100 performs the predetermined operation based on the corrected data representing the shape of the work target before the shovel 100 performs the predetermined operation and the data representing the trajectory of the shovel 100 when performing the predetermined operation. As a result, even if the accuracy of the shape of the work target before the execution of the predetermined operation of the shovel 100 is low, the shape of the work target before the execution of the predetermined operation of the shovel 100 can be corrected in accordance with the predetermined operation of the shovel 100. Therefore, the controller 30 can more appropriately grasp the shape of the soil and sand that is the work target of the shovel 100 within the observation target area, and as a result, can more appropriately generate the target trajectory of the work portion of the shovel 100. Therefore, the controller 30 can cause the shovel 100 to more appropriately perform autonomous operation.

[0254] Note that some or all of the functions of the reaction force estimation unit 302F and the accuracy estimation unit 302G may be transferred to the information processing device 200. This makes it possible to reduce the processing load on the shovel 100.

[0255] <Other Examples> The first to third examples of the functional configuration of the operation support system SYS described above may be modified or changed as appropriate.

[0256] For example, in the second example of the functional configuration of the operation support system SYS described above, the sensor 40 may be omitted from the shovel 100, and the work object shape acquisition unit 302B may estimate the shape of the work object of the shovel 100 and acquire data representing that shape in a manner similar to that of the third example described above. In this case, a functional unit similar to the reaction force estimation unit 302F of the third example described above may be added to the work support unit 302. Also, in this case, the work support unit 302 may be added with the function of the accuracy estimation unit 302G of the third example described above and part of the function of the operation control unit 302D, i.e., a function to limit a predetermined operation of the shovel 100 based on the accuracy of the estimation result of the shape of the work object of the shovel 100.

[0257] Furthermore, in the third example of the functional configuration of the operation support system SYS described above, the function of the accuracy estimation unit 302G and some of the functions of the operation control unit 302D (the function of restricting a specified operation of the shovel 100 based on the accuracy of the estimated result of the shape of the work target of the shovel 100) may be omitted.

[0258] [Specific example of processing related to the use of data representing the shape of the work object] Next, with reference to Figures 15 to 18, a specific example of processing related to the use of data representing the shape of the work object of the shovel 100, which is acquired by the work object shape acquisition unit 302B, will be described.

[0259] <First Example> Fig. 15 is a diagram showing a first example of processing related to the use of data representing the shape of the work target. Specifically, Fig. 15 is a diagram showing one example of processing in which the controller 30 autonomously operates the shovel 100 based on the data representing the shape of the work target to perform excavation work in the work target area.

[0260] Excavation work is performed by repeating a series of operations: excavation operation, boom raising and swinging operation, soil discharge operation, and boom lowering and swinging operation. The excavation operation is the operation of the attachment AT for excavating the soil and sand at the work target with the bucket 6. For example, the excavation operation is realized as a combined operation of at least two driven elements: the boom 4, the arm 5, and the bucket 6. The boom raising and swinging operation is the operation of the attachment AT and the upper rotating body 3 for scooping up the soil and sand excavated in the excavation operation with the bucket 6 and transporting the soil and sand to a discharge site away from the work target area. For example, the boom raising and swinging operation is realized as a combined operation of raising the boom 4 and swinging the upper rotating body 3. The soil discharge operation is the operation of the attachment AT for discharging the soil and sand from the bucket 6 onto the ground at the discharge site. For example, the soil discharge operation is realized as a combined operation of opening the arm 5 and opening the bucket 6. The boom lowering and swinging operation is an operation of the attachment AT and the upper rotating body 3 to return the bucket 6 to the work area. The boom lowering and swinging operation is realized, for example, as a combined operation of lowering the boom 4 and swinging the upper rotating body 3.

[0261] This flowchart starts, for example, when an input is received that indicates the start of excavation work by autonomous operation of the shovel 100. The input may be made, for example, by an operator in the cabin 10 via the input device 52, or may be made from outside the shovel 100 (the information processing device 200 or the remote operation support device 400) via the communication device 60. The same may be true for an input that indicates the start of loading work by autonomous operation of the shovel 100.

[0262] As shown in Fig. 15, in step S10, the controller 30 sets a target area for excavation work (work target area). The target area for excavation work is defined, for example, by the range in a planar view where excavation work is to be performed and the depth to be excavated. The range in a planar view where excavation work is to be performed within the work target area corresponds to the observation target area of ​​the work target shape acquisition unit 302B. The work target area may be set in response to input from an operator in the cabin 10 via the input device 52, or may be set by input from outside the excavator 100 (for example, the information processing device 200 or the remote operation support device 400) via the communication device 60.

[0263] When the process of step S10 is completed, the controller 30 proceeds to step S11.

[0264] In step S11, the controller 30 sets an initial shape of the work object. For example, if the sensor 40 is mounted on the shovel 100, the initial shape of the work object is set based on measurement data acquired by the sensor 40 before work begins. Alternatively, if the sensor 40 is not mounted on the shovel 100, the initial shape of the work object may be set to a predefined temporary shape. The temporary shape is, for example, a planar shape obtained by extending the contact plane of the undercarriage 1 of the shovel 100 to the work object area. Alternatively, the initial shape of the work object may be set manually in response to input from an operator or the like. Alternatively, the initial shape of the work object may be set based on measurement data of the shape of the work object around the shovel 100 that is input from the outside (for example, the information processing device 200 or the remote operation support device 400) via the communication device 60.

[0265] When the process of step S11 is completed, the controller 30 proceeds to step S12.

[0266] In step S12, the target trajectory generating unit 302C of the controller 30 generates a target trajectory for the working portion of the shovel 100 based on data representing the estimated result of the shape of the current work object.

[0267] When the process moves from step S11 to step S12, the data representing the estimation result of the current work object shape is the data representing the initial shape of the work object set in step S12. When the process moves from step S16 to step S12, the data representing the estimation result of the current work object shape is the data representing the latest work object shape acquired by the work object shape acquisition unit 302B.

[0268] When the process of step S12 is completed, the controller 30 proceeds to step S13.

[0269] In step S13, the operation control unit 302D of the controller 30 sets operation conditions for the excavation operation of the shovel 100. For example, the operation control unit 302D may set a speed condition for the excavation operation based on the accuracy of the estimation result of the shape of the work object estimated by the accuracy estimation unit 302G. The operation control unit 302D may also set an excavation depth condition for the excavation operation based on the accuracy of the estimation result of the shape of the work object estimated by the accuracy estimation unit 302G.

[0270] When the process of step S13 is completed, the controller 30 proceeds to step S14.

[0271] In step S14, the operation control unit 302D causes the excavator 100 to perform an excavation operation based on the target trajectory generated in step S12 and the operation conditions set in step S13.

[0272] When the process of step S14 is completed, the controller 30 proceeds to step S15.

[0273] In step S15, the controller 30 performs processing to acquire data representing the shape of the work target (earth and sand shape data) for the observation target area defined based on the work target area set in step S10. Details of this processing will be described later.

[0274] When the process of step S15 is completed, the controller 30 proceeds to step S16.

[0275] In step S16, the controller 30 determines whether a condition indicating the end of work (work end condition) has been met. For example, the controller 30 compares the set depth of the work area with the height position of the sediment in the work area in the latest sediment shape data acquired in step S15, and determines that the work end condition has been met if excavation has progressed to the set depth of the work area. If the work end condition has not been met, the controller 30 returns to step S12 and repeats the processing from step S12 to step S16. If the work end condition has been met, the controller 30 ends the processing of this flowchart.

[0276] In this way, in this example, the controller 30 can cause the shovel 100 to autonomously perform excavation work in the work area.

[0277] <Second Example> Fig. 16 is a diagram showing a second example of processing related to the use of data representing the shape of a work object. Specifically, Fig. 16 is a diagram showing an example of processing in which the controller 30 autonomously operates the excavator 100 based on data representing the shape of the work object to load earth and sand onto a truck.

[0278] The work of loading a truck is performed by repeating a series of operations: an excavation operation, a boom-raising and swinging operation, an earth-discharging operation, and a boom-lowering and swinging operation. The excavation operation is an operation of the attachment AT for scooping up soil and sand from a predetermined location on the ground into the bucket 6. For example, the excavation operation is realized as a combined operation of at least two driven elements: the boom 4, the arm 5, and the bucket 6. The boom-raising and swinging operation is an operation of the attachment AT and the upper rotating body 3 for carrying the soil and sand scooped up into the bucket 6 onto the truck bed. For example, the boom-raising and swinging operation is realized as a combined operation of raising the boom 4 and swinging the upper rotating body 3. The earth-discharging operation is an operation of the attachment AT for discharging the soil and sand from the bucket 6 onto the truck bed. For example, the earth-discharging operation is realized as a combined operation of opening the arm 5 and opening the bucket 6. The boom lowering and swinging operation is an operation of the attachment AT and the upper rotating body 3 for returning the bucket 6 to a predetermined location for scooping up the soil and sand. The boom lowering and swinging operation is realized, for example, as a combined operation of lowering the boom 4 and swinging the upper rotating body 3. In addition, in this example, the explanation will be given on the assumption that the position and shape of the pile of soil and sand to be loaded (sand pile) generated as a result of the excavation work are known. This is because the position from which the excavator 100 discharges soil during the excavation work is determined in advance with the excavator 100 as a reference, and the general shape of the pile of soil and sand to be generated can be roughly estimated taking into account the size and depth of the work area.

[0279] This flowchart starts, for example, when an input indicating the start of loading work by autonomous operation of the excavator 100 is received.

[0280] As shown in FIG. 16 , in step S20, the controller 30 detects the relative position of the truck to be loaded with earth and sand with respect to the shovel 100. For example, when the absolute positions of the shovel 100 and the truck can be ascertained, the controller 30 detects the relative position of the truck with respect to the shovel 100 based on the absolute positions of the shovel 100 and the truck. In this case, the controller 30 detects the relative position of the truck with respect to the shovel 100 based on, for example, the output of a GNSS sensor mounted on the shovel 100 and position information of the truck received from the truck via the communication device 60. Furthermore, the controller 30 may receive information on the relative position of the truck with respect to the shovel 100 from an external device (e.g., the information processing device 200) via the communication device 60, the information being acquired based on the output of an imaging device, a distance sensor, or the like installed at the work site. Furthermore, when a sensor S6 is mounted on the shovel 100, the controller 30 may detect the relative position of the truck with respect to the shovel 100 by recognizing the truck based on the output of the sensor S6.

[0281] When the process of step S20 is completed, the controller 30 proceeds to step S21.

[0282] In step S21, the controller 30 sets an initial shape of the work object. In this example, the controller 30 estimates the position of the truck bed based on the position of the truck relative to the excavator 100 detected in step S20, and sets the truck bed as the work object, with its bottom surface as the initial shape.

[0283] Note that, if the direction in which the truck is parked with respect to the shovel 100 has not been determined, the relative direction of the truck with respect to the shovel 100 may be detected in step S20. For example, if the respective direction of the shovel 100 and the truck can be determined, the controller 30 detects the relative direction of the truck with respect to the shovel 100 based on the respective direction of the shovel 100 and the truck. In this case, the controller 30 detects the direction of the truck with respect to the shovel 100 based on, for example, the output of a direction sensor and information on the direction of the truck received from the truck via the communication device 60. Furthermore, the controller 30 may receive information on the relative direction of the truck with respect to the shovel 100 from an external device (e.g., the information processing device 200, etc.) via the communication device 60, the information being acquired based on the output of an imaging device, a distance sensor, or the like installed at the work site. Furthermore, if a sensor S6 is mounted on the shovel 100, the controller 30 may detect the relative direction of the truck with respect to the shovel 100 by recognizing the truck based on the output of the sensor S6.

[0284] When the process of step S21 is completed, the controller 30 proceeds to step S22.

[0285] In step S22, the target trajectory generating unit 302C of the controller 30 generates a target trajectory for the working portion of the shovel 100 based on data representing the estimated shape of the current work object.

[0286] When the process moves from step S21 to step S22, the data representing the estimation result of the current work object shape is the data representing the initial shape of the work object set in step S22. When the process moves from step S25 to step S22, the data representing the estimation result of the current work object shape is the data representing the latest work object shape acquired by the work object shape acquisition unit 302B.

[0287] When the process of step S22 is completed, the controller 30 proceeds to step S23.

[0288] In step S23, the operation control unit 302D of the controller 30 causes the shovel 100 to perform a series of operations that combine excavation operations, boom-raising and swinging operations, soil-discharging operations, and boom-lowering and swinging operations along the target trajectory generated in step S22.

[0289] When the process of step S23 is completed, the controller 30 proceeds to step S24.

[0290] In step S24, the controller 30 performs processing to acquire data representing the shape of the work target (earth and sand shape data) for the observation target area corresponding to the truck bed detected in step S20. Details of this processing will be described later.

[0291] When the process of step S24 is completed, the controller 30 proceeds to step S26.

[0292] In step S26, the controller 30 determines whether a condition indicating the end of work (work end condition) has been met. For example, the controller 30 determines that the work of loading the soil onto the truck has ended if the height of the soil from the bottom of the loading platform is relatively large compared to a predetermined standard, based on data representing the shape of the work object. If the work end condition has been met, the controller 30 ends the processing of this flowchart. If the work end condition has not been met, the controller 30 returns to step S22 and repeats the processing from step S22 to step S26.

[0293] In this example, in step S24, data representing the shape of the work target may be data representing the shape of the soil in the loading platform of the truck (soil shape data), as well as data representing the shape of the soil in the loading platform of the truck (soil shape data). In this case, in step S22, the target trajectory generating unit 302C of the controller 30 generates a target trajectory for the series of operations based on both the soil shape data of the soil in the loading platform of the truck and the soil shape data of the loading platform of the truck, both of which are acquired in step S24.

[0294] In this way, in this example, the controller 30 can autonomously execute the loading operation.

[0295] <Third Example> Fig. 17 is a diagram showing a third example of processing related to the use of data representing the shape of a work object. Specifically, Fig. 17 is a diagram showing an example of processing in which, when excavator 100 performs excavation work while performing an excavation operation, controller 30 displays an image representing the shape of the work object on display device 50A or the like based on the data representing the shape of the work object.

[0296] This flowchart starts, for example, when an input indicating the start of manual excavation work is received from an operator. The input may be made, for example, by the operator in the cabin 10 via the input device 52, or by an operator performing remote operation via the remote operation support device 400. In the latter case, the input content at the remote operation support device 400 is received by the controller 30 via the communication device 60.

[0297] As shown in FIG. 17, the processes in steps S30 and S31 are the same as the processes in steps S10 and S11 in FIG. 15, and therefore a description thereof will be omitted.

[0298] When the process of step S31 is completed, the controller 30 proceeds to step S32.

[0299] In step S32, the display processing unit 302E of the controller 30 displays an image (initial shape image) representing the initial shape of the soil to be worked on on the display device 50A or the display device of the remote operation support device 400 based on the data representing the initial shape set in step S31.

[0300] The display processing unit 302E of the controller 30 transmits a display command including data representing an image of the initial shape to the remote operation assistance device 400 via the communication device 60, thereby causing the display device of the remote operation assistance device 400 to display the initial shape image. The same may be true for step S42 in Fig. 18 described later.

[0301] When the process of step S32 is completed, the controller 30 proceeds to step S33.

[0302] In step S33, the controller 30 monitors the operation of the shovel 100 based on the outputs of the sensors S1 to S9.

[0303] When the process of step S33 is completed, the controller 30 proceeds to step S34.

[0304] In step S34, the controller 30 determines whether the excavation operation has been performed and completed. If the excavation operation has been performed and completed, the controller 30 proceeds to step S35, and otherwise proceeds to step S37.

[0305] In step S35, the controller 30 performs processing to acquire data representing the shape of the work target (earth and sand shape data) for the observation target area defined based on the work target area set in step S30. Details of this processing will be described later.

[0306] When the process of step S35 is completed, the controller 30 proceeds to step S36.

[0307] In step S36, the display processing unit 302E of the controller 30 updates the image (soil shape image) representing the shape of the soil to be worked on, which is displayed on the display device 50A or the display device of the remote operation support device 400, based on the soil shape data acquired in step S35.

[0308] The display processing unit 302E of the controller 30 updates the soil shape image on the display device of the remote operation support device 400 by transmitting an update command including data representing the latest soil shape image to the remote operation support device 400 via the communication device 60. The same may be applied to step S46 in Fig. 18 described below.

[0309] When the process of step S36 is completed, the controller 30 proceeds to step S37.

[0310] In step S37, the controller 30 determines whether a condition for terminating the display of the soil shape image (display termination condition) is met. For example, the display termination condition may be that a predetermined input is received from an operator or the like via the input device 52 or the input device of the remote operation support device 400. Alternatively, the display termination condition may be that no excavation operation is being performed for a certain period of time or more. If the display termination condition is not met, the controller 30 returns to step S33 and repeats the processing from step S33 to step S37. If the display termination condition is met, the controller 30 terminates the processing of this flowchart.

[0311] In this way, in this example, the controller 30 can update an image showing the shape of the soil and gravel in the work area in accordance with the progress of excavation work in the work area and display it on the display device 50A or the display device of the remote operation support device 400.

[0312] <Fourth Example> Fig. 18 is a diagram showing a fourth example of processing related to the use of data representing the shape of a work object. Specifically, Fig. 18 is a diagram showing an example of processing in which, when the excavator 100 is loading earth and sand into a truck, the controller 30 displays an image representing the shape of the work object on the display device 50A or the like based on the data representing the shape of the work object.

[0313] This flowchart starts, for example, when an input indicating the start of manual loading work is received from an operator. The input may be made, for example, by the operator in the cabin 10 via the input device 52, or by an operator performing remote operation via the remote operation support device 400. In the latter case, the input content from the remote operation support device 400 is received by the controller 30 via the communication device 60.

[0314] As shown in FIG. 18, the processes in steps S40 and S41 are the same as the processes in steps S20 and S21 in FIG. 16, and therefore a description thereof will be omitted.

[0315] When the process of step S41 is completed, the controller 30 proceeds to step S42.

[0316] In step S42, the display processing unit 302E of the controller 30 displays an image (initial shape image) representing the initial shape of the soil to be worked on on the display device 50A or the display device of the remote operation support device 400 based on the data representing the initial shape set in step S41.

[0317] When the process of step S42 is completed, the controller 30 proceeds to step S43.

[0318] In step S43, the controller 30 monitors the operation of the shovel 100 based on the outputs of the sensors S1 to S9.

[0319] When the process of step S43 is completed, the controller 30 proceeds to step S44.

[0320] In step S44, the controller 30 determines whether the earth unloading operation has been executed and completed. If the earth unloading operation has been executed and completed, the controller 30 proceeds to step S45, otherwise proceeds to step S47.

[0321] In step S45, the controller 30 performs processing to acquire data representing the shape of the work target (earth and sand shape data) for the observation target area corresponding to the truck bed detected in step S40. Details of this processing will be described later.

[0322] When the process of step S45 is completed, the controller 30 proceeds to step S46.

[0323] In step S46, the display processing unit 302E of the controller 30 updates the image (soil shape image) representing the shape of the soil to be worked on, which is displayed on the display device 50A or the display device of the remote operation support device 400, based on the soil shape data acquired in step S45.

[0324] When the process of step S46 is completed, the controller 30 proceeds to step S47.

[0325] In step S47, the controller 30 determines whether a condition for terminating the display of the soil shape image (display termination condition) is met. For example, the display termination condition may be that a predetermined input is received from an operator or the like via the input device 52 or the input device of the remote operation support device 400. Alternatively, the display termination condition may be that a state in which no soil discharge operation is being performed continues for a certain period of time or more. If the display termination condition is not met, the controller 30 returns to step S43 and repeats the processing from step S43 to step S47. If the display termination condition is met, the controller 30 terminates the processing of this flowchart.

[0326] In this way, in this example, the controller 30 can display on the display device 50A and the display device of the remote operation support device 400 an image showing the shape of the soil and sand in the bed of the truck, updating it in accordance with the progress of the work of loading soil and sand into the bed of the truck, which is the work object.

[0327] [Specific Example of Processing Relating to Acquisition of Data Representing the Shape of Work Object] A specific example of processing by work object shape acquisition section 302B will be described with reference to FIGS.

[0328] <First Example> Fig. 19 is a flowchart that shows a first example of a process for acquiring data representing the shape of a work target. Fig. 20 is a diagram showing an example of an observation target area. Fig. 21 is a diagram showing an example of an influence area.

[0329] In this example, the explanation will be given on the premise that the first example (FIG. 6) or the second example (FIG. 7) of the functional configuration of the operation support system SYS described above is used, and that the predetermined operation of the shovel 100 is an excavation operation.

[0330] <Process for Acquiring Shape of Work Object> The flowchart in Fig. 19 is performed, for example, after the execution of an excavation operation by the shovel 100. Specifically, the flowchart in Fig. 19 corresponds to the processing of step S15 in Fig. 15 or step S35 in Fig. 17 described above. Hereinafter, this flowchart will be described on the premise that data representing the shape of the work object (shape of earth and sand) after the execution of the kth (k: positive integer) excavation operation by the shovel 100 since the start of excavation work is acquired.

[0331] For example, as shown in Fig. 20, an observation target area TA around the shovel 100 is divided into a predetermined number N of grids. The observation target area TA is an area around the shovel 100 from which the work target shape acquisition unit 302B acquires data representing the shape of the soil. In this example, the data representing the shape of the soil is the height h of the soil for each grid i (i = 1 to N) of the observation target area TA. ｅ ｋ , and the uncertainty s corresponding to the estimation accuracy ｅ ｋ The height of the soil and sand h is obtained. ｅ ｋ and uncertainty s ｅ ｋ are expressed by the following equations (1) and (2).

[0332]

[0333] In step S102, the work object shape acquisition unit 302B inputs data on the shape of the soil before the excavation operation of the shovel 100 is performed. The data on the shape of the soil before the excavation operation of the shovel 100 is input as data on the shape of the soil after the (k-1)th excavation operation is performed (height h ｅ ｋ―１ and uncertainty s ｅ ｋ―１ ) is equivalent to

[0334] In addition, in the case of the first excavation operation (k=1), the work object shape acquisition unit 302B sets the initial value of the data of the soil shape (the height h ｅ ０ and uncertainty s ｅ ０ )

[0335] The initial value of the soil shape data may be obtained based on the output of the sensor 40 (shape sensor), may be obtained from outside the excavator 100, or may be a predefined assumed value. The predefined assumed value is, for example, zero (0) when it is assumed that the entire observation target area TA (all grids i) is at the same height as the ground at the position of the excavator 100.

[0336] When the processing of step S102 is completed, the work object shape acquisition unit 302B proceeds to step S104.

[0337] In step S104, the work object shape acquisition unit 302B inputs a log of the current excavation operation of the excavator 100 for each grid i in the observation target area TA. The log includes the height b ｋ , the angle φ of the cutting edge (toe) of the bucket 6 ｋ , excavation reaction force f ｋ , sediment properties λ ｋ , and impact information κ ｋ The log also contains other information ω ｋ may include:

[0338] Height b of the cutting edge of the bucket 6 ｋ means the height passed by the cutting edge of the bucket 6 during the current excavation operation of the shovel 100 for each grid i in the observation target area TA, and is expressed by the following equation (3).

[0339]

[0340] Height b of the cutting edge of the bucket 6 ｋ For the grid i corresponding to the position where the bucket 6 has not passed, it is not necessary to input, or an appropriate value may be input. ｋ , and excavation reaction force f ｋ The same applies to the height b of the cutting edge of the bucket 6. ｋ is acquired, for example, based on data on the trajectory of the bucket 6 during the current excavation operation of the shovel 100. The data on the trajectory of the bucket 6 is acquired, for example, based on time-series data of the outputs of the sensors S1 to S6 during the current excavation operation of the shovel 100.

[0341] Angle φ of the cutting edge of the bucket 6 ｋ means the angle of the cutting edge of the bucket 6 relative to a predetermined reference plane (e.g., a horizontal plane) when the cutting edge of the bucket 6 passes through during the current excavation operation of the shovel 100, for each grid i in the observation area TA, and is expressed by the following equation (4).

[0342]

[0343] Angle φ of the cutting edge of the bucket 6 ｋis acquired, for example, based on data on the trajectory of the bucket 6 during the current excavation operation of the shovel 100 and time-series data on the attitude angle of the bucket 6. The time-series data on the attitude angle of the bucket 6 is acquired, for example, based on time-series data on the output of the sensor S3 during the current excavation operation of the shovel 100.

[0344] Excavation reaction force f ｋ means the reaction force acting on the bucket 6 from the ground during the current excavation operation of the shovel 100 for each grid i in the observation target area TA, and is expressed by the following equation (5).

[0345]

[0346] Excavation reaction force f ｋ is acquired, for example, based on the time series data of the outputs of sensors S7 to S9 during the current excavation operation of the shovel 100.

[0347] The excavation reaction force for each grid i may be expressed as a vector. In this case, the sediment characteristic λ for each grid i in the observation area TA is ｉ ｋ is three-dimensional rather than one-dimensional.

[0348] Sediment characteristics λ ｋ means the characteristics of the sediment for each grid i in the observation area TA, and is expressed by the following equation (6).

[0349]

[0350] Sediment characteristics λ ｋ is, for example, the angle of repose for each grid i in the observation area TA. ｋ is, for example, predetermined. ｋ may be obtained based on the time series data of the output of the sensor 40 and the outputs of the sensors S7 to S9.

[0351] In this case, the sediment characteristics λ for each grid point i in the observation area TA may be different. ｉ ｋ is not one-dimensional but multi-dimensional.

[0352] Impact information κ ｋrepresents whether or not the current excavation operation of the excavator 100 has a direct impact on the soil shape for each grid i in the observation target area TA. For example, the impact information κ ｋ means whether or not the cutting edge of the bucket 6 passes through for each grid i in the observation target area TA, and is expressed by the following equation (7).

[0353]

[0354] For example, during the current excavation operation, the blade tip of the bucket 6 passes through the grid i (i=1, . . . , N) of the observation target area TA, and the height of the blade tip of the bucket 6 at that time is equal to the height h of the soil and sand. ｅ，ｉ ｋ―１ If the following is true, impact information κ ｉ ｋ is set to "+1" (κ ｉ ｋ On the other hand, during this excavation operation, the cutting edge of the bucket 6 did not pass through the grid i (i = 1, ..., N) of the observation area TA, or passed through it but the height at that time was the height h of the earth and sand. ｅ，ｉ ｋ―１ If higher, impact information κ ｉ ｋ is set to "-1" (κ ｉ ｋ =-1).

[0355] Other Information ω ｋ is, for example, the weight w of the earth and sand contained in the bucket 6 after the excavation operation of the shovel 100 is performed. ｋ and volume v ｋ and is expressed by the following equation (8).

[0356]

[0357] When the processing of step S104 is completed, the work object shape acquisition unit 302B proceeds to step S106.

[0358] In step S106, the work object shape acquisition unit 302B calculates the uncertainty s ｅ，ｉ ｋ－１ Update the following.

[0359] For example, the work object shape acquisition unit 302B acquires the shape of the earth and sand (height h of the earth and sand) by the current excavation operation of the shovel 100.ｅ，ｉ ｋ Then, the work object shape acquisition unit 302B calculates the uncertainty s of the earth and sand shape for the grid i included in the influence area Ω. ｅ，ｉ ｋ has increased, and is updated using the following equation (9).

[0360]

[0361] For example, as shown in Figure 21, the influence area Ω is set as a range obtained by expanding the excavation area EA by a predetermined amount in all directions. The excavation area EA is the area within the observation area TA through which the cutting edge of the bucket 6 passes during the current excavation operation of the excavator 100 and the height of the cutting edge of the bucket 6 at that time is greater than the height h of the soil and sand. ｅ，ｉ ｋ―１ This is the region corresponding to the set of lattice i where:

[0362] Uncertainty ｅ，ｉ ｋ The increase in c ｉ For example, the uncertainty s ｅ，ｉ ｋ The increase in c ｉ may be set so that, within the influence area Ω, the excavation area EA is the largest and decreases with increasing distance from the excavation area EA.

[0363] When the processing of step S106 is completed, the work object shape acquisition unit 302B proceeds to step S108.

[0364] In step S108, the work object shape acquisition unit 302B identifies the shape of the soil before the current excavation operation by the shovel 100 is performed, based on the output of the sensor 40. As a result, the work object shape acquisition unit 302B can identify the shape of the soil after the current excavation operation by the shovel 100 is performed, for the grid i within the observation target area TA where the sensor 40 was able to measure the height of the soil.

[0365] For example, the work object shape acquisition unit 302B applies a Kalman filter and corrects the shape of the soil and sand after the shovel 100 has performed the previous excavation operation based on the output of the sensor 40, thereby identifying the shape of the soil and sand after the shovel 100 has performed the current excavation operation.

[0366] When the Kalman filter is applied, the variance s ｉ The height of the soil is z ｉ When observed, the height of the sediment h ｅ，ｉ ｋ and its uncertainty s ｅ，ｉ ｋ are expressed by the following equations (10) to (14).

[0367]

[0368] Equations (10) to (14) can be summarized as a function K, which is expressed by the following equation.

[0369]

[0370] Therefore, for the grid i in the observation area TA, the sensor 40 calculates the dispersion s ｌ，ｉ The height of the soil is z ｌ，ｉ When the height of the soil and sand h based on the output of the sensor 40 is measured, ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ is expressed by the following equation (16).

[0371]

[0372] As a result, the work object shape acquisition unit 302B acquires the height h of the earth and sand based on the output of the sensor 40 after the current excavation operation of the shovel 100 is performed. ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ can be obtained (identified).

[0373] The height h of the soil and sand based on the output of the sensor 40 for each grid i in the observation area TA ｌ ｋ is expressed by the following equation (17).

[0374]

[0375] The sensor 40 may not be able to measure the height of all the grids i in the observation area TA due to occlusion, etc. ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ is obtained by equation (16) only for the grid i where the height of the soil was measured by the sensor 40. The height of the soil h for the grid i where the height of the soil was not measured by the sensor 40 is ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ For example, the previous value (height of the soil and sand h ｌ，ｉ ｋ―１ and uncertainty s ｌ，ｉ ｋ―１ ) is maintained. In addition, the height of the soil and sand for the grid i where the height of the soil and sand could not be measured by the sensor 40 is maintained at h ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ may be set to a value indicating that it is unknown ("unknown"). Hereinafter, the measurement feasibility information m indicating whether the sensor 40 was able to measure the soil shape for each grid point i in the observation target area TA will be referred to as ｌ ｋ For example, the measurement availability information m ｌ ｋ are expressed by the following equations (18) to (20).

[0376]

[0377] When the processing of step S108 is completed, the work object shape acquisition unit 302B proceeds to step S110.

[0378] In step S110, the work object shape acquisition unit 302B uses a function g corresponding to the learned model LM1 to obtain the soil shape (soil height z ｐ ｋ and its uncertainty s ｐ ｋ ) is inferred.

[0379]

[0380] The function g is configured, for example, mainly using a deep neural network (DNN). It is also possible to use U-Net with input and output in the form of images. In this case, other information ω in the above equation (8) ｋ Since it does not take the form of an image, it may be expanded into the form of an image, or may be directly input to the intermediate layer.

[0381] Then, as in step S108, the work object shape acquisition unit 302B applies a Kalman filter and estimates the shape of the soil after the current excavation operation of the shovel 100 is performed based on the inference result of the function g. For the grid i in the observation target area, the height h of the soil based on the inference result of the function g is calculated as ｐ，ｉ ｋ and uncertainty s ｐ，ｉ ｋ is expressed by the following equation (22).

[0382]

[0383] When the processing of step S110 is completed, the work object shape acquisition unit 302B proceeds to step S112.

[0384] In step S112, the work object shape acquisition unit 302B acquires data representing the final shape of the earth and sand after the excavation operation of the shovel 100 is performed (height h of the earth and sand) based on the outputs of the processes of both steps S108 and S110. ｅ ｋ and uncertainty s ｅ ｋ That is, the work object shape acquisition unit 302B integrates the data on the shape of the soil and sand based on the output of the sensor 40, which is output in the process of step S108, and the data on the shape of the soil and sand based on the inference result of the function g, which is output in the process of step S110.

[0385] For example, the work target shape acquisition unit 302B acquires data on the shape of the soil (height h of the soil) identified based on the output of the sensor 40 in the process of step S108. ｌ，ｉ ｋ and uncertainty s ｌ，ｉ ｋ ) and the final data of the soil shape (soil height h ｅ，ｉ ｋand uncertainty s ｅ，ｉ ｋ )

[0386] However, as described above, there are lattices i in which the shape of the soil or sand is not identified based on the output of the sensor 40 in the process of step S108 due to occlusion of the sensor 40, that is, the measurement possibility information m ｌ ｋ Therefore, for this lattice i, the work object shape acquisition unit 302B uses the data of the earth and sand shape (height h of the earth and sand) specified based on the result of the inference of the function g in the process of step S110. ｐ，ｉ ｋ and uncertainty s ｐ，ｉ ｋ ) is the final data of the soil shape.

[0387] That is, the work object shape acquisition unit 302B generates the final data of the earth and sand shape using the following equations (23) and (24).

[0388]

[0389] Furthermore, for grid i where the sensor 40 has been able to measure the soil shape after the shovel 100 has performed its current excavation operation, the data on the soil shape identified based on the results of inference using function g in the processing of step S110 may be used as the final soil shape data.

[0390] When the processing of step S110 is completed, the work object shape acquisition unit 302B proceeds to step S112.

[0391] In step S112, the work object shape acquisition unit 302B acquires the data of the earth and sand shape (height h ｅ ｋ and uncertainty s ｅ ｋ ) is output.

[0392] This allows, for example, the target trajectory generation unit 302C to generate a target trajectory for the working part of the shovel 100, i.e., the bucket 6 (cutting edge), based on the data on the soil shape output from the work object shape acquisition unit 302B.

[0393] Furthermore, for example, the display processing unit 302E can generate an image showing the shape of the soil in the observation target area TA around the shovel 100 based on the data on the soil shape output from the work target shape acquisition unit 302B. Therefore, the display processing unit 302E can display the image showing the shape of the soil in the observation target area TA around the shovel 100 on the display device 50A, or transmit it to the remote operation support device 400 or the remote monitoring support device via the communication device 60 and display it on these display devices. Furthermore, the display processing unit 302E can add uncertainty s to the image showing the shape of the soil in the observation target area TA around the shovel 100. ｅ ｋ For example, the display processing unit 302E may reflect the uncertainty s ｅ ｋ is expressed by the color of the portion corresponding to the grid i of the image representing the shape of the soil and sand. In this way, the display processing unit 302E can display an image representing the shape of the soil and sand in the observation target area TA around the shovel 100 so that the difference in uncertainty for each grid i within the observation target area TA around the shovel 100 can be recognized.

[0394] The work object shape acquisition unit 302B may output other data obtained in the process of the process of this flowchart in addition to the data on the earth and sand shape. For example, the work object shape acquisition unit 302B may output the measurement feasibility information m ｌ ｋ This allows the controller 30 to distinguish between grid points i in the observation area TA where the sensor 40 was able to measure the sediment shape and grid points i in which the sensor 40 was unable to measure the sediment shape. Therefore, for example, when displaying an image representing the sediment shape in the observation area TA on the display device 50A or the like, the display processing unit 302E can distinguish between grid points i in which the output of the sensor 40 is reflected and grid points in which the output of the sensor 40 is not reflected.

[0395] When the processing of step S114 is completed, the work object shape acquisition unit 302B ends the processing of this flowchart.

[0396] In this way, in this example, the work object shape acquisition unit 302B can use the function g to estimate the shape of the soil after the excavation operation from the shape of the soil before the excavation operation is performed, taking into account the trajectory (track) of the bucket 6 during the excavation operation of the shovel 100, the excavation reaction force, the characteristics of the soil, etc.

[0397] In this example, even when the predetermined operation is an earth-discharging operation, the work object shape acquisition unit 302B can generate and output data on the earth and sand shape after the earth-discharging operation of the shovel 100 by the same processing as described above. In this case, for example, the other information ωk is expressed by the following equation (25).

[0398]

[0399] As shown in equation (25), other information ω ｋ is the weight W of the earth and sand contained in the bucket 6 before the excavator 100 performs the earth discharging operation. ｋ－１ and volume V ｋ－１ and the weight W of the soil contained in the bucket 6 after the soil discharge operation is performed. ｋ and volume V ｋ In addition, as shown in equation (25), when the discharge destination is the bed of a truck, other information ω ｋ may include the positions ρ of the four sides of the truck bed in a plan view. This allows the work object shape acquisition unit 302B to estimate the shape of the soil in the bed taking into account the shape of the truck bed.

[0400] In addition, for work performed by a combination of excavation and soil discharge operations of the shovel 100, the work object shape acquisition unit 302B can generate and output data on the soil shape after the excavation or soil discharge operation of the shovel 100 by processing similar to that described above.

[0401] <<Specific Example of a Method for Generating a Trained Model>> A specific example of a method for generating the trained model LM1 will be described.

[0402] In this example, a method for generating a function g corresponding to the trained model LM1 used in the process of FIG. 16 will be described.

[0403] The training data cj (j=1 to L) of the training data set D is expressed, for example, by the following equation (26).

[0404]

[0405] Teacher data c ｊ is the input data h^ ｅ ｊ , s^ ｅ ｊ , h ｌ ｊ , s ｌ ｊ , m ｌ ｊ , b ｊ , φ ｊ , f ｊ , λ ｊ , κ ｊ , ω ｊ and the height h of the earth and sand after the excavation operation of the shovel 100 as the correct data. ｒ ｊ The input data is a combination of ｅ ｊ , s^ ｅ ｊ , h ｌ ｊ , s ｌ ｊ , m ｌ ｊ , b ｊ , φ ｊ , f ｊ , λ ｊ , κ ｊ , ω ｊ is the input data (h ｅ ｋ－１ , s ｅ ｋ－１ , h ｌ ｋ , s ｌ ｋ , m ｌ ｋ , b ｋ , φ ｋ , f ｋ , λ ｋ , κ ｋ , ω ｋ ) corresponds to

[0406] As shown in the following equation (27), the function g has a parameter W, and machine learning is performed in such a manner that the parameter W is optimized by the training data set D.

[0407]

[0408] For example, the parameter W is optimized so that the loss function E(W) in the following equation (28) is minimized, thereby generating a function g corresponding to the trained model LM1.

[0409]

[0410] As described above, the teacher dataset D may be generated from the log acquired by the log acquisition unit 2001, or may be generated from the log acquired by the simulator unit 2002, or may be generated from both logs.

[0411] In the simulator unit 2002, for example, as described above, particle simulation such as DEM is adopted, and the height h of the soil and sand is calculated by ray tracing of a shape sensor such as LIDAR that is virtually arranged with respect to the position of the particle. ｒ ｊ is obtained.

[0412] Furthermore, as described above, the teacher dataset D may include a base teacher dataset generated from the log acquired by the simulator unit 2002, and a teacher dataset for fine tuning generated from the log acquired by the log acquisition unit 2001. In this case, the number of teacher data included in the teacher dataset for fine tuning may be relatively small.

[0413] Input data h ｌ ｊ , S ｌ ｊ , m ｌ ｊIn connection with the acquisition of the function g, a portion of the measurement data of the sensor group 300 or the measurement data of a virtual shape sensor arranged in the virtual space of the simulator unit 2002 may be virtually masked. As a result, even if no occlusion occurs in the sensor group 300 or the virtual shape sensor arranged in the virtual space of the simulator unit 2002, measurement data in which occlusion has virtually occurred can be acquired. Furthermore, the occlusion area within the observation target area TA may be specifically calculated by ray tracing the shape sensor arranged in the virtual space of the simulator unit 2002. Furthermore, the occlusion area may be changed by changing the position of the shape sensor arranged in the virtual space of the simulator unit 2002. This makes it possible to realize robust machine learning for the function g.

[0414] Also, the input data h^ ｅ ｊ , s^ ｅ ｊ corresponds to the previous output (inference result) of function g and may be obtained using multiple excavation operations repeated in time series. Furthermore, in order to suppress an increase in the number of data points, it may be substituted by learning while virtually mixing in large noise. This allows for more robust machine learning of function g.

[0415] In this way, the information processing device 200 ｊ A training data set D including the above can be generated, and a function g corresponding to the trained model LM1 can be generated by machine learning based on the training data set D.

[0416] Second Example FIG. 22 is a flowchart outlining a second example of processing for acquiring data representing the shape of a work object.

[0417] In this example, the explanation will be based on the third example (Figure 8) of the functional configuration of the operation support system SYS described above, or a modified example of the second example (Figure 7) of the functional configuration of the operation support system SYS described above, and will proceed on the assumption that the specified operation of the shovel 100 is an excavation operation.

[0418] The flowchart in Fig. 22 is performed, for example, after the excavation operation of the shovel 100. The flowchart in Fig. 22 corresponds to the processing of step S15 in Fig. 15 and the processing of step S35 in Fig. 17 described above. Hereinafter, the explanation of this flowchart will be given on the premise that data representing the shape of the work target (shape of earth and sand) after the kth excavation operation of the shovel 100 since the start of excavation work is acquired.

[0419] In this example, as in the first example described above, the explanation will proceed on the assumption that the observation target area TA around the excavator 100 is divided into a predetermined number N of grids (see FIG. 20). Also, in this example, data representing the shape of the work target after the kth excavation operation (earth and sand shape data) is calculated based on the height h of the earth and sand. ｅ ｋ The explanation will proceed on the assumption that

[0420] As shown in Fig. 22, in step S202, the controller 30 acquires data representing the trajectory x of the working part during each excavation operation of the shovel 100. For example, when k = 1, the controller 30 acquires the trajectory x of the working part during the first (first) excavation operation of the shovel 100. １ Furthermore, for example, when k≧2, the controller 30 acquires the trajectory x of the working part during each of the excavation operations of the shovel 100 from the first to the kth time. １ ~x ｋ Get the data.

[0421] For example, the controller 30 calculates the trajectory x during the m-th (1≦m≦k) excavation operation of the shovel 100. ｍ The controller 30 acquires data representing the target trajectory x of the working part during the m-th (1≦m≦k) excavation operation of the shovel 100, which is generated by the target trajectory generating unit 302C. ｍ As data representing the above, measurement data of the trajectory (track) of the work part may be acquired based on the outputs of the sensors S1 to S5.

[0422] Note that, from the start of the excavation work of the shovel 100 until the completion of the excavation work, the result of the processing of step S202 is stored, for example, in the auxiliary storage device 30A or the memory device 30B of the controller 30. As a result, in step S202 after the execution of the kth excavation operation, the controller 30 can easily and quickly calculate the trajectory x １ ~x ｋ The data can be obtained.

[0423] When the process of step S202 is completed, the controller 30 proceeds to step S204.

[0424] In step S204, the work object shape acquisition unit 302B of the controller 30 acquires the measurement results of the reaction force (measured reaction force) F on the work part during each excavation operation of the shovel 100 based on the outputs of the sensors S7 to S9. ｏ For example, when k=1, the work object shape acquisition unit 302B acquires time-series data of the measured reaction force F ｏ １ Furthermore, for example, when k≧2, the work object shape acquisition unit 302B acquires the time-series measured reaction force F ｏ １ ~F ｏ ｋ Get the data.

[0425] Note that, from the start of the excavation work of the shovel 100 until the completion of the excavation work, the result of the processing of step S204 is stored, for example, in the auxiliary storage device 30A or memory device 30B of the controller 30. As a result, in step S204 after the execution of the kth excavation work, the work object shape acquisition unit 302B can easily and quickly calculate the measured reaction force F ｏ １ ~F ｏ ｋ The data can be obtained.

[0426] When the process of step S204 is completed, the controller 30 proceeds to step S206.

[0427] In step S206, the reaction force estimation unit 302F of the controller 30 calculates the reaction force estimation result (estimated reaction force) F of the reaction force acting on the work part during each excavation operation of the shovel 100. ｅ For example, when k=1, the work object shape acquisition unit 302B acquires time-series data of the measured reaction force F ｅ １ Furthermore, for example, when k≧2, the work object shape acquisition unit 302B acquires the time-series measured reaction force F ｅ １ ~F ｅ ｋ Get the data.

[0428] For example, the reaction force estimation unit 302F uses a function g corresponding to the following learned model LM3 to calculate the m-th estimated reaction force F ｅ ｍ Obtain time series data.

[0429]

[0430] Furthermore, the reaction force estimation unit 302F introduces a variable γ that represents the characteristics of the soil and sand being worked on, and calculates the m-th estimated reaction force F ｅ ｍ Time series data may be acquired.

[0431]

[0432] As a result, the estimated reaction force F ｅ ｍ The accuracy of the calculation can be improved.

[0433] When the process of step S206 is completed, the controller 30 proceeds to step S208.

[0434] In step S208, the work object shape acquisition unit 302B calculates the measured reaction force F ｏ ｍ and estimated reaction force F ｅ ｍThe difference (estimated error) between these values ​​is calculated, and the data of the initial shape of the work object is corrected based on the estimated error. The initial shape is set in the process of step S11 in FIG. 15 or step S31 in FIG. 17, and the data is the height h of the earth and sand. ｅ ０ Specifically, the work object shape acquisition unit 302B determines the initial shape of the work object (height h of the earth and sand) so as to minimize the estimation error. ｅ ０ ) is corrected.

[0435] For example, when k=1, that is, after the first excavation operation is performed, the work object shape acquisition unit 302B calculates the corrected initial shape (height of earth and sand h) by the following equation (31) based on the above equation (29). ～ ｅ ０ ) data.

[0436]

[0437] Furthermore, for example, when k≧2, that is, when the excavation operation is executed for the second time or later, the work object shape acquisition unit 302B calculates the corrected initial shape (height of earth and sand h ) by the following equation (32) based on the above equation (29). ～ ｅ ０ ) data.

[0438]

[0439] For example, the work object shape acquisition unit 302B uses a function r corresponding to the learned model LM1 to obtain the shape of the work object after the mth excavation operation (height of earth and sand h ｅ ｍ ) data is calculated.

[0440]

[0441] Similarly to the function g, the work object shape acquisition unit 302B introduces a variable γ that represents the characteristics of the soil and sand to be worked on, and calculates the shape of the work object (height h of the soil and sand) after the mth excavation operation is performed using the following equation (34): ｅ ｍ ) data may be calculated.

[0442]

[0443] This allows the shape of the work object (height of the soil and sand h ｅ ｍ ) can improve the accuracy.

[0444] When the process of step S208 is completed, the controller 30 proceeds to step S210.

[0445] In step S210, the accuracy estimation unit 302G of the controller 30 estimates the accuracy of the initial shape of the work object corrected in the processing of step S208.

[0446] For example, the accuracy estimation unit 302G uses a function w corresponding to the learned model LM4 to calculate the accuracy c of the initial shape of the work object corrected after the kth excavation operation is performed, using the following equation (35): ｋ Estimate.

[0447]

[0448] Furthermore, orbit x １：ｋ represents the trajectory of the working part in the first to kth excavation operations, and the measured reaction force F ｏ １：ｋ represents the measurement results of the reaction force on the working part in the first to kth excavation operations.

[0449] In addition, the work object shape acquisition unit 302B introduces a variable γ that represents the characteristics of the soil and sand of the work object, as in the case of the functions g and r, and calculates the accuracy c of the initial shape of the work object corrected after the kth excavation operation is performed using the following equation (36): ｋ may be estimated.

[0450]

[0451] As a result, the accuracy c of the initial shape of the work object corrected after the execution of the kth excavation operation is ｋ This can improve the estimation accuracy.

[0452] When the process of step S210 is completed, the process proceeds to step S212.

[0453] In step S212, the work object shape acquisition unit 302B estimates the latest shape of the work object after the kth excavation operation has been performed, and acquires data representing that shape (earth and sand shape data).

[0454] Specifically, the work object shape acquisition unit 302B acquires the initial shape of the work object (height h of the earth and sand) corrected in step S208. ～ ｅ ０ ) as a starting point, the latest shape of the work object is estimated based on the trajectories x0 to xk from the first to kth trajectories.

[0455] For example, when k = 1, the work object shape acquisition unit 302B uses a function r corresponding to the learned model LM1 to obtain the shape of the work object after the first excavation operation (height h of the earth and sand) using the following equation (37): ｅ １ ) to obtain data representing the

[0456]

[0457] Furthermore, when k≧2, the work object shape acquisition unit 302B uses a function r corresponding to the learned model LM1 to obtain the shape of the work object after the kth excavation operation (height of earth and sand h ｅ ｋ ) to obtain data representing the

[0458]

[0459] When the process of step S212 is completed, the controller 30 ends the process of this flowchart.

[0460] In this way, in this example, the controller 30 corrects the initial shape of the work object based on the difference (estimation error) between the estimated reaction force and the measured reaction force, and can estimate the latest shape of the work object after the excavation operation of the shovel 100 is performed, starting from the corrected initial shape of the work object.

[0461] <Third Example> FIG. 23 is a flowchart outlining a third example of processing for acquiring data representing the shape of a work object.

[0462] In this example, the explanation will be based on the first example (Figure 6), the second example (Figure 7), the third example (Figure 8) or a modified example of the second example of the functional configuration of the operation support system SYS described above, and will proceed on the assumption that the specified operation of the shovel 100 is an earth-discharging operation.

[0463] The flowchart in Fig. 23 is executed, for example, after the execution of the earth unloading operation of the shovel 100. Specifically, the flowchart in Fig. 23 corresponds to the processing of step S24 in Fig. 16 and the processing of step S45 in Fig. 18 described above. Hereinafter, the explanation of this flowchart will be given on the premise that data representing the shape of the work target (earth and sand shape) after the execution of the kth earth unloading operation of the shovel 100 since the start of loading work is acquired.

[0464] In this example, as in the first and second examples described above, the explanation will be given on the assumption that the observation target area TA around the excavator 100, i.e., the truck bed, is divided into a predetermined number N of grids (see FIG. 20). Also, in this example, the data representing the shape of the work target after the kth excavation operation (earth and sand shape data) is calculated based on the height b ｅ ｋ The explanation will proceed on the assumption that

[0465] As shown in FIG. 23, in step S302, the controller 30 calculates the weight w of the earth and sand contained in the bucket 6 before the kth earth discharging operation is performed based on the outputs of the sensors S7 to S9. ｋ Estimate.

[0466] When the process of step S302 is completed, the controller 30 proceeds to step S304.

[0467] In step S304, the controller 30 calculates the trajectory x of the working part when the excavator 100 performs the kth earth-discharging operation. ｋ Obtain data representing the

[0468] For example, the controller 30 calculates the trajectory x of the working part when the excavator 100 performs the kth earth removal operation. ｋThe controller 30 acquires data representing the target trajectory x of the working part when the kth earth unloading operation of the shovel 100 is performed, which is generated by the target trajectory generating unit 302C. ｋ As data representing the kth earth unloading operation, measurement data of the trajectory (track) of the working part during the kth earth unloading operation of the shovel 100 may be obtained based on the outputs of the sensors S1 to S5.

[0469] When the process of step S304 is completed, the controller 30 proceeds to step S306.

[0470] In step S306, the work object shape acquisition unit 302B of the controller 30 estimates the latest shape of the work object after the kth earth-discharging operation of the shovel 100, i.e., the shape of the soil and sand in the truck bed, and acquires data representing that shape.

[0471] For example, the work object shape acquisition unit 302B uses a function a corresponding to the learned model LM1 to obtain the shape of the soil in the truck bed after the kth earth-discharging operation of the excavator 100 (height b ｅ ｋ ) to obtain data representing the

[0472]

[0473] Furthermore, the work object shape acquisition unit 302B introduces a variable γ that represents the characteristics of the soil and sand, and calculates the shape of the soil and sand in the truck bed after the kth earth-discharging operation of the excavator 100 has been performed (height b ｅ ｋ ) may be obtained.

[0474]

[0475] This makes it possible to improve the accuracy of estimating the shape of the soil and sand in the truck bed after the excavator 100 has performed the kth earth-discharging operation.

[0476] When the process of step S306 is completed, the controller 30 ends the process of this flowchart.

[0477] In this way, in this example, the controller 30 can estimate the latest shape of the work object after the shovel 100 performs the earth-discharging operation based on the shape of the work object before the shovel 100 performs the earth-discharging operation and the trajectory of the work part when the shovel 100 performs the earth-discharging operation.

[0478] [Operation] Next, the operation of the work machine, information processing device, and program according to this embodiment will be described.

[0479] In a first aspect of this embodiment, a work machine is provided with a processing device. The work machine is, for example, the above-mentioned shovel 100. The processing device is, for example, the above-mentioned controller 30. Specifically, the processing device acquires data representing the state of the work machine, which data is related to the shape of the work object, in accordance with the operation of the work machine, and estimates the shape of the work object after the operation of the work machine based on the acquired data.

[0480] In addition, in the first aspect of this embodiment, the information processing device may acquire data representing the state of the work machine and relating to the shape of the work object in accordance with the operation of the work machine, and may estimate the shape of the work object of the work machine after the operation of the work machine based on the acquired data. The information processing device is, for example, the above-mentioned controller 30, information processing device 200, or remote operation support device 400.

[0481] In addition, in a first aspect of this embodiment, the program may cause the information processing device to execute a first procedure and a second procedure. Specifically, in the first procedure, data representing the state of the work machine, which is data related to the shape of the work object, may be acquired in accordance with the operation of the work machine. Then, in the second procedure, the shape of the work object of the work machine may be estimated based on the data acquired in the first procedure.

[0482] This allows the work machine or information processing device (hereinafter referred to as "work machine, etc.") to estimate the state of the work object by taking into account changes in the shape of the work object in response to the operation of the work machine, for example. As a result, the work machine, etc. can more appropriately grasp the shape of the work object.

[0483] Furthermore, in a second aspect of this embodiment, based on the first aspect described above, the work machine may be equipped with a first acquisition device. The first acquisition device is, for example, the sensors S1 to S3 described above. Specifically, the first acquisition device may acquire data related to the trajectory of the working part of the work machine. Then, a processing device or information processing device (hereinafter referred to as "processing device, etc.") may estimate the shape of the work object based on the data related to the trajectory of the working part when a predetermined operation is performed.

[0484] This allows the work machine or the like to estimate the state of the work object by taking into account changes in the shape of the work object from the trajectory of the work part and the positional relationship between the work part and the work object.

[0485] Furthermore, in a third aspect of this embodiment, based on the second aspect described above, the work machine may be equipped with a second acquisition device. The second acquisition device may be, for example, the sensors S7 to S9 described above. Specifically, the second acquisition device may acquire data related to the reaction force from the work object to the work part, or data related to the weight of earth and sand held by the work part. The processing device or the like may then estimate the shape of the work object based on the data related to the reaction force to the work part when a predetermined operation is performed, or the data related to the weight of earth and sand held by the work part.

[0486] This allows the work machine etc. to, for example, recognize contact with underground rocks or the like from the reaction force on the working part when the work machine is performing a predetermined operation, and to recognize characteristics such as the hardness of the soil. Also, the work machine etc. can recognize the amount of soil to be discharged onto the work object from the weight of the soil held in the bucket, which is the working part. Therefore, the work machine etc. can take these recognition results into consideration and more appropriately estimate the shape of the work object.

[0487] Furthermore, in a fourth aspect of this embodiment, based on the second aspect described above, the work machine may be equipped with a measuring device. The measuring device is, for example, the sensor 40 described above. Specifically, the measuring device may measure the shape of a work object around the work machine. The processing device or the like may then estimate the shape of the work object based on measurement data of the shape of the work object and data related to the trajectory of the work part when a predetermined operation is performed.

[0488] This allows the work machine or the like to estimate the shape of the work object, for example, starting from the shape of the work object at a certain point in time and taking into account changes in the shape of the work object from the trajectory of the work part.

[0489] Furthermore, in a fifth aspect of this embodiment, based on the fourth aspect described above, the processing device etc. may estimate the shape of the work object after the execution of a specified operation based on measurement data of the shape of the work object before the execution of the specified operation, measurement data of the shape of the work object after the execution of the specified operation, and data relating to the trajectory of the work part when the specified operation is performed.

[0490] This allows a work machine, etc. to, for example, estimate the shape of a work object after the work machine has performed a specified operation based on measurement data from a measuring device, and to estimate the shape of the work object from the trajectory of the work area for areas that cannot be measured due to occlusion, etc.

[0491] Furthermore, in a sixth aspect of this embodiment, assuming the fourth or fifth aspect described above, the processing device etc. may acquire data representing the shape of the work object based on the estimated shape of the work object and measurement data of the shape of the work object acquired by the measuring device after the estimated shape is output.

[0492] This allows the processing device, etc. to more appropriately estimate data representing the shape of the work object by utilizing the results of machine learning based on the difference between the estimated result of the shape of the work object and the measurement result of the measuring device after the estimated result has been output.

[0493] Furthermore, in a seventh aspect of this embodiment, based on the third aspect described above, the processing device or the like may estimate the reaction force from the work object to the work part when a predetermined operation is performed, based on data representing the shape of the work object before the work machine performs a predetermined operation and data related to the trajectory of the work part when the work machine performs the predetermined operation. The processing device or the like may also correct the data representing the shape of the work object before the work machine performs a predetermined operation, based on the estimated result of the reaction force from the work object to the work part when the work machine performs the predetermined operation and data related to the actual reaction force from the work object to the work part when the work machine performs the predetermined operation. The processing device or the like may then estimate the shape of the work object after the predetermined operation is performed, based on the corrected data representing the shape of the work object before the work machine performs the predetermined operation and data related to the trajectory of the work part when the work machine performs the predetermined operation.

[0494] As a result, even if the accuracy of data representing the shape of the work object before the work machine performs a predetermined operation is relatively low, the processing device etc. can correct that data based on the estimation error in the reaction force estimation result when the work machine performs the predetermined operation, and therefore the processing device etc. can estimate the shape of the work object after the work machine performs the predetermined operation with relatively high accuracy.

[0495] Furthermore, in an eighth aspect of this embodiment, based on the seventh aspect described above, the processing device etc. may correct the data representing the initial shape of the work object based on the estimated results of the reaction force from the work object to the work part when a predetermined operation is performed by the work machine multiple times, and data relating to the actual reaction force from the work object to the work part when a predetermined operation is performed. The processing device etc. may then estimate the shape of the work object after multiple executions of the predetermined operation by the work machine based on the data representing the corrected initial shape of the work object and data relating to the trajectory of the work part when the predetermined operation is performed by the work machine multiple times.

[0496] As a result, even if the accuracy of data representing the initial shape of the work machine is relatively low, the processing device etc. can correct that data based on the estimation error in the reaction force estimation results when the work machine performs a predetermined operation, and therefore the processing device etc. can estimate with relatively high accuracy the shape of the work object after the work machine performs a predetermined operation.

[0497] Furthermore, in a ninth aspect of this embodiment, assuming the seventh or eighth aspect described above, the processing device etc. may limit at least one of the operating speed and operating range of a predetermined operation of the work machine based on data representing the uncertainty of the shape of the work object according to its position within the work object, with respect to data representing the shape of the work object before the work machine performs the predetermined operation.

[0498] This allows the processing device etc. to slow down the speed of a predetermined operation of the work machine or narrow the operating range, for example, when there is a relatively high degree of uncertainty in the shape of the data representing the shape of the work object before the predetermined operation of the work object is performed.As a result, even when the predetermined operation of the work machine is performed based on data representing the shape of the work object before the predetermined operation of the work object is performed with relatively low accuracy, it is possible to prevent a situation in which the reaction force from the work object to the working part of the work machine becomes excessive.

[0499] In addition, in a tenth aspect of this embodiment, assuming any one of the first to ninth aspects described above, the processing device or information processing device may estimate the shape of the work object based on the characteristics of the soil and sand that is the work object, or the soil and sand that is added to the work object in accordance with the operation of the work machine.

[0500] This allows the work machine or the like to take into account the characteristics of the soil and sand and more appropriately estimate the shape of the work object.

[0501] In addition, in an eleventh aspect of this embodiment, based on the above-mentioned tenth aspect, the characteristics of the soil may include at least one of the angle of repose of the soil, the moisture content of the soil, and the grain size of the soil.

[0502] This allows the work machine or the like to more appropriately estimate the shape of the work object by taking into account the angle of repose, moisture content, and grain size of the soil and sand.

[0503] In a twelfth aspect of the present embodiment, assuming any one of the first to eleventh aspects described above, there may be a plurality of predetermined actions of the work machine, and the processing device or information processing device may estimate the shape of the work object in accordance with a predetermined action executed by the work machine out of the plurality of predetermined actions.

[0504] This allows the work machine or the like to estimate the state of the work target in accordance with the predetermined operation being performed by the work machine.

[0505] In a thirteenth aspect of the present embodiment, based on any one of the second to ninth aspects described above, the working part of the work machine may be a bucket, and the predetermined operation of the work machine may be an excavation operation or an earth-discharging operation.

[0506] This allows the work machine or the like to more appropriately estimate the shape of the soil or sand that is the work target, taking into account changes in the shape of the work target that occur in response to the work machine's excavation and earth-discharging operations.

[0507] In a fourteenth aspect of the present embodiment, based on any one of the first to thirteenth aspects described above, the work machine may be equipped with a control device that controls the operation of the work machine based on the results of estimating the shape of the work object. Also, the information processing device may be equipped with a control unit that controls the operation of the work machine based on the results of estimating the shape of the work object.

[0508] This allows the operation of the work machine etc. to be controlled in accordance with the shape of the work object.

[0509] In a fifteenth aspect of the present embodiment, based on any one of the first to fourteenth aspects described above, the work machine may be provided with a display device that displays an image representing the shape of the work object based on the results of estimating the shape of the work object. The display device is, for example, the output device 50 described above.

[0510] Furthermore, in a fifteenth aspect of this embodiment, assuming any one of the first to fourteenth aspects described above, the program may cause the assistance device to execute a first to third procedure. Specifically, in the first procedure, data representing the state of the work machine, which is data relating to the shape of the work object, may be acquired in accordance with the operation of the work machine. Furthermore, in the second procedure, the shape of the work object of the work machine may be estimated based on the data acquired in the first procedure. Then, in the third procedure, an image representing the shape of the work object may be displayed on the display unit based on the results of the estimation of the shape of the work object. The assistance device is, for example, a remote operation assistance device 400.

[0511] This allows the work machine or the like to present the estimated results of the shape of the work object to the operator or the like.

[0512] In a sixteenth aspect of this embodiment, based on the fifteenth aspect described above, the processing device or the like may estimate the shape of the work object after the work machine operates in accordance with the operation of the work machine, and based on the results of the estimation of the shape of the work object, may acquire data representing the shape of the work object after the work machine operates, as well as data representing the uncertainty of the shape of the work object depending on the position of the acquired data within the work object. The display device may then display an image representing the shape of the work object so that differences in uncertainty depending on the position within the work object can be identified.

[0513] Furthermore, in a sixteenth aspect of this embodiment, based on the fifteenth aspect described above, the program may cause the assistance device to execute a second procedure of estimating the shape of a work object of the work machine after operation of the work machine in accordance with operation of the work machine, and acquiring data representing the shape of the work object after operation of the work machine based on the result of the estimation of the shape of the work object, and a fourth procedure of acquiring an indication of the uncertainty of the estimation result of the shape of the work object according to the position within the work object for the data acquired in the second procedure. The program may then cause the assistance device to execute a third procedure of displaying an image representing the estimation result of the shape of the work object on a display unit so that differences in uncertainty according to the position within the work object can be identified.

[0514] This allows the work machine, etc. to allow the user to visually recognize the estimated results of the shape of the work object, while also allowing the user to recognize the difference in uncertainty of the estimated results depending on the position within the observation range.

[0515] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

[0516] Finally, this application claims priority based on Japanese Patent Application No. 2023-200126, filed on November 27, 2023, the entire contents of which are incorporated herein by reference.

[0517] 1 Lower traveling body 3 Upper rotating body 4 Boom 5 Arm 6 Bucket 30 Controller 31 Hydraulic control valve 32 Shuttle valve 33 Hydraulic control valve 40 Sensor 40B Sensor 40F Sensor 40L Sensor 40R Sensor 50 Output device 50A Display device 52 Input device 60 Communication device 100 Excavator 150 Support device 200 Information processing device 300 Sensor group 300-1 to 300-M Sensor 301 Operation log providing unit 301A Operation log recording unit 301B Operation log storage unit 301C Operation log transmission unit 302 Work support unit 302A Learned model storage unit 302B Work object shape acquisition unit 302C Target trajectory generation unit 302D Operation control unit 302E Display processing unit 302F Reaction force estimation unit 302G Accuracy estimation unit 400 Remote operation support device 2001 Log acquisition unit 2002 Simulator unit 2003 Log storage unit 2004 Teacher data generation unit 2004A to 2004D Teacher data generation unit 2005 Machine learning unit 2005A to 2005D Machine learning unit 2006 Learned model storage unit 2007 Distribution unit AT Attachment LM1 to LM4 Learned model S1 to S9 Sensor SYS Operation support system

Claims

1. A work machine comprising a processing device that acquires data representing the state of the work machine in accordance with the operation of the work machine, the data relating to the shape of a work object, and estimates the shape of the work object after the work machine has started based on the acquired data.

2. A work machine as described in claim 1, further comprising a first acquisition device for acquiring data relating to the trajectory of a working part of the work machine, said processing device estimating the shape of the work object based on the data relating to the trajectory of the working part when a specified operation is performed.

3. A work machine as described in claim 2, further comprising a second acquisition device that acquires data relating to the reaction force from the work object to the working part, or data relating to the weight of soil held by the working part, and the processing device estimates the shape of the work object based on the data relating to the reaction force to the working part when a specified operation is performed, or the data relating to the weight of soil held by the working part.

4. A work machine as described in claim 2, further comprising a measuring device for measuring the shape of the work object in the vicinity of the work machine, and wherein the processing device estimates the shape of the work object based on the measurement data of the shape of the work object and data relating to the trajectory of the working part when the specified operation is performed.

5. A work machine as described in claim 4, wherein the processing device estimates the shape of the work object after the specified operation is performed based on measurement data of the shape of the work object before the specified operation is performed, measurement data of the shape of the work object after the specified operation is performed, and data relating to the trajectory of the working part when the specified operation is performed.

6. A work machine as described in claim 4 or 5, wherein the processing device acquires data representing the shape of the work object based on an estimation result of the shape of the work object and measurement data of the shape of the work object acquired by the measuring device after output of the estimation result.

7. The work machine described in claim 3, wherein the processing device: estimates a reaction force from the work object to the work part when the specified motion is performed based on data representing the shape of the work object before the performance of the specified motion and data relating to the trajectory of the work part when the specified motion is performed; corrects the data representing the shape of the work object before the performance of the specified motion based on the estimated reaction force from the work object to the work part when the specified motion is performed and data relating to the actual reaction force from the work object to the work part when the specified motion is performed; and estimates the shape of the work object after the specified motion is performed based on the corrected data representing the shape of the work object before the performance of the specified motion and data relating to the trajectory of the work part when the specified motion is performed.

8. The work machine described in claim 7, wherein the processing device corrects data representing the initial shape of the work object based on an estimated result of the reaction force from the work object to the working part when the specified operation is performed for each of the multiple specified operations executed starting from the initial shape of the work object, and data relating to an actual reaction force from the work object to the working part when the specified operation is performed, and estimates the shape of the work object after the multiple specified operations are performed based on the data representing the initial shape of the work object after the correction and data relating to the trajectory of the work part when the specified operation is performed for each of the multiple specified operations.

9. A work machine as claimed in claim 7 or 8, wherein the processing device limits at least one of the operating speed and operating range of the specified operation based on data representing the uncertainty of the shape of the work object according to the position within the work object with respect to data representing the shape of the work object before the specified operation is performed.

10. A work machine as claimed in any one of claims 1 to 5, 7 and 8, wherein the processing device estimates the shape of the work object based on characteristics of the soil in the work object or soil added to the work object in response to the operation of the work machine.

11. The work machine according to claim 10, wherein the characteristics include at least one of an angle of repose of the soil, a moisture content of the soil, and a grain size of the soil.

12. A work machine as claimed in any one of claims 2 to 5, 7 and 8, wherein there are a plurality of the predetermined actions, and the processing device estimates the shape of the work object according to the predetermined action executed by the work machine among the plurality of the predetermined actions.

13. A work machine according to any one of claims 2 to 5, 7 and 8, wherein the working part is a bucket, and the predetermined operation is an excavation operation or an earth removal operation.

14. A work machine as claimed in any one of claims 1 to 5, 7 and 8, comprising a control device that controls the operation of the work machine based on the results of estimating the shape of the work object.

15. A work machine as claimed in any one of claims 1 to 5, 7 and 8, comprising a display device which displays an image representing the shape of the work object based on the results of estimating the shape of the work object.

16. The work machine described in claim 15, wherein the processing device estimates the shape of the work object after the work machine operates in accordance with the operation of the work machine, and based on the estimation result of the shape of the work object, acquires data representing the shape of the work object after the work machine operates, and acquires data representing the uncertainty of the shape of the work object according to the position within the work object for the acquired data, and the display device displays an image representing the shape of the work object so that the difference in the uncertainty according to the position within the work object can be identified.

17. An information processing device that acquires data representing a state of a work machine in accordance with the operation of the work machine, the data relating to changes in the shape of an object of work performed by the work machine, and estimates the shape of the object of work after the operation of the work machine based on the acquired data.

18. A program that causes an information processing device to execute the steps of: acquiring data representing the state of a work machine in accordance with the operation of the work machine, the data relating to changes in the shape of a work object of the work machine; and estimating the shape of the work object after the work machine operates based on the acquired data.

19. A program that causes an assistance device to execute the following steps: acquiring data representing the state of the work machine in accordance with the operation of the work machine, the data relating to changes in the shape of a work object of the work machine; estimating the shape of the work object after the work machine operates based on the acquired data; and displaying the shape of the work object on a display unit based on the estimated shape of the work object.