Support devices, work machines, and programs

Abstract

To provide a technique capable of more properly operating a work machine.SOLUTION: A controller 30 according to one embodiment of the present disclosure comprises: a work object shape acquisition unit 302B which acquires data about the shape of a work object in the periphery of a shovel 100; an estimation unit 302C which estimates an operation conforming to the shape of the work object in the periphery of the shovel 100 from a plurality of candidate operations of the shovel 100 in the prescribed work on the basis of the data about the shape of the work object in the periphery of the shovel 100 by using a learned model LM that is subjected to machine learning with teacher data about the operation of the shovel 100 by the operation of an operator who has a relatively high level of skill associated with the shape of the work object; and a proposal unit 302D which proposes the one or the plurality of operations in the plurality of candidate operations to the user on the basis of the estimation result of the estimation unit 302C.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to an assistance device for a work machine. [Background technology]

[0002] BACKGROUND ART Work machines such as excavators are known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-029769 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing a certain task using a work machine, it is necessary to selectively use multiple actions depending on the state of the work target, such as the shape of the terrain. For example, when leveling ground with a shovel, actions such as sweeping soil and sand forward with the back of the bucket, horizontal pulling, and compaction are used. Therefore, for example, an inexperienced operator may find it difficult to select an appropriate action from multiple possible actions, which may result in a decrease in work efficiency, etc.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a technique that enables a work machine to operate more appropriately. [Means for solving the problem]

[0006] In order to achieve the above object, in one embodiment of the present disclosure, an acquisition unit that acquires data relating to the shape of a work target in the vicinity of the work machine; A predetermined work is performed based on data relating to the shape of the work object around the work machine. can be performed with The work machine A plurality of candidate actions of different types, each of which is predefined a suggestion unit that suggests an action from among a plurality of candidate actions to a user; Support devices are provided. In another embodiment of the present disclosure, an acquisition unit that acquires data relating to the shape of a work target in the vicinity of the work machine; a suggestion unit that suggests to a user an action from among a plurality of candidate actions of the work machine in a predetermined task based on data relating to the shape of a work object in the vicinity of the work machine; an estimation unit that estimates an action that matches the shape of the work object in the vicinity of the work machine from among a plurality of candidate actions based on data related to the shape of the work object in the vicinity of the work machine, using a trained model that has been machine-learned using teacher data related to the action of the work machine as operated by a relatively skilled operator, which is associated with the shape of the work object; the proposing unit proposes an action from among the plurality of candidate actions based on the estimation result of the estimating unit. Support devices are provided.

[0007] In addition, the present disclosure Furthermore In other embodiments, The above-mentioned support device is provided. A work machine is provided.

[0008] In still another embodiment of the present disclosure, Support devices include: an acquisition step of acquiring data relating to the shape of a work object around the work machine; A predetermined work is performed based on data relating to the shape of the work object around the work machine. can be performed with The work machine A plurality of candidate actions of different types, each of which is predefined a suggestion step of suggesting an action from among a plurality of candidate actions to the user; Programs are offered. In still another embodiment of the present disclosure, Support devices include: an acquisition step of acquiring data relating to the shape of a work object around the work machine; a proposing step of proposing to a user an action from among a plurality of candidate actions of the work machine for a predetermined task based on data relating to the shape of a work object around the work machine; an estimation step of estimating an action that matches the shape of the work object around the work machine from among the plurality of candidate actions based on data related to the shape of the work object around the work machine, using a trained model that has been machine-learned using training data related to the action of the work machine as operated by a relatively skilled operator, which is associated with the shape of the work object; In the proposing step, an action is proposed from among the plurality of candidate actions based on the estimation result in the estimating step. Programs are offered. [Effects of the Invention]

[0009] According to the above-described embodiment, the work machine can be operated more appropriately. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an excavator operation support system. [Figure 2] FIG. 1 is a top view showing an example of a shovel. [Figure 3] FIG. 2 is a diagram illustrating an example of a configuration for remotely controlling a shovel. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the shovel. [Figure 5] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a functional block diagram showing a first example of a functional configuration related to an operation suggestion function of the excavator operation support system. [Figure 7] 10 is a flowchart schematically illustrating a first example of processing related to a function of suggesting an operation of a shovel. [Figure 8] FIG. 10 is a functional block diagram showing a second example of a functional configuration related to an operation suggestion function of the excavator operation support system. [Figure 9] 10 is a flowchart schematically illustrating a second example of processing related to the operation suggestion function of a shovel. [Figure 10] FIG. 10 is a diagram showing a first example of the display content of the display device, relating to the operation suggestion function of the shovel. [Figure 11] FIG. 10 is a diagram showing a second example of the display content of the display device, relating to the shovel operation suggestion function. [Figure 12] FIG. 10 is a diagram showing a third example of the display content of the display device, relating to the shovel operation suggestion function. [Figure 13] FIG. 10 is a diagram showing a third example of the display content of the display device, relating to the shovel operation suggestion function. [Figure 14] FIG. 10 is a diagram showing a fourth example of the display content of the display device, relating to the shovel operation suggestion function. [Figure 15] FIG. 10 is a diagram showing a fifth example of the display content of the display device, relating to the shovel operation suggestion function. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment will be described with reference to the drawings.

[0012] [Outline of the operation support system] First, an overview of the operation support system SYS according to this embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0013] FIG. 1 is a diagram showing an example of an operation support system SYS. In FIG. 1, a left side view of a shovel 100 is shown. FIG. 2 is a top view showing an example of the shovel 100. FIG. 3 is a diagram showing an example of a configuration related to remote operation of a shovel. Hereinafter, the direction in which the attachment AT extends when viewed from above the shovel 100 (the upward direction in FIG. 2) will be defined as "front," and directions on the shovel 100 or directions seen from the shovel 100 may be described.

[0014] As shown in FIG. 1, the operation support system SYS includes an excavator 100 and an information processing device 200.

[0015] The operation support system SYS uses the information processing device 200 to cooperate with the shovel 100 and provide support regarding the operation of the shovel 100.

[0016] The operation support system SYS may include one or more excavators 100.

[0017] The excavator 100 is a work machine that receives support for operation in the operation support system SYS.

[0018] As shown in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

[0019] The lower traveling body 1 uses crawlers 1C to travel the excavator 100. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The crawlers 1CL are hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawlers 1CL are hydraulically driven by a traveling hydraulic motor 1MR. This allows the lower traveling body 1 to travel independently.

[0020] The upper rotating body 3 is rotatably mounted on the lower traveling body 1 via a rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by a rotating hydraulic motor 2M.

[0021] Boom 4 is attached to the front center of upper rotating body 3 so as to be able to tilt up and down about a rotation axis that runs along the left-right direction. Arm 5 is attached to the tip of boom 4 so as to be able to rotate about a rotation axis that runs along the left-right direction. Bucket 6 is attached to the tip of arm 5 so as to be able to rotate about a rotation axis that runs along the left-right direction.

[0022] The bucket 6 is an example of an end attachment, and is used, for example, for excavation work.

[0023] The bucket 6 is attached to the tip of the arm 5 in a manner that allows it to be appropriately replaced depending on the work content of the excavator 100. In other words, instead of the bucket 6, a bucket of a different type from the bucket 6, such as a relatively large bucket, a slope bucket, or a dredging bucket, may be attached to the tip of the arm 5. Also, a type of end attachment other than a bucket, such as a mixer, breaker, or crusher, may be attached to the tip of the arm 5. Also, a spare attachment such as a quick coupling or a tiltrotator may be provided between the arm 5 and the end attachment.

[0024] The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0025] The cabin 10 is a control room where an operator sits and operates the excavator 100. The cabin 10 is mounted on the front left side of the upper rotating body 3, for example.

[0026] For example, the excavator 100 operates driven elements such as the lower traveling body 1 (i.e., a pair of left and right crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6 in response to operations by an operator inside the cabin 10.

[0027] Furthermore, instead of or in addition to being configured to be operable by an operator inside the cabin 10, the shovel 100 may be configured to be remotely operable from outside the shovel 100. When the shovel 100 is remotely operated, the inside of the cabin 10 may be unmanned. The following description will be given on the assumption that the operation of the operator includes at least one of operation of the operating device 26 by the operator inside the cabin 10 and remote operation by an external operator.

[0028] For example, as shown in FIG. 3, remote operation includes a mode in which the shovel 100 is operated by operation input relating to the actuator of the shovel 100 performed by a remote operation support device 300.

[0029] The remote operation support device 300 is provided, for example, in a management center that externally manages the work of the shovel 100. The remote operation support device 300 may also be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the work status of the shovel 100 from the vicinity of the shovel 100.

[0030] The shovel 100 may transmit, for example, via a communication device 60 described below, an image (hereinafter referred to as a "peripheral image") showing the surroundings including the area in front of the shovel 100, based on an image captured by an imaging device 40 described below. The remote operation support device 300 may then display the image (peripheral image) received from the shovel 100 on a display device. Furthermore, various information images (information screens) displayed on an output device 50 (display device) inside the cabin 10 of the shovel 100 may also be displayed on the display device of the remote operation support device 300. This allows an operator using the remote operation support device 300 to remotely operate the shovel 100 while checking, for example, the contents of an image, information screen, or the like showing the surroundings of the shovel 100 displayed on the display device. The excavator 100 may operate actuators and drive driven elements such as the lower running body 1, upper rotating body 3, boom 4, arm 5, and bucket 6 in accordance with a remote control signal indicating the content of the remote control, which is received by the communication device 60 from the remote control support device 300.

[0031] Remote control may also include, for example, a mode in which the shovel 100 is operated by an external voice input or gesture input to the shovel 100 by a person (e.g., a worker) around the shovel 100. Specifically, the shovel 100 recognizes voices uttered by surrounding workers or gestures made by the workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., an imaging device) mounted on the shovel 100. Then, the shovel 100 may operate actuators in accordance with the content of the recognized voices or gestures to drive driven elements such as the lower traveling body 1 (left and right crawlers 1C), upper rotating body 3, boom 4, arm 5, and bucket 6.

[0032] Furthermore, the work of the shovel 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 300 may be provided. The remote monitoring support device is, for example, the information processing device 200. This allows a monitor, who is a user of the remote monitoring support device, to monitor the status of the work of the shovel 100 while checking a peripheral image displayed on a display device of the remote monitoring support device. Furthermore, for example, if the monitor determines it is necessary from a safety standpoint, the monitor can intervene in the operation by the operator of the shovel 100 and bring about an emergency stop by making a predetermined input using an input device of the remote monitoring support device.

[0033] The information processing device 200 communicates with the shovel 100 to cooperate with each other and provide support regarding the operation of the shovel 100.

[0034] The information processing device 200 is, for example, a server or a management terminal device installed in an administration office within the work site of the shovel 100, or in a management center that manages the operating status of the shovel 100 and is located in a place different from the work site of the shovel 100. The management terminal device may be, for example, a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (mobile terminal) such as a tablet terminal, a smartphone, or a laptop PC. In the latter case, workers at the work site, supervisors who supervise the work, managers who manage the work site, etc. can move around the work site carrying the portable information processing device 200. In addition, in the latter case, an operator can, for example, bring the portable information processing device 200 into the cabin of the shovel 100. Furthermore, a plurality of information processing devices 200 may be provided depending on the purpose, for example, for remote monitoring, for processing related to a function of proposing operations of the shovel 100 to the operator (described later), etc.

[0035] The information processing device 200 acquires, for example, data relating to the operating state from the shovel 100. This enables the information processing device 200 to grasp the operating state of the shovel 100 and monitor the presence or absence of abnormalities in the shovel 100. Furthermore, the information processing device 200 displays data relating to the operating state of the shovel 100 via a display device 208, which will be described later, and allows the user to confirm the data.

[0036] Furthermore, the information processing device 200 transmits to the shovel 100, for example, various data such as programs and reference data used in the processing of the controller 30, etc., to the shovel 100. This allows the shovel 100 to perform various processes related to the operation of the shovel 100 using the various data downloaded from the information processing device 200.

[0037] Furthermore, the information processing device 200 performs processing to support a function (hereinafter referred to as an "operation suggestion function") relating to proposing an operation of the shovel 100 to the operator, which will be described later (see FIG. 6). Details will be described later.

[0038] [Hardware configuration of the operation support system] Next, the hardware configuration of the operation support system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS.

[0039] <Excavator hardware configuration> FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel 100.

[0040] In Figure 4, the paths through which mechanical power is transmitted are indicated by double lines, the paths through which high-pressure hydraulic oil that drives the hydraulic actuator flows are indicated by solid lines, the paths through which pilot pressure is transmitted are indicated by dashed lines, and the paths through which electrical signals are transmitted are indicated by dotted lines.

[0041] The shovel 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0042] <Hydraulic drive system> 4, the hydraulic drive system of the excavator 100 includes the hydraulic actuator HA that hydraulically drives each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes the engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0043] The hydraulic actuator HA includes traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

[0044] Note that the hydraulic actuators HA of the shovel 100 may be partially or entirely replaced with electric actuators. In other words, the shovel 100 may be a hybrid shovel or an electric shovel.

[0045] The engine 11 is the prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, on the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a controller 30 (described later), and drives the main pump 14 and the pilot pump 15.

[0046] It should be noted that instead of or in addition to the engine 11, the excavator 100 may be equipped with another prime mover (for example, an electric motor).

[0047] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilting angle") in response to a control command from the controller 30.

[0048] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0049] The control valve 17 drives the hydraulic actuators HA in response to an operator's operation of the operating device 26 or the details of remote operation. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operation command corresponding to the operator's operation or the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (also referred to as "directional control valves") that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0050] ≪Operation system≫ As shown in FIG. 4, the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0051] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

[0052] The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic devices as pilot pressure.

[0053] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements, thereby enabling the operator to operate the driven elements that are the targets of the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0054] For example, as shown in Fig. 4, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32, and is connected to the control valve 17 via a pilot line 27 connected to an outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0055] Furthermore, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, that is, a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0056] Furthermore, the control valves (directional control valves) that are built into the control valve 17 and drive the hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, an operation signal output from the operating device 26 may be input directly to the control valve 17, i.e., the electromagnetic solenoid type control valve.

[0057] Furthermore, as described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output a control command to the electric actuator or a driver that drives the electric actuator, depending on the operation content of the operation device 26 or the content of remote operation specified by the remote operation signal. Furthermore, when the excavator 100 is remotely operated, the operation device 26 may be omitted.

[0058] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). In other words, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA. The hydraulic control valve 31 may be provided, for example, in a pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow path area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to a secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4 , the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via a shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, the controller 30 can supply pilot pressure according to the operation of the operating device 26 from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the excavator 100 based on the operation of the operator.

[0059] Furthermore, the controller 30 may, for example, control the hydraulic control valve 31 to realize remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal corresponding to the content of remote operation specified in a remote operation signal received from the remote operation support device 300 to the hydraulic control valve 31 via the communication device 60. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of remote operation to the control valve 17, thereby realizing operation of the shovel 100 based on remote operation by the operator.

[0060] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having a higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. A shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). One of the two inlet ports of the shuttle valve 32 is connected to a pilot line 27A on the secondary side of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of a corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator that is to be operated by the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31, the controller 30 can control the corresponding control valve regardless of the operation of the operating device 26 by the operator. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper revolving body 3, attachment AT) regardless of the operating state of the operating device 26 by the operator, thereby realizing a remote operation function.

[0061] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the remote control function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0062] <User Interface> As shown in FIG. 4, the user interface system of the excavator 100 includes an operation device 26, an output device 50, and an input device 52.

[0063] The output device 50 outputs various information to a user of the shovel 100 (for example, an operator of the cabin 10 or an external remote control operator) or to people in the vicinity of the shovel 100 (for example, a worker or a driver of a work vehicle).

[0064] For example, the output device 50 includes a lighting device, a display device 50A (see FIG. 6), etc. that output various types of information visually. The lighting device is, for example, a warning light (indicator lamp), etc. The display device 50A is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc. For example, as shown in FIG. 2, the lighting device and the display device 50A may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting device and the display device 50A may be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the excavator 100.

[0065] Furthermore, for example, the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various types of information auditorily. The sound output device 50B includes, for example, a buzzer, a speaker, etc. The sound output device 50B may be provided, for example, inside or outside the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the excavator 100.

[0066] Furthermore, for example, the output device 50 may include a device that outputs various types of information in a tactile manner, such as by vibrating the cockpit.

[0067] The input device 52 receives various inputs from the user of the excavator 100, and signals corresponding to the received inputs are taken into the controller 30. The input device 52 is provided, for example, inside the cabin 10, and receives inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3, and receive inputs from workers or the like around the excavator 100.

[0068] For example, the input device 52 includes an operation input device that accepts operation inputs. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0069] Furthermore, for example, the input device 52 may include an audio input device that accepts audio input from the user. The audio input device includes, for example, a microphone.

[0070] Furthermore, for example, the input device 52 may include a gesture input device that accepts gesture inputs from the user. The gesture input device includes, for example, an imaging device that captures an image of a gesture made by the user.

[0071] Furthermore, for example, the input device 52 may include a biometric input device that accepts biometric input from the user. The biometric input includes input of biometric information such as the user's fingerprint or iris.

[0072] <Communications> As shown in FIG. 4, the communication system of the shovel 100 according to this embodiment includes a communication device 60.

[0073] The communication device 60 connects to an external communication line and communicates with a device provided separately from the shovel 100. The device provided separately from the shovel 100 may include a device external to the shovel 100, as well as a portable terminal device (mobile terminal) brought into the cabin 10 by the user of the shovel 100. The communication device 60 may be, for example, a 4G (4 th Generation) and 5G (5 th The communication device 60 may include a mobile communication module conforming to standards such as the IEEE 802.11 Generation. The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. The communication device 60 may also include multiple communication devices in accordance with the communication lines to be connected.

[0074] For example, the communication device 60 communicates with external devices such as the information processing device 200 and the remote operation support device 300 within the work site through a local communication line established at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line established at the work site or a local network (LAN: Local Area Network) using WiFi6.

[0075] Furthermore, for example, the communication device 60 communicates with the information processing device 200 and the remote operation support device 300 outside the work site through a wide area communication line that includes the work site, i.e., a wide area network (WAN). The wide area network includes, for example, a wide area mobile communication network, a satellite communication network, the Internet network, etc.

[0076] <Control System> 4, the control system of the shovel 100 includes a controller 30. The control system of the shovel 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S5.

[0077] The controller 30 performs various controls related to the shovel 100 .

[0078] The functions of the controller 30 may be realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus B1.

[0079] The auxiliary storage device 30A is a non-volatile storage means that stores the programs to be installed as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

[0080] For example, when an instruction to start a program is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0081] The CPU 30C executes, for example, a program loaded into the memory device 30B, and realizes various functions of the controller 30 according to instructions from the program.

[0082] The interface device 30D functions as, for example, a communication interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of communication interfaces in accordance with the types of communication lines to be connected.

[0083] The interface device 30D also functions as an external interface for reading data from a recording medium and writing data to a recording medium. The recording medium is, for example, a dedicated tool connected to a connector installed inside the cabin 10 via a detachable cable. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes various functions of the controller 30 may be provided by, for example, a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may also be downloaded from another computer external to the excavator 100 via the communication device 60 and installed in the auxiliary storage device 30A.

[0084] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.

[0085] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. A detection signal of the pilot pressure by the operating pressure sensor 29 corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 is taken into the controller 30.

[0086] If the operating device 26 is an electrical type, the operating pressure sensor 29 is omitted because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26.

[0087] The imaging device 40 acquires an image of the periphery of the shovel 100. Furthermore, the imaging device 40 may acquire (generate) three-dimensional data (hereinafter simply referred to as "three-dimensional data of objects") representing the positions and outer shapes of objects around the shovel 100 within the imaging range (angle of view) based on the acquired image and data related to distance, which will be described later. The three-dimensional data of objects around the shovel 100 is, for example, data of coordinate information of a point cloud representing the surface of the object, range image data, etc.

[0088] 2, the imaging device 40 includes a camera 40F that images the area in front of the upper rotating body 3, a camera 40B that images the area behind the upper rotating body 3, a camera 40L that images the area to the left of the upper rotating body 3, and a camera 40R that images the area to the right of the upper rotating body 3. As a result, the imaging device 40 can capture images of the entire circumference of the shovel 100, i.e., a range covering an angular direction of 360 degrees, when viewed from above the shovel 100. Furthermore, the operator can visually recognize images captured by the cameras 40B, 40L, and 40R and peripheral images such as processing images generated based on the captured images through the output device 50 (display device) and the remote control display device, and can check the conditions to the left, right, and rear of the upper rotating body 3. Furthermore, by visually recognizing images captured by the camera 40F and peripheral images such as processing images generated based on the captured images through the remote control display device, the operator can remotely operate the shovel 100 while checking the operation of the attachment AT including the bucket 6. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X."

[0089] Camera 40X is, for example, a monocular camera. Camera 40X may also be, for example, a stereo camera, a TOF (Time Of Flight) camera, or the like (hereinafter collectively referred to as a "3D camera") that can acquire data related to distance (depth) in addition to two-dimensional images.

[0090] The output data of the imaging device 40 (camera 40X) (for example, image data or three-dimensional data of objects around the shovel 100) is taken into the controller 30 via a one-to-one communication line or an in-vehicle network. This allows the controller 30 to monitor objects around the shovel 100 based on the output data of the camera 40X. Also, for example, the controller 30 can determine the surrounding environment of the shovel 100 based on the output data of the camera 40X. Also, for example, the controller 30 can determine the attitude state of the attachment AT shown in the captured image based on the output data of the camera 40X (camera 40F). Also, for example, the controller 30 can determine the attitude state of the body (upper rotating body 3) of the shovel 100 based on the output data of the camera 40X, using objects around the shovel 100 as a reference.

[0091] Note that some of the cameras 40F, 40B, 40L, and 40R may be omitted. For example, if the shovel 100 is not remotely operated, the cameras 40F and 40L may be omitted. This is because it is relatively easy for the operator in the cabin 10 to confirm the conditions in front of and to the left of the shovel 100. Also, instead of or in addition to the imaging device 40 (camera 40X), a distance sensor may be provided on the upper rotating body 3. The distance sensor is attached, for example, to the top of the upper rotating body 3, and acquires data regarding the distance and direction of surrounding objects relative to the shovel 100. Furthermore, the distance sensor may acquire (generate) three-dimensional data (for example, point cloud coordinate information data) of objects around the shovel 100 within its sensing range based on the acquired data. The distance sensor is, for example, a LIDAR (Light Detection and Ranging). Also, for example, the distance sensor may be, for example, a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, or the like.

[0092] Sensor S1 is attached to the boom 4 and detects the attitude angle (hereinafter referred to as "boom angle") around the rotation axis of the base end of the boom 4, which corresponds to the connection part between the boom 4 and the upper rotating body 3. Sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may be true for sensors S2 and S4 below. Sensor S1 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may be true for sensor S2 below. The boom angle detection signal from sensor S1 is input to controller 30. This allows controller 30 to grasp the attitude state of the boom 4.

[0093] The sensor S2 is attached to the arm 5 and detects the attitude angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The detection signal of the arm angle by the sensor S2 is input to the controller 30. This enables the controller 30 to grasp the attitude state of the arm 5.

[0094] Sensor S3 is attached to bucket 6 and detects the attitude angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of bucket 6, which corresponds to the connection part between bucket 6 and arm 5. The arm angle detection signal from sensor S3 is input to controller 30. This enables controller 30 to grasp the attitude state of bucket 6.

[0095] The sensor S4 detects the inclination state of the machine body (for example, the upper rotating body 3) relative to a predetermined reference plane (for example, a horizontal plane). The sensor S4 is attached to the upper rotating body 3, for example, and detects the inclination angles (hereinafter referred to as "fore-aft inclination angle" and "left-right inclination angle") of the excavator 100 (i.e., the upper rotating body 3) about two axes in the front-rear and left-right directions. The detection signals corresponding to the inclination angles (fore-aft inclination angle and left-right inclination angle) detected by the sensor S4 are input to the controller 30. This allows the controller 30 to grasp the inclination state of the machine body (upper rotating body 3).

[0096] The sensor S5 is attached to the upper rotating body 3 and outputs detection information related to the rotation state of the upper rotating body 3. The sensor S5 detects, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The detection information related to the rotation state detected by the sensor S5 is input into the controller 30. This allows the controller 30 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0097] If sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocity around three axes, the rotation state (e.g., rotation angular velocity) of the upper rotating body 3 may be detected based on the detection signal of sensor S4. In this case, sensor S5 may be omitted. Also, if it is possible to grasp the attitude state of the upper rotating body 3, attachment AT, etc. based on the output of image capture device 40 or a distance sensor, at least some of sensors S1 to S5 may be omitted.

[0098] <Hardware configuration of information processing device> FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200. As shown in FIG.

[0099] The functions of the information processing device 200 are realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed calculation device 205, a communication interface 206, an input device 207, and a display device 208, which are connected via a bus B2.

[0100] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. Examples of the recording medium 201A include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. This allows the information processing device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions.

[0101] The information processing device 200 may acquire various data and programs used in processing from an external device via the communication interface 206.

[0102] The auxiliary storage device 202 stores various installed programs as well as files and data required for various processes. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state disk (SSD), a flash memory, etc.

[0103] When an instruction to start a program is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a dynamic random access memory (DRAM) or an SRAM.

[0104] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the information processing device 200 in accordance with the programs.

[0105] The high-speed arithmetic unit 205 performs arithmetic processing at a relatively high speed in cooperation with the CPU 204. The high-speed arithmetic unit 205 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0106] The high speed calculation device 205 may be omitted depending on the required calculation processing speed.

[0107] The communication interface 206 is used as an interface for communicatively connecting with an external device. This allows the information processing device 200 to communicate with an external device such as the shovel 100 through the communication interface 206. The communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected device and the like.

[0108] The input device 207 accepts various inputs from the user.

[0109] The input device 207 includes, for example, an operation input device that accepts mechanical operation input from a user. The operation input device includes, for example, a button, a toggle, a lever, etc. The operation input device also includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0110] The input device 207 also includes, for example, a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0111] The input device 207 includes, for example, a gesture input device capable of receiving a gesture input from a user. The gesture input device includes, for example, a camera capable of capturing an image of a user's gesture.

[0112] The input device 207 includes, for example, a biometric input device capable of accepting biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about a user's fingerprint or iris.

[0113] The display device 208 displays an information screen or an operation screen for the user. For example, the display device 208 includes the above-mentioned remote control display device. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0114] Note that, like the information processing device 200, the remote operation support device 300 may be realized by any hardware or any combination of hardware and software, and may have a similar hardware configuration. For example, like the information processing device 200 ( FIG. 5 ), the remote operation support device 300 is primarily configured with a computer including a CPU, a memory device, an auxiliary storage device, an interface device, an input device, and a display device. The memory device is, for example, an SRAM or DRAM. The auxiliary storage device is, for example, an HDD, an SSD, an EEPROM, or a flash memory. The interface device includes an external interface for connecting to an external recording medium and a communication interface for communicating with the outside, such as the shovel 100. The input device includes, for example, a lever-type operation input device. This allows the operator to use the operation input device to input operations related to the actuator of the shovel 100, and the remote operation support device 300 can transmit a signal corresponding to the operation input to the shovel 100 using the communication interface. Therefore, the operator can remotely control the shovel 100 using the remote operation support device.

[0115] [First example of the excavator motion suggestion function] Next, with reference to FIGS. 6 and 7 in addition to FIGS. 1 to 5, a first example of a function for suggesting to a user (operator) the operation of the shovel 100 (operation suggestion function) will be described.

[0116] <Functional configuration> FIG. 6 is a functional block diagram showing a first example of a functional configuration related to the operation suggestion function of the operation support system SYS.

[0117] The shovel 100 includes a support device 150. The support device 150 supports the operation of the shovel 100 by the operator.

[0118] As shown in FIG. 6, the support device 150 includes a controller 30, an imaging device 40, an output device 50, and a communication device 60.

[0119] The controller 30 includes an operation log providing unit 301 and a work support unit 302 as functional units.

[0120] When the operation support system SYS includes a plurality of shovels 100, there may be an shovel 100 in which the controller 30 includes only the operation log providing unit 301 and the work support unit 302, and an shovel 100 in which the controller 30 includes only the latter. In this case, the former shovel 100 has only the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the operation support function (operation suggestion function) for the operator in the latter shovel 100. The same may be true for the case of a second example (FIG. 8) described below.

[0121] The information processing device 200 includes, as functional units, an operation log acquisition unit 2001, an operation log storage unit 2002, a teacher data generation unit 2003, a machine learning unit 2004, a trained model storage unit 2005, and a distribution unit 2006.

[0122] The operation log providing unit 301 is a functional unit for acquiring an operation log of the shovel 100, which is the original data for realizing the operation suggestion function, and providing it to the information processing device 200. Specifically, the operation log is acquired when an operator who has a long history of operating the shovel 100 and is relatively experienced (hereinafter, for convenience, referred to as an "expert") operates the shovel 100, and is provided to the information processing device 200.

[0123] The operation log of the shovel 100 includes data on the shape of a work object around the shovel 100 and data on the operation of the shovel 100 performed on the shape of the work object. The data on the shape of the work object around the shovel 100 is, for example, data on the topographical shape of the ground at the work site as the work object of the shovel 100. The data on the shape of the work object of the shovel 100 is, for example, image data from the imaging device 40 or three-dimensional data of the work object obtained from the image data. The data on the operation of the shovel 100 is, for example, data representing the operation content of the operator. The data representing the operation content of the operator is, for example, output data of the operating pressure sensor 29 in the case of a hydraulic pilot type operating device 26, or output data (operation signal data) of the operating device 26 in the case of an electric operating device 26. The data on the operation of the shovel 100 may also be data representing the operating state of the shovel 100 actually performed in response to the operation of the operator. The data representing the operating state of the shovel 100 is, for example, output data of the sensors S1 to S5, or data relating to the posture state of the shovel 100 acquired from the output data of the sensors S1 to S5.

[0124] Operation log provider 301 includes operation log recording unit 301A, operation log storage unit 301B, and operation log transmission unit 301C.

[0125] The operation log recording unit 301A acquires an operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, every time an operation of the shovel 100 is executed, the operation log recording unit 301A records data on the shape of the work target around the shovel 100 at the start of or immediately before the execution of the operation, and data on the operation of the shovel 100 in the operation log storage unit 301B.

[0126] The operation log storage unit 301B stores an operation log of the shovel 100 in an accumulated form. For example, the operation log storage unit 301B stores data relating to the shape of a work target around the shovel 100 for each operation of the shovel 100 and data relating to the operation of the shovel 100 in a linked form. Specifically, the operation log storage unit 301B may accumulate record data indicating a correspondence between the data relating to the shape of a work target around the shovel 100 for each operation of the shovel 100 and the data relating to the operation of the shovel 100, and a database of operation logs may be constructed.

[0127] The operation log stored in operation log storage unit 301B that has already been transmitted to information processing device 200 by operation log transmission unit 301C (described later) may be deleted afterward.

[0128] The operation log transmission unit 301C transmits the operation log of the shovel 100 stored in the operation log storage unit 301B to the information processing device 200 via the communication device 60. The operation log transmission unit 301C may also transmit to the information processing device 200 data relating to the shape of the work target around the shovel 100 for each operation of the shovel 100, and record data indicating the correspondence between the data relating to the operation of the shovel 100.

[0129] For example, the operation log transmission unit 301C transmits an untransmitted operation log of the shovel 100 stored in the operation log storage unit 301B to the information processing device 200 in response to a signal (hereinafter referred to as a "transmission request signal") requesting transmission of the operation log of the shovel 100, which is received from the information processing device 200. Furthermore, the operation log transmission unit 301C may automatically transmit an untransmitted operation log of the shovel 100 stored in the operation log storage unit 301B to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, when the shovel 100 stops operating (when the key switch is turned off) or when it starts operating (when the key switch is turned on).

[0130] The operation log acquisition unit 2001 acquires the operation log of the shovel 100 received from the shovel 100 .

[0131] The operation log acquisition unit 2001 acquires the operation log of the shovel 100 by transmitting a transmission request signal to the shovel 100 in response to an operation by a user of the information processing device 200 or automatically at a predetermined timing. The operation log acquisition unit 2001 may also acquire the operation log of the shovel 100 transmitted from the shovel 100 at a predetermined timing.

[0132] The operation log storage unit 2002 stores an accumulated operation log of the shovel 100 acquired by the operation log acquisition unit 2001. For example, similar to the case of the operation log storage unit 301B, the operation log storage unit 2002 stores data relating to the shape of the work target around the shovel 100 for each operation of the shovel 100 and data relating to the operation of the shovel 100 in a linked manner.

[0133] The training data generation unit 2003 generates training data for machine learning based on the operation log of the shovel 100 stored in the operation log storage unit 2002. The training data generation unit 2003 may generate training data automatically by batch processing, or may generate training data in response to input from a user of the information processing device 200. The training data is a combination of data relating to the shape of a work object around the shovel 100 as input data and data representing the operation of the shovel 100 corresponding to the shape of the work object, which corresponds to the input data as correct output data (hereinafter referred to as "correct data").

[0134] The correct answer data includes, for example, data representing the type of action selected from a plurality of candidate actions that can be performed in a predetermined task. For example, in the case of leveling the ground at a work site, the plurality of candidate actions include a sweeping action, a leveling action, a rolling action, a broom action, etc. The sweeping action is, for example, an action in which the attachment AT is operated and the bucket 6 is pushed forward along the ground, thereby sweeping earth and sand forward with the back of the bucket 6. In the sweeping action, for example, the attachment AT is operated by the boom 4. ofThe attachment AT lowers the boom 4 and opens the arm 5. The horizontal towing operation is, for example, an operation in which the attachment AT operates to move the tip of the bucket 6 approximately horizontally along the ground surface, pulling it toward the user, thereby smoothing out unevenness on the ground surface. In the horizontal towing operation, the attachment AT raises the boom 4 and closes the arm 5. The compaction operation is, for example, an operation in which the attachment AT operates to press the ground with the back surface of the bucket 6. The compaction operation may also be an operation in which the bucket 6 is pushed forward along the ground surface, the back surface of the bucket 6 sweeps out soil and sand to a predetermined position forward, and then the back surface of the bucket 6 presses the ground at the predetermined position against the ground. In the compaction operation, for example, the attachment AT lowers the boom 4 while pressing the ground. The broom operation is, for example, an operation in which the upper rotating body 3 is operated to rotate the bucket 6 left and right while keeping it along the ground surface. Furthermore, the broom operation may be, for example, an operation in which the attachment AT and the upper rotating body 3 are operated to alternately rotate left and right while keeping the bucket 6 along the ground, while pushing the bucket 6 forward. In the broom operation, for example, the upper rotating body 3 repeatedly rotates left and right. In addition to the alternately rotating left and right of the upper rotating body 3, the broom operation may also involve the attachment AT lowering the boom 4 and opening the arm 5, as in the case of the sweeping operation. Furthermore, the correct data may include, for example, data representing the trajectory of the bucket 6 when the excavator 100 is operating.

[0135] The machine learning unit 2004 generates a trained model LM by performing machine learning on the base learning model based on the set of training data generated by the training data generation unit 2003. The trained model LM (base learning model) includes, for example, a neural network such as a DNN (Deep Neural Network).

[0136] The trained model LM, for example, uses data regarding the shape of a work object around the shovel 100 as an input condition and outputs a predicted probability for each of a plurality of candidate actions to be performed in a predetermined task. This predicted probability represents the reliability of the candidate action. As described above, the trained model LM reflects the action log of the skilled worker when operating the shovel 100, and it is considered that the higher the predicted probability, the higher the reliability of selecting the candidate action. Furthermore, this predicted probability represents the degree of suitability for the shape of the work object around the shovel 100, which is an input condition. This is because it is considered that the higher the predicted probability, the more likely the skilled worker will determine that the candidate action is suitable for the shape of the work object. Furthermore, the trained model LM may use data regarding the shape of the work object around the shovel 100 as an input condition and output data representing the trajectory of the bucket 6 (hereinafter referred to as the "target trajectory") for each of a plurality of candidate actions. Furthermore, the trained model may use data regarding the shape of the work object around the shovel 100 as an input condition and output multiple pieces of data representing the target trajectory of the bucket 6 for each of a plurality of candidate actions, along with a predicted probability for each of the multiple target trajectories of the bucket 6. As with the prediction probability of the candidate motion, this prediction probability represents the reliability of the target trajectory of the target and the degree of conformance to the shape of the work object around the excavator 100 as an input condition. Furthermore, a trained model LM may be generated for each of a plurality of different tasks. For example, a trained model LM is generated for each task such as ground leveling, slope construction, and embankment work.

[0137] The trained model storage unit 2005 stores the trained model LM output by the machine learning unit 2004.

[0138] The distribution unit 2006 distributes the learned model LM to the excavator 100.

[0139] For example, when a trained model LM is generated by the machine learning unit 2004, the distribution unit 2006 distributes the most recently generated trained model LM to the shovel 100. In addition, the distribution unit 2006 may distribute the latest trained model LM in the trained model storage unit 2005 to the shovel 100 in response to a signal received from the shovel 100 requesting distribution of the trained model LM.

[0140] The work support unit 302 is a functional unit for supporting the work of the excavator 100 operated by the operator.

[0141] The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, an estimation unit 302C, and a proposal unit 302D.

[0142] The trained model storage unit 302A stores the trained model LM that is distributed from the information processing device 200 and received via the communication device 60.

[0143] The work object shape acquisition unit 302B acquires data relating to the shape of the work object (terrain shape) around the shovel 100 based on the output of the imaging device 40 and the distance sensor.

[0144] The estimation unit 302C estimates, from among a plurality of candidate actions that may be performed in a predetermined task, an action that has a relatively high reliability or compatibility with the shape of the work object around the shovel 100, based on data related to the shape of the work object around the shovel 100. Furthermore, the estimation unit 302C may estimate, for each of a plurality of candidate actions, one or more target trajectories of the bucket 6 that have a relatively high reliability or compatibility, based on data related to the shape of the work object around the shovel 100.

[0145] Specifically, the estimation unit 302C may use the learned model LM to estimate an operation that has a relatively high reliability and conformance with respect to the shape of the work object around the shovel 100, using data related to the shape of the work object around the shovel 100 as an input condition. Furthermore, the estimation unit 302C may use the learned model LM to estimate a target trajectory of one or more buckets 6 that has a relatively high reliability and conformance with respect to the shape of the work object around the shovel 100, using data related to the shape of the work object around the shovel 100 as an input condition.

[0146] Based on the estimation result of the estimation unit 302C, the suggestion unit 302D suggests to the operator in the cabin 10, via the output device 50 such as the display device 50A, an operation of the shovel 100 that has a relatively high reliability or suitability with respect to the shape of the work object around the shovel 100. This allows even an inexperienced operator to select a more appropriate operation that matches the current shape of the work object around the shovel 100. This improves the convenience for the operator and also improves the work efficiency of the shovel 100. The operation suggested to the operator may be one or more. For example, the suggestion unit 302D suggests an operation with a relatively high suitability by notifying the operator of the suitability (reliability) values ​​for all or some of the multiple candidate operations with respect to the shape of the work object around the shovel 100 (see FIG. 10 described below).

[0147] Furthermore, based on the estimation result of the estimation unit 302C, the proposing unit 302D may propose, via the output device 50, one target trajectory of the bucket 6 for the proposed operation that has a relatively high degree of reliability and compatibility with the shape of the work target around the shovel 100. This allows even an inexperienced operator to grasp a more appropriate target trajectory of the bucket 6 that matches the current shape of the work target around the shovel 100, and to operate the shovel 100 to achieve that target trajectory. This can further improve the convenience for the operator and can further improve the work efficiency of the shovel 100.

[0148] Furthermore, the suggestion unit 302D may propose, via the output device 50 for the operation target, multiple target trajectories of the bucket 6 for the proposed operation that have relatively high reliability and compatibility with the shape of the work object around the shovel 100. This allows the operator to grasp multiple target trajectories of the bucket 6 that are more appropriate for the current shape of the work object around the shovel 100, and operate the shovel 100 to achieve one target trajectory of his / her choice. This makes it possible to improve the work efficiency of the shovel 100 in a manner that reflects the operator's intentions. For example, the suggestion unit 302D proposes a target trajectory of the bucket 6 with a relatively high compatibility by notifying the operator of a numerical value of compatibility (reliability) with the shape of the work object around the shovel 100 for each of the multiple target trajectories of the proposed operation (see FIGS. 11 and 13 described below).

[0149] Furthermore, when the shovel 100 is remotely operated, the suggestion unit 302D may suggest an action or a target trajectory of the bucket 6 that has a relatively high reliability or suitability to the operator using the remote operation support device 300 via the communication device 60. In this case, the suggestion unit 302D transmits data representing the content of the suggestion to the remote operation support device 300 via the communication device 60. This allows the remote operation support device 300 to suggest an action or a target trajectory of the bucket 6 that has a relatively high reliability or suitability to the operator using the remote operation support device 300 using a display device, a sound output device, or the like.

[0150] <Processing> FIG. 7 is a flowchart schematically showing a first example of processing related to the operation suggestion function of the shovel 100.

[0151] 7 starts when a predetermined input for starting the action suggestion function is received via, for example, the input device 52 or the input device of the remote operation support device 300. The same may be true for the flowchart in FIG. 9 described below.

[0152] As shown in FIG. 7, in step S102 (an example of an acquisition step), the work object shape acquisition unit 302B acquires data relating to the shape of the work object around the shovel 100 based on the output of the imaging device 40.

[0153] When the process of step S102 is completed, the controller 30 proceeds to step S104.

[0154] In step S104, the estimation unit 302C estimates an operation that has a relatively high degree of compatibility (reliability) with the current shape of the work target around the excavator 100, based on the data acquired in step S102.

[0155] When the process of step S104 is completed, the controller 30 proceeds to step S106.

[0156] In step S106 (an example of a proposing step), the proposing unit 302D causes the display device 50A to display a motion to be proposed from among the plurality of candidate motions and a target trajectory of the motion, based on the estimation result in step S104.

[0157] When the process of step S106 is completed, the controller 30 proceeds to step S108.

[0158] In step S108, the controller 30 determines whether or not the driven element (actuator) has been operated. If the driven element has not been operated, the controller 30 proceeds to step S110, and if the driven element has been operated, the controller 30 proceeds to step S112.

[0159] In step S110, the controller 30 determines whether a termination condition is met. The termination condition is, for example, receipt of a predetermined input from the operator, via the input device 52 or an input device of the remote operation support device 300, indicating the termination of the action suggestion function. Alternatively, the termination condition may be receipt of a predetermined input from the operator, via the input device 52 or an input device of the remote operation support device 300, indicating the completion of the task ... If the termination condition is met, the controller 30 ends the processing of this flowchart, and if the termination condition is not met, the controller 30 returns to step S108.

[0160] On the other hand, in step S112, the controller 30 determines whether or not the operation of the driven element corresponding to one operation of the shovel 100 has been completed, based on the operation state of the operation device 26, the operating state of the shovel 100, etc. The controller 30 can grasp the operation state of the operation device 26, the operating state of the shovel 100, etc., based on the output of the operation pressure sensor 29, the operation signal output from the operation device 26, the outputs of the sensors S1 to S5, etc. If the operation of the driven element corresponding to one operation of the shovel 100 has been completed, the controller 30 proceeds to step S114, and if it has not been completed, the controller 30 waits until it is completed (repeats the processing of step S112).

[0161] In step S114, the controller 30 determines whether or not the termination condition is met. If the termination condition is met, the controller 30 ends the processing of this flowchart, and if the termination condition is not met, the controller 30 returns to step S102.

[0162] In this way, in this example, the support device 150 can suggest to the operator, via the display device 50A and the remote operation support device 300, the operation of the shovel 100 and the target trajectory of the bucket 6 that have a high degree of compatibility (reliability) with respect to the shape of the work object around the shovel 100.

[0163] [Second example of the excavator motion suggestion function] Next, a second example of the function of suggesting an operation of the shovel 100 to the user (operator) will be described with reference to FIGS. 8 and 9 in addition to FIGS. 1 to 5. FIG.

[0164] The following description will focus on the differences from the first example described above, and the description of the same or corresponding content as the first example described above may be simplified or omitted.

[0165] <Functional configuration> FIG. 8 is a functional block diagram showing a first example of a functional configuration related to the operation suggestion function of the operation support system SYS.

[0166] As shown in FIG. 8, the support device 150 of the excavator 100 includes a controller 30, a hydraulic control valve 31, an imaging device 40, an output device 50 (display device 50A), an input device 52, and a communication device 60.

[0167] The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302, similar to the first example described above.

[0168] The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, an estimation unit 302C, a proposal unit 302D, and a motion control unit 302E.

[0169] The operation control unit 302E controls the hydraulic control valve 31 in response to instructions input from the operator received via the input device 52 and the communication device 60, and automatically executes the operation of the shovel 100 proposed to the operator by the suggestion unit 302D. As a result, the support device 150 can cause the shovel 100 to automatically execute the proposed operation in accordance with the shape of the work target currently around the shovel 100, assuming the instruction input from the operator. Therefore, even an inexperienced operator can more appropriately execute the operation of the shovel 100 in accordance with the shape of the work target currently around the shovel 100, simply by inputting the instruction. Therefore, it is possible to further improve the convenience for the operator and to further improve the work efficiency of the shovel 100.

[0170] For example, if there is one proposed motion, the motion control unit 302E controls the hydraulic control valve 31 in response to an instruction input from the operator, and automatically executes the motion of the shovel 100 proposed by the proposing unit 302D. Furthermore, for example, if there are multiple proposed motions, the motion control unit 302E automatically executes one of the multiple proposed motions selected by an instruction input from the operator. Furthermore, for example, if the proposing unit 302D proposes one target trajectory for the bucket 6, the motion control unit 302E automatically executes the proposed motion of the shovel 100 so that the bucket 6 moves along the proposed target trajectory. Furthermore, for example, if the proposing unit 302D proposes multiple target trajectories for the bucket 6, the motion control unit 302E automatically executes the proposed motion so that the bucket 6 moves along one target trajectory selected by an instruction input from the operator among the multiple target trajectories.

[0171] The operation log recording unit 301A records an operation log in the operation log storage unit 301B, the operation log including data related to the shape of the work object acquired by the work object shape acquisition unit 302B and data representing the operation and target trajectory performed by the operation control unit 302E. This allows the operation log transmission unit 301C to accumulate, in the operation log storage unit 301B, an operation log including data related to the shape of the work object and data representing the operation and target trajectory of the shovel 100 actually performed by the operator based on the shape of the work object. The operation log transmission unit 301C can then upload the accumulated operation log to the information processing device 200. Therefore, the machine learning unit 2004 can use the operation log to re-train or additionally train the trained model LM, thereby updating the trained model LM.

[0172] In addition, the machine learning unit 2004 may compare the re-trained or additionally trained trained model LM with the current trained model LM using specified evaluation data, and if the evaluation result of the former is higher, update the trained model LM in the trained model memory unit 2005.

[0173] <Processing> FIG. 9 is a flowchart schematically illustrating a second example of processing related to the operation suggestion function of the shovel 100.

[0174] As shown in FIG. 9, the processing in steps S202 to S206 is the same as steps S102 to S106 in FIG. 7, and therefore a description thereof will be omitted.

[0175] When the process of step S206 is completed, the controller 30 proceeds to step S208.

[0176] In step S208, the controller 30 determines whether or not an input instructing the execution of the proposed action (hereinafter, "execution instruction input") has been received from the operator via the input device 52 or the communication device 60. If the execution instruction input has not been received, the controller 30 proceeds to step S210, and if it has been received, the controller 30 proceeds to step S212.

[0177] In step S210, the controller 30 determines whether or not the termination condition is met. If the termination condition is met, the controller 30 ends the processing of this flowchart, and if the termination condition is not met, the controller 30 returns to step S208.

[0178] Meanwhile, in step S212, the operation control unit 302E controls the hydraulic control valve 31 to automatically execute the operation specified by the input of the execution instruction. Furthermore, when a target trajectory is specified by the input of the execution instruction, the operation control unit 302E executes the operation of the shovel 100 specified by the input of the execution instruction so that the bucket 6 moves along the target trajectory specified by the input of the execution instruction.

[0179] When the process of step S212 is completed, the controller 30 proceeds to step S214.

[0180] In step S214, the operation log recording unit 301A records an operation log in the operation log memory unit 301B, which includes data regarding the shape of the work object acquired by the work object shape acquisition unit 302B, and data representing the operations and target trajectories performed by the operation control unit 302E.

[0181] When the process of step S214 is completed, the controller 30 proceeds to step S216.

[0182] In step S216, the controller 30 determines whether the termination condition is met. If the termination condition is met, the controller 30 ends the processing of this flowchart. If the termination condition is not met, the controller 30 returns to step S202.

[0183] In this example, the work object shape acquisition unit 302B may acquire data relating to the shape of the work object by predicting a change in the shape of the work object due to the operation of the shovel 100 in the previous process of step S212.

[0184] In this way, in this example, the support device 150 can automatically execute the operation of the shovel 100 and the target trajectory of the bucket 6 that have a high degree of compatibility (reliability) with the shape of the work object around the shovel 100 in response to instructions from the operator.

[0185] Furthermore, in this example, the support device 150 can accumulate an operation log including data on the shape of the work target around the shovel 100 and data on the automatically performed operation and target trajectory of the shovel 100. Therefore, the support device 150 can update the learned model LM using the accumulated operation log.

[0186] [Example of display content for the excavator operation suggestion function] Next, specific examples of the display contents of the display device 50A relating to the operation suggestion function of the shovel 100 will be described with reference to FIGS.

[0187] 10 to 15 may be displayed on the display device of the remote operation support device 300.

[0188] <Example 1> FIG. 10 is a diagram showing a first example (screen 1000) of the display content of the display device 50A, relating to the operation suggestion function of the shovel 100. As shown in FIG.

[0189] Screen 1000 includes images 1001 to 1006.

[0190] Image 1001 is an image representing a work target in the vicinity of shovel 100. In this example, image 1001 is an image representing a work target in the vicinity of shovel 100 (the ground at the work site) as viewed from a predetermined viewpoint in the vicinity of shovel 100, which is generated based on the output (image data) of imaging device 40 using a known image processing technique.

[0191] The image 1002 is an image that schematically represents the shovel 100. In this example, the image 1002 is an image that schematically represents the shovel 100 when viewed from the same viewpoint as the image 1001, and is displayed superimposed on the image 1001.

[0192] Image 1003 is an image that shows, in a list format, motions to be suggested by suggestion unit 302D from among a plurality of candidate motions. In this example, image 1003 includes images 1003A to 1003D that represent rows of sweeping motion, horizontal dragging motion, rolling motion, and broom motion from among the plurality of candidate motions as motions to be suggested. In addition, in this example, images 1003A to 1003D show the reliability (suitability) of each of the broom motion, sweeping motion, horizontal dragging motion, and rolling motion. This allows the operator to select a motion to be performed by excavator 100 by hand or automatically, taking into consideration the reliability (suitability) from among the suggested motions.

[0193] Note that the image 1003 may only show the action with the highest reliability (suitability) (in this example, the sweeping action). That is, the suggestion unit 302D may suggest to the operator, through the image 1003, only the action with the highest reliability (suitability) among a plurality of candidate actions for a predetermined task. Furthermore, the image 1003 may only show, among the plurality of candidate actions, actions whose reliability (suitability) is equal to or greater than a predetermined standard (for example, 30%) (for example, the sweeping action and the horizontal pulling action). That is, the suggestion unit 302D may suggest to the operator, among the plurality of candidate actions, only actions whose reliability (suitability) is equal to or greater than a predetermined standard.

[0194] Image 1004 is an image that represents a target trajectory for each proposed motion, which is represented in image 1003. Image 1004 is displayed superimposed on image 1001 in the vicinity of image 1002. This allows the operator to easily grasp the target trajectory for each proposed motion while comparing it with image 1001 that represents the state of the ground at the work site around shovel 100 and image 1002 that represents shovel 100. Image 1004 includes images 1004A to 1004D.

[0195] Image 1004A is an image showing the target trajectory of the sweeping operation.

[0196] Image 1004B is an image showing the target trajectory of the horizontal pulling operation.

[0197] Image 1004C is an image showing the target trajectory of the rolling operation.

[0198] Image 1004D is an image showing the target trajectory of the broom motion.

[0199] The images 1004A-1004D may be displayed in a manner that allows the user to distinguish between the portion of the target trajectory that contacts the work object (ground) and the other portion. For example, the images 1004A-1004D may use different colors for the portion of the target trajectory that contacts the work object and the other portion. This helps create a sense of perspective on the image 1001 for the images 1004A-1004D that correspond to the target trajectory.

[0200] In this example, a matte cursor is displayed on image 1003A corresponding to the sweeping action. Furthermore, in this example, image 1004A corresponding to the sweeping action of image 1004 is displayed with a thicker line than images 1004B-1004D corresponding to the other actions. This indicates that the sweeping action has been selected. For example, the operator can select one of the sweeping action, horizontal dragging action, compaction action, and broom action by designating one of images 1003A-1003D using input device 52. Similarly, for example, the operator can select one of the sweeping action, horizontal dragging action, compaction action, and broom action by designating one of images 1004A-1004D using input device 52.

[0201] An image 1005 is an icon for confirming the execution of an action selected by the user (operator) from among the suggested actions.

[0202] For example, the operator can use the input device 52 to operate the image 1005 to cause the excavator 100 to automatically execute the selected operation.

[0203] Image 1006 is an icon for terminating the operation suggestion function of the shovel 100. The same applies to images 1106, 1206, 1306, 1406, and 1506 described below.

[0204] Thus, in this example, the controller 30 causes the display device 50A to display a plurality of actions from among a plurality of candidate actions for ground leveling work, together with the reliability (suitability) of each action with respect to the shape of the work object (terrain shape) currently in the vicinity of the shovel 100. This allows the controller 30 to suggest to the operator an action with a relatively high reliability (suitability) with respect to the shape of the work object (terrain shape) currently in the vicinity of the shovel 100.

[0205] <Example 2> FIG. 11 is a diagram showing a second example (screen 1100) of the display content of the display device 50A, relating to the operation suggestion function of the shovel 100.

[0206] The following description will focus on the differences from the first example described above, and the description of the same or corresponding content as the first example described above may be simplified or omitted.

[0207] Screen 1100 includes images 1101 to 1106.

[0208] Image 1101, like image 1001 in FIG. 10, is an image showing a work target around the excavator 100.

[0209] The image 1102 is an image that schematically represents the shovel 100, similar to the image 1002 in FIG.

[0210] Image 1103 is an image that represents, in list form, multiple target trajectories of the bucket 6 for one motion to be proposed by proposing unit 302D from among multiple candidate motions. In this example, image 1103 includes images 1103A to 1103D that represent the respective rows of four target trajectories of the bucket 6 ("sweeping out I" to "sweeping out IV") for the sweeping out motion as one motion to be proposed. In addition, in this example, images 1103A to 1103D express the reliability (fit) of each of the four target trajectories of the bucket 6. This allows the operator to select, for one motion to be proposed (sweeping out motion), a target trajectory of the bucket 6 to be executed by the excavator 100 by his own action or automatically, taking into consideration the reliability from among the four target trajectories of the bucket 6.

[0211] Note that the image 1103 may only show the target trajectory of the bucket 6 with the highest reliability (suitability) (in this example, "sweeping out I"). In other words, the suggestion unit 302D may suggest to the operator, through the image 1103, only the action with the highest reliability (suitability) among multiple target trajectories for one action to be proposed in a predetermined task. Furthermore, the image 1103 may only show target trajectories (for example, "sweeping out I" and "sweeping out II") with a reliability (suitability) equal to or higher than a predetermined standard (for example, 30%) among multiple target trajectories for the bucket 6. In other words, the suggestion unit 302D may suggest to the operator only target trajectories with a reliability (suitability) equal to or higher than a predetermined standard among multiple target trajectories for the bucket 6 for one action to be proposed. The same may be true for the image 1203A described below.

[0212] Image 1104 is an image showing four target trajectories for one proposed motion shown in image 1103. Like image 1004, image 1104 is displayed superimposed on image 1001 in the vicinity of image 1002. This allows the operator to easily grasp the multiple target trajectories for one proposed motion (sweeping motion) while comparing them with image 1101 showing the state of the ground at the work site around shovel 100 and image 1102 showing shovel 100. Image 1104 includes images 1104A to 1104D.

[0213] Image 1104A is an image representing the target trajectory of the sweeping operation corresponding to image 1103A ("Sweeping I").

[0214] Image 1104B is an image showing the target trajectory of the sweeping operation corresponding to image 1103B ("Sweeping II").

[0215] Image 1104C is an image representing the target trajectory of the sweeping operation, corresponding to image 1103C ("Sweeping III").

[0216] Image 1104D is an image showing the target trajectory of the sweeping operation corresponding to image 1103C ("Sweeping IV").

[0217] In this example, a matte cursor is represented in image 1103A corresponding to "sweeping operation I." Also, in this example, image 1104A corresponding to the target trajectory of "sweeping operation I" in image 1104 is represented with a thicker line than images 1104B to 1104D corresponding to the other target trajectories. This represents a state in which the target trajectory of "sweeping I" has been selected from among the four target trajectories. For example, the operator can select one of the four target trajectories by designating one of images 1103A to 1103D using input device 52. Similarly, for example, the operator can select one of the four target trajectories of bucket 6 by designating one of images 1104A to 1104D using input device 52.

[0218] An image 1105 is an icon for executing one proposed operation (sweeping operation) so that the bucket 6 moves along a target trajectory selected by a user (operator) from among a plurality of target trajectories.

[0219] For example, the operator can use the input device 52 to manipulate the image 1105 to cause the shovel 100 to automatically perform one of the proposed operations so that the bucket 6 moves along the selected target trajectory.

[0220] Thus, in this example, controller 30 causes display device 50A to display multiple target trajectories of bucket 6 for one operation related to ground leveling work, together with the reliability (degree of suitability) of each of the target trajectories with respect to the current shape of the work target (terrain shape) around shovel 100. This allows controller 30 to propose to the operator multiple target trajectories of bucket 6 with a relatively high reliability (degree of suitability) with respect to the current shape of the work target (terrain shape) around shovel 100.

[0221] <Example 3> 12 and 13 are diagrams showing a third example (screens 1200 and 1300) of the display contents of the display device 50A relating to the operation suggestion function of the shovel 100. In FIG.

[0222] The following description will focus on the differences from the first and second examples described above, and the description of the same or corresponding content as the first and second examples described above may be simplified or omitted.

[0223] Screen 1200 includes images 1201 to 1206.

[0224] Image 1201, like image 1001 in FIG. 10, is an image showing a work target around excavator 100.

[0225] The image 1202 is an image that schematically represents the shovel 100, similar to the image 1002 in FIG.

[0226] 10, image 1203 is an image that shows, in a list format, motions to be suggested by suggester 302D from among a plurality of candidate motions. In this example, image 1203 includes images 1203A to 1203D that show, as motions to be suggested, rows of sweeping motions, horizontal dragging motions, rolling motions, and broom motions from among a plurality of candidate motions. In this example, images 1203A to 1203D also show the reliability (degree of suitability) of each of the broom motions, sweeping motions, horizontal dragging motions, and rolling motions.

[0227] 11, image 1203A is an image that shows, in list form, multiple target trajectories for bucket 6 for one proposed action (sweeping action). In this example, image 1203A includes images 1203A1 to 1203A4 that show the respective rows of four target trajectories for bucket 6 ("sweeping I" to "sweeping IV") for the sweeping action that has the highest reliability (suitability) among multiple candidate actions. In this example, images 1203A1 to 1203A4 also show the reliability (suitability) of each of the four target trajectories for bucket 6.

[0228] 10, the image 1204 is an image that represents a target trajectory for each proposed motion, which is represented in the image 1203. The image 1204 includes images 1204A to 1204D.

[0229] Image 1204A is an image showing the target trajectory of the sweeping operation. Specifically, it is an image showing the target trajectory ("sweeping I") with the highest reliability among the target trajectories ("sweeping I" to "sweeping IV") of the bucket 6 corresponding to the sweeping operation.

[0230] Images 1204B to 1204D are the same as images 1004B to 1004D in FIG. 10, and therefore a description thereof will be omitted.

[0231] In this example, a matte cursor is displayed in image 1003A corresponding to the sweeping operation. Also, in this example, image 1004A corresponding to the sweeping operation of image 1004 is displayed with a thicker line than images 1004B to 1004D corresponding to the other operations. This indicates that the sweeping operation has been selected.

[0232] An image 1205 is an icon for confirming the execution of an action selected by the user (operator) from among the suggested actions.

[0233] Furthermore, image 1205 is an icon for transitioning to screen 1300 for selecting four target trajectories of bucket 6 corresponding to the sweeping action when the sweeping action with the highest reliability among the suggested actions has been selected. In other words, when image 1205 is operated via input device 52 in the state of screen 1200, transition to screen 1300 occurs.

[0234] Screen 1300 includes images 1301 to 1306.

[0235] 12, image 1303 is an image that shows, in a list format, motions to be suggested by suggestion unit 302D from among a plurality of candidate motions. Specifically, image 1303 includes images 1203A to 1203D that show, as suggested motions, the respective rows of sweeping motion, horizontal pulling motion, compacting motion, and broom motion from among a plurality of candidate motions.

[0236] 12, image 1303A is an image that shows, in list form, multiple target trajectories of the bucket 6 for one proposed action (sweeping action). Specifically, image 1203A includes images 1303A1 to 1303A4 that show the respective rows of four target trajectories of the bucket 6 ("sweeping I" to "sweeping IV") for the sweeping action that has the highest reliability (suitability) among multiple candidate actions.

[0237] 11, the image 1304 is an image that represents four target trajectories for one motion of the proposed target, which is represented in the image 1103. Specifically, the image 1304 includes images 1304A to 1304D.

[0238] Images 1304A to 1304D are the same as images 1104A to 1104D in FIG. 11, respectively, and therefore a description thereof will be omitted.

[0239] In this example, a matte cursor is displayed on image 1303A1 corresponding to "sweeping operation I." Also, in this example, image 1304A corresponding to the target trajectory of "sweeping operation I" in image 1304 is displayed with a thicker line than images 1304B to 1304D corresponding to the other target trajectories. This indicates that the target trajectory of "sweeping I" has been selected from the four target trajectories.

[0240] An image 1305 is an icon for executing one proposed operation (sweeping operation) so that the bucket 6 moves along a target trajectory selected by the user (operator) from among a plurality of target trajectories.

[0241] Thus, in this example, the controller 30 displays a plurality of actions among a plurality of candidate actions related to ground leveling work together with their reliability on the display device 50A, and also displays a plurality of target trajectories for the most reliable action on the display device 50A. This allows the controller 30 to propose to the operator a plurality of actions that have a relatively high reliability for the current shape of the work target (terrain shape) around the shovel 100, and a plurality of target trajectories for the bucket 6 for the most reliable action.

[0242] <Example 4> FIG. 14 is a diagram showing a fourth example (screen 1400) of the display content of the display device 50A, relating to the operation suggestion function of the shovel 100.

[0243] The following description will focus on the differences from the first to third examples described above, and the description of the same or corresponding content as the first to third examples described above may be simplified or omitted.

[0244] Screen 1400 includes images 1401 to 1406.

[0245] Image 1401, like image 1001 in FIG. 10, is an image showing a work target around excavator 100.

[0246] The image 1402 is an image that schematically represents the shovel 100, similar to the image 1002 in FIG.

[0247] Image 1403, like image 1003 in Fig. 10, is an image that shows, in a list format, motions to be suggested by suggestion unit 302D from among a plurality of candidate motions. Specifically, image 1403 includes images 1403A to 1403D that show, as suggested motions, the respective rows of sweeping motion, horizontal pulling motion, compacting motion, and broom motion from among a plurality of candidate motions.

[0248] 10, the image 1404 is an image that represents a target trajectory for each proposed motion that is expressed in the image 1403. Specifically, the image 1404 includes images 1404A to 1404D.

[0249] Images 1404A to 1404D are the same as images 1004A to 1004D in FIG. 10, respectively, and therefore a description thereof will be omitted.

[0250] In this example, the worker W is shown in the image area where the image 1403A of the image 1401 is superimposed. Therefore, if the operation with the highest reliability (sweeping operation) is selected, there is a possibility that the attachment AT may get too close to the worker W or that the attachment AT may come into contact with the worker W.

[0251] In contrast, in this example, a matte cursor is displayed in image 1403B corresponding to the horizontal pulling operation, and image 1404B corresponding to the horizontal pulling operation is displayed with a thicker line than images 1404A, 1404C, and 1404D corresponding to the other operations. In other words, in this example, the operator selects an operation (horizontal pulling operation) other than the sweeping operation via input device 52 and is attempting to have shovel 100 execute it. This makes it possible to prevent the attachment AT from getting too close to or coming into contact with worker W.

[0252] Image 1405 is the same as image 1005 in FIG. 10, and therefore a description thereof will be omitted.

[0253] In this way, in this example, the target trajectory of the proposed operation is displayed superimposed on image 1401 showing the state of the periphery of the shovel 100, so that the operator can understand the relationship between the target trajectory and obstacles such as the worker W at the work site. Therefore, the safety of the shovel 100 can be improved while improving the convenience for the operator and the work efficiency of the shovel 100.

[0254] [Example 5] FIG. 15 is a diagram showing a fifth example (screen 1500) of the display content of the display device 50A, relating to the operation suggestion function of the shovel 100.

[0255] The following description will focus on the differences from the first to fourth examples described above, and the description of the same or corresponding content as the first to fourth examples described above may be simplified or omitted.

[0256] Screen 1500 includes images 1501 to 1506.

[0257] Image 1501, like image 1001 in FIG. 10, is an image showing a work target around excavator 100.

[0258] The image 1502 is an image that schematically represents the shovel 100, similar to the image 1002 in FIG.

[0259] Image 1503, like image 1003 in Fig. 10, is an image that shows, in a list format, motions to be suggested by suggestion unit 302D from among a plurality of candidate motions. Specifically, image 1503 includes images 1503A to 1503D that show, as suggested motions, rows of horizontal pulling motion, sweeping motion, compacting motion, and broom motion from among a plurality of candidate motions.

[0260] 10, the image 1504 is an image that represents a target trajectory for each proposed motion that is represented in the image 1503. Specifically, the image 1504 includes images 1504A to 1504D.

[0261] Images 1404A to 1404D are the same as images 1004A to 1004D in FIG. 10, respectively, and therefore a description thereof will be omitted.

[0262] Image 1004A is an image showing the target trajectory of the sweeping operation.

[0263] Image 1004B is an image showing the target trajectory of the horizontal pulling operation.

[0264] Image 1004C is an image showing the target trajectory of the rolling operation.

[0265] Image 1004D is an image showing the target trajectory of the broom motion.

[0266] In this example, the image area in which images 1504A, 1504C, and 1504D of image 1501 are superimposed and displayed is included in area 1501A where ground leveling work has already been completed. Therefore, if the most reliable operation (the leveling operation corresponding to image 1504A) is selected, not only will unnecessary work be performed on the area where ground leveling work has already been completed, but the effects of the unnecessary operation will require work to be undone. As a result, there is a possibility that the work efficiency of excavator 100 will decrease and the progress of work at the work site will be delayed.

[0267] In contrast, in this example, a matte cursor is displayed in image 1503B corresponding to the sweeping operation, and image 1504B corresponding to the sweeping operation is displayed with a thicker line than images 1504A, 1504C, and 1504D corresponding to the other operations. That is, in this example, the operator selects an operation (horizontal pulling operation) other than the sweeping operation and is about to have the shovel 100 execute it. This makes it possible to prevent the shovel 100 from performing work in an area where work has already been completed. Therefore, it is possible to prevent a decrease in the work efficiency of the shovel 100, a delay in the progress of work at the work site, etc.

[0268] Furthermore, in this example, an image indicating that the area 1501A is an area where ground leveling work has already been completed (such as an image that covers the area 1501A with diagonal lines) is superimposed on the area 1501A. This allows the operator to more reliably understand that the area 1501A is an area where ground leveling work has been completed. In this case, information regarding the area at the work site where work has been completed is distributed from the information processing device 200 to the shovel 100, for example.

[0269] Image 1505 is the same as image 1005 in FIG. 10, and therefore a description thereof will be omitted.

[0270] In this way, in this example, the target trajectory of the proposed operation is displayed superimposed on image 1401 showing the state of the periphery of the shovel 100, allowing the operator to understand the relationship between the target trajectory and the completion status of work at the work site, etc. Therefore, the operator can select a more appropriate operation and target trajectory in accordance with the completion status of work at the work site.

[0271] [Other embodiments] Next, another embodiment will be described.

[0272] The above-described embodiments may be combined, modified, or changed as appropriate.

[0273] For example, in the above-described embodiment, the suggestion unit 302D may suggest only the proposed motion among a plurality of candidate motions that can be performed in a predetermined task, and may omit proposing the proposed motion.

[0274] Furthermore, in the above-described embodiment and its modified examples, the suggestion unit 302D may intentionally suggest not only a target trajectory with a relatively high reliability (fit) relative to the shape of the work target around the shovel 100, but also a motion or target trajectory with a relatively low reliability (fit). For example, a motion with a relatively low reliability (fit) is a motion of the shovel 100 or a target trajectory of the bucket 6 estimated based on the learned model LM and with a reliability (fit) lower than a predetermined standard. Furthermore, a motion with a relatively low reliability (fit) may be a motion of the shovel 100 or a target trajectory of the bucket 6 estimated based on a learned model other than the learned model LM, i.e., a learned model that has undergone machine learning using a teacher dataset including inappropriate motions and target trajectories. This reduces the operator's reliance on the motion suggestion function and encourages the operator to use the motion suggestion function based on their own appropriate judgment. In this case, when a motion with a relatively low reliability / fit is selected, this fact may be recorded as a log in the auxiliary storage device 30A of the controller 30, etc. Furthermore, the log may include the operator's identification information. This allows the support device 150 (controller 30) to label, for example, operators who have a relatively low reliability (degree of suitability) and a high probability of selecting the operation of the shovel 100 or the target trajectory of the bucket 6. Therefore, a manager or the like at the work site can manage the usage status of the operation suggestion function for each operator by checking the logs, labels, etc. after the fact.

[0275] Furthermore, in the above-described embodiment and its variations, some or all of the functions of the support device 150 may be transferred to the remote operation support device 300. For example, the function of the proposing unit 302D is transferred to the remote operation support device 300. Furthermore, in addition to the proposing unit 302D, the function of the estimating unit 302C may be transferred to the remote operation support device 300. Furthermore, in addition to the proposing unit 302D and the estimating unit 302C, the function of the work object shape acquiring unit 302B may be transferred to the remote operation support device 300. In this case, image data from the imaging device 40 is transmitted from the excavator 100 to the remote operation support device 300.

[0276] Furthermore, in the above-described embodiment, some or all of the functions of the support device 150 may be transferred to the information processing device 200. For example, the work object shape acquisition unit 302B is transferred to the information processing device 200. In this case, image data from the imaging device 40 is transmitted from the shovel 100 to the information processing device 200. In addition to the work object shape acquisition unit 302B, the function of the estimation unit 302C may be transferred to the information processing device 200. In addition to the work object shape acquisition unit 302B and the estimation unit 302C, the function of the proposal unit 302D may be transferred to the information processing device 200. This allows, for example, a portable information processing device 200 brought into the cabin 10 to propose to the operator one or more movements from among a plurality of candidate movements of the shovel 100 that can be performed in a predetermined task, and to propose the trajectory of those movements.

[0277] Furthermore, the support device 150 in the above-described embodiment and its modified examples may propose to the operator one or more actions from among a plurality of candidate actions for a predetermined task of a work machine other than the shovel 100. For example, the other work machine is a forestry machine having a harvester device. In this case, the support device 150 may propose an action for one or more trees that are the target of an action by the harvester device from among a plurality of candidate actions for trees present at the work site.

[0278] [Effect] Next, the operation of the support device according to this embodiment will be described.

[0279] In this embodiment, the support device includes an acquisition unit and a proposal unit. The support device is, for example, the support device 150. The work machine is, for example, a shovel 100. The acquisition unit is, for example, a work object shape acquisition unit 302B. The proposal unit is, for example, a proposal unit 302D. Specifically, the acquisition unit acquires data related to the shape of the work object around the work machine (for example, the shape of the terrain). Then, the proposal unit proposes to the user an action from among multiple candidate actions of the work machine for a specified task, based on the data related to the shape of the work object around the work machine.

[0280] In this embodiment, the work machine may also be equipped with the above-mentioned assistance device.

[0281] This allows the assistance device to suggest to the operator of the work machine, for example, from among multiple candidate actions that can be performed in a given task, an action that best suits the shape of the work object around the work machine, thereby enabling the work machine to operate more appropriately and improving the work efficiency of the work machine.

[0282] Furthermore, in this embodiment, the assistance device includes an estimation unit. The estimation unit is, for example, estimation unit 302C. Specifically, the estimation unit uses a trained model that has been machine-learned using training data related to the operation of the work machine performed by a relatively skilled operator, associated with the shape of the work object, to estimate an operation that is compatible with the shape of the work object around the work machine from among multiple candidate operations, based on data related to the shape of the work object around the work machine. The trained model is, for example, trained model LM. Then, the proposal unit may propose an operation from among the multiple candidate operations based on the estimation result of the estimation unit.

[0283] This allows the assistance device to use the learned model to propose an action that best suits the shape of the work object around the work machine from among multiple candidate actions that can be performed for a specified task.

[0284] Furthermore, in this embodiment, the suggestion unit may suggest a plurality of actions from among a plurality of candidate actions to the user based on data relating to the shape of the work object in the vicinity of the work machine.

[0285] This allows the support device to provide options to the operator and encourage the operator to make a decision based on his or her own will. Therefore, by reflecting the operator's decision, the work machine can be operated more appropriately.

[0286] Furthermore, in this embodiment, the plurality of proposed actions may include, among the plurality of candidate actions, an action that has a relatively low degree of suitability for the shape of the work object in the vicinity of the work machine.

[0287] This allows the support device to encourage the operator to make decisions based on his or her own will, and therefore allows the work machine to operate more appropriately by reflecting the operator's decisions.

[0288] In addition, in this embodiment, the suggestion unit may suggest an action from among multiple candidate actions based on data regarding the shape of the work object around the work machine, along with the degree of suitability of that action for the shape of the work object around the work machine.

[0289] This allows the support device to provide the operator with information to decide whether or not to perform the operation, thereby encouraging the operator to make a more appropriate decision and allowing the work machine to operate more appropriately.

[0290] In addition, in this embodiment, the suggestion unit may propose multiple actions from multiple candidate actions based on data regarding the shape of the work object around the work machine, along with the degree of suitability of each of the multiple actions for the shape of the work object around the work machine.

[0291] This allows the support device to provide the operator with multiple options regarding the operation of the work machine and to provide information to help make that choice, thereby encouraging the operator to make better decisions and causing the work machine to operate more appropriately.

[0292] In this embodiment, the proposing unit may also propose an action from among a plurality of candidate actions, based on data relating to the shape of the work object around the work machine, along with the trajectory of the work part of the work machine resulting from that action. The work part is, for example, the bucket 6.

[0293] This allows the support device to provide the operator with information to decide whether or not to perform the operation, thereby encouraging the operator to make a more appropriate decision and allowing the work machine to operate more appropriately.

[0294] Furthermore, in this embodiment, the proposing unit may propose an action from among a plurality of candidate actions, together with a plurality of trajectories of the working part resulting from that action, based on data relating to the shape of the work object in the vicinity of the work machine.

[0295] This allows the assistance device to provide the operator with multiple options for the trajectory of the work part corresponding to the proposed motion, thereby encouraging the operator to make better decisions and allowing the work machine to operate more appropriately.

[0296] In addition, in this embodiment, the suggestion unit may propose an action from among multiple candidate actions based on data regarding the shape of the work object around the work machine, along with multiple trajectories of the work part resulting from that action and the degree of fit of each of the multiple trajectories to the shape of the work object around the work machine.

[0297] This allows the support device to provide the operator with multiple options for the work area and information to help them make that selection, thereby encouraging the operator to make better decisions and allowing the work machine to operate more appropriately.

[0298] In this embodiment, the assistance device may also include a display unit. The display unit is, for example, display device 50A. The suggestion unit may then display on the display unit the trajectory of the working part resulting from a proposed action among a plurality of candidate actions, superimposed on an image showing the state of the surroundings of the work machine.

[0299] This allows the operator to more easily understand the relationship between the trajectory of the work portion and the shape of the work object around the work machine, and therefore the assistance device encourages the operator to make better decisions and allows the work machine to operate more appropriately.

[0300] In addition, in this embodiment, the suggestion unit may display on the display unit in different ways the trajectory portion of the trajectory of the proposed action from among multiple candidate actions that comes into contact with the work object and the other trajectory portion.

[0301] This allows the support device to assist the operator in understanding the trajectory more appropriately by enhancing the sense of perspective on the image showing the surroundings of the work machine.

[0302] In this embodiment, the assistance device also includes a control unit. The control unit is, for example, the operation control unit 302E. Specifically, the control unit may cause the work machine to automatically execute the operation suggested by the suggestion unit in response to an instruction input from the user.

[0303] This makes it possible to further improve convenience for the user (operator) and to operate the work machine more appropriately.

[0304] In addition, in this embodiment, the acquisition unit may acquire data regarding the shape of the work object around the work machine by predicting the shape of the work object around the work machine after the operation automatically executed by the control unit is performed.

[0305] This allows the assistance device to repeatedly suggest operations for the work machine and to more smoothly suggest operations for the work machine when assisting the work of the work machine, thereby further improving the work efficiency of the work machine.

[0306] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims. [Explanation of symbols]

[0307] 1 Undercarriage 1C, 1CL, 1CR Crawler 1ML, 1MR Travel Hydraulic Motor 2. Swivel mechanism 2M Swing Hydraulic Motor 3 Upper rotating body 4. Boom 5 Arm 6 buckets 7 Boom cylinder 8 Arm Cylinder 9 Bucket cylinder 10 Cabins 11 Engine 13 Regulator 14 Main pump 15 Pilot pump 17 Control valve 26 Operating device 29 Operating pressure sensor 30 Controllers 31 Hydraulic control valve 32 Shuttle valve 33 Hydraulic control valve 40 Imaging device 50 Output Device 50A display device 50B Sound output device 52 Input Device 60 Communication Equipment 100 Shovel 150 Support equipment 200 Information processing device 300 Remote operation support device 301 Operation log provider 301A Operation log recording section 301B Operation log storage unit 301C Operation log transmission unit 302 Work Support Department 302A Trained model memory unit 302B Work object shape acquisition unit 302C Estimation Department 302D Proposal Department 302E Operation control unit 2001 Operation log acquisition unit 2002 Operation log storage unit 2003 Teacher Data Generation Department 2004 Machine Learning Department 2005 Trained model memory 2006 Distribution Department AT Attachment HA Hydraulic Actuator LM pre-trained model S1~S5 sensors SYS Operation Support System

Claims

1. an acquisition unit that acquires data relating to the shape of a work target in the vicinity of the work machine; a suggestion unit that suggests to a user an action from among a plurality of predefined candidate actions that are different types of candidate actions of the work machine that can be performed in a predetermined task, based on data related to the shape of a work object in the vicinity of the work machine; Support equipment.

2. An acquisition unit that acquires data regarding the shape of a work target around a work machine; a suggestion unit that suggests to a user an action from among a plurality of candidate actions of the work machine in a predetermined task based on data relating to the shape of a work object in the vicinity of the work machine; an estimation unit that estimates an action that matches the shape of the work object in the vicinity of the work machine from among a plurality of candidate actions based on data related to the shape of the work object in the vicinity of the work machine, using a trained model that has been machine-learned using teacher data related to the action of the work machine as operated by a relatively skilled operator, which is associated with the shape of the work object; the proposing unit proposes an action from among the plurality of candidate actions based on the estimation result of the estimating unit. Support equipment.

3. the suggestion unit suggests a plurality of actions from among the plurality of candidate actions to a user based on data relating to the shape of a work object in the vicinity of the work machine. The support device according to claim 1 or 2.

4. The plurality of motions to be proposed include a motion that has a relatively low degree of suitability to the shape of the work object around the work machine among the plurality of candidate motions. The support device according to claim 3 .

5. the suggestion unit suggests an action from among the plurality of candidate actions, based on data relating to the shape of a work object in the vicinity of the work machine, together with a degree of suitability of the action for the shape of the work object in the vicinity of the work machine; 5. An assistance device according to any one of claims 1 to 4.

6. the suggestion unit suggests a plurality of actions from among the plurality of candidate actions, based on data relating to the shape of a work object in the vicinity of the work machine, together with a degree of suitability of each of the plurality of actions for the shape of the work object in the vicinity of the work machine. The support device according to claim 5.

7. the suggestion unit suggests an action from among the plurality of candidate actions, together with a trajectory of a working portion of the work machine resulting from the action, based on data relating to the shape of a work object in the vicinity of the work machine; 7. An assistance device according to any one of claims 1 to 6.

8. the suggestion unit suggests an action from among the plurality of candidate actions, together with a plurality of trajectories of the working part resulting from the action, based on data relating to the shape of a work object in the vicinity of the work machine; The support device according to claim 7.

9. the suggestion unit suggests an action from among the plurality of candidate actions, together with a plurality of trajectories of the work part resulting from the action and a degree of fit of each of the plurality of trajectories to the shape of the work object in the vicinity of the work machine, based on data relating to the shape of the work object in the vicinity of the work machine; The support device according to claim 8.

10. A display unit is provided, the suggestion unit causes the display unit to display a trajectory of the working part resulting from a proposed motion among the plurality of candidate motions, superimposed on an image showing the surroundings of the work machine.

10. An assistance device according to any one of claims 7 to 9.

11. the suggestion unit causes the display unit to display, in different modes, a trajectory portion of a trajectory of a proposed motion from among the plurality of candidate motions that contacts the work object and other trajectory portions. The assistance device according to claim 10.

12. a control unit that causes the work machine to automatically execute the operation proposed by the proposal unit in response to an instruction input from a user, Assistance device according to any one of claims 1 to 11.

13. the acquisition unit acquires data relating to the shape of the work object in the vicinity of the work machine by predicting the shape of the work object in the vicinity of the work machine after the operation automatically executed by the control unit has been performed. Assistance device according to claim 12.

14. A support device according to any one of claims 1 to 13, Work machinery.

15. Support devices include: an acquisition step of acquiring data relating to the shape of a work object around the work machine; a proposing step of proposing to a user an action from among a plurality of predefined candidate actions, which are different types of candidate actions of the work machine that can be performed in a predetermined task, based on data relating to the shape of a work object in the vicinity of the work machine; program.

16. The support device, an acquisition step of acquiring data relating to the shape of a work object around the work machine; a proposing step of proposing to a user an action from among a plurality of candidate actions of the work machine for a predetermined task based on data relating to the shape of a work object around the work machine; an estimation step of estimating an action that matches the shape of the work object around the work machine from among the plurality of candidate actions based on data related to the shape of the work object around the work machine, using a trained model that has been machine-learned using training data related to the action of the work machine as operated by a relatively skilled operator, which is associated with the shape of the work object; In the proposing step, an action is proposed from among the plurality of candidate actions based on the estimation result in the estimating step. program.

Images