Work machine and operation support system

Abstract

The present disclosure provides a work machine capable of setting a restricted area on the basis of an instruction in a natural language. This work machine (shovel 100) comprises an attachment (AT) for work, an information acquisition unit (301), a language conversion unit (303), an instruction acquisition unit (302), and an area setting unit (306). The information acquisition unit (301) acquires information pertaining to the posture of the attachment (AT) and the surrounding environment. The language conversion unit (303) converts the information acquired by the information acquisition unit (301) into a natural language. The instruction acquisition unit (302) acquires an instruction in the natural language from an operator. The area setting unit (306) sets restricted areas (RA1 to RA10) in which entry or operation speed is restricted at a work site. The area setting unit (306) sets the restricted areas (RA1 to RA10) on the basis of a result obtained by interpreting, using a language model (LM), the instruction acquired by the instruction acquisition unit (302) and the information converted into the language by the language conversion unit (303), or on the basis of a result obtained by interpreting the instruction acquired by the instruction acquisition unit (302) using the language model (LM), in combination with the information acquired by the information acquisition unit (301).

Description

Work machines, operation support systems

[0001] The present disclosure relates to a work machine and an operation assistance system.

[0002] 2. Description of the Related Art A display device for a shovel that suitably conveys instructions from an external worker to an operator is known (see Patent Document 1 below).

[0003] The display device for an excavator described in Patent Document 1 includes a communication unit, a display unit, a voice acquisition unit that acquires voices in the cabin, and a voice-to-text conversion unit that converts the voices acquired by the voice acquisition unit into text information. The display unit displays, in chronological order, transmission information acquired by the communication unit and transmitted from an external terminal and the text information converted by the voice-to-text conversion unit.

[0004] With this configuration, the display device of the shovel in Patent Document 1 can display external instructions acquired by the communication unit as text information on the display unit in chronological order. Furthermore, when the shovel operator responds to external instructions by voice, the response is converted into text information, displayed on the display device, and transmitted to the outside, making it easy to respond.

[0005] International Publication No. 2020 / 080501

[0006] However, with the technology described in Patent Document 1, the operator of the shovel must manually set the restricted area in which the entry and operating speed of the shovel are restricted.

[0007] The present disclosure provides a work machine and an operation assistance system that are capable of setting a restricted area based on instructions in natural language.

[0008] One aspect of the present disclosure provides a work machine comprising: a work attachment; an information acquisition unit that acquires information about the posture of the attachment and the surrounding environment; a languageization unit that verbalizes the information acquired by the information acquisition unit in natural language; an instruction acquisition unit that acquires instructions in natural language from an operator; and an area setting unit that sets a restricted area at a work site where entry or movement speed is restricted, wherein the area setting unit sets the restricted area based on the result of interpreting the instructions acquired by the instruction acquisition unit and the information verbalized by the languageization unit using a language model, or based on the result of interpreting the instructions acquired by the instruction acquisition unit using the language model and the information acquired by the information acquisition unit.

[0009] Another aspect of the present disclosure provides an operation assistance system comprising an information acquisition unit that acquires information related to the posture of an attachment of a work machine and the environment around the work machine; a languageization unit that verbalizes the information acquired by the information acquisition unit in natural language; an instruction acquisition unit that acquires instructions in natural language from an operator of the work machine; and an area setting unit that sets a restricted area in which the intrusion or operating speed of the work machine is restricted, wherein the area setting unit sets the restricted area based on the result of interpreting the instructions acquired by the instruction acquisition unit and the information verbalized by the languageization unit using a language model, or based on the result of interpreting the instructions acquired by the instruction acquisition unit using the language model and the information acquired by the information acquisition unit.

[0010] According to the above aspects of the present disclosure, it is possible to provide a work machine and an operation assistance system that are capable of setting a restricted area based on instructions in natural language.

[0011] FIG. 1 is a side view showing an example of a shovel. FIG. 2 is a top view showing an example of a shovel. FIG. 3 is a diagram showing an example of an operation support system. FIG. 4 is a diagram showing an example of the hardware configuration of a shovel. FIG. 5 is a diagram showing an example of the hardware configuration of a remote operation support device. FIG. 6 is a functional block diagram showing an example of a restricted area setting function of a shovel. FIG. 7 is a diagram showing an example of the environment around a shovel. FIG. 8 is a diagram showing an example of an example question assigned to a language model. FIG. 9 is a diagram showing an example of a combination of prompts and outputs of a language model. FIG. 10 is a diagram showing another example of the environment around a shovel. FIG. 11 is a diagram showing an example of a display of a restricted area of ​​a shovel. FIG. 12 is a flow diagram showing an example of processing related to setting a restricted area of ​​a shovel.

[0012] Hereinafter, embodiments of a work machine and an operation assistance system according to the present disclosure will be described with reference to the drawings. The embodiments described below are examples and do not limit the invention. Not all features and combinations thereof in the embodiments of the present disclosure are necessarily essential to the invention. Note that identical or corresponding reference numerals are used in the drawings to designate identical or corresponding components, and redundant description may be omitted.

[0013] Fig. 1 is a side view showing a shovel 100, which is an example of an embodiment of a work machine according to the present disclosure. Fig. 2 is a top view of the shovel 100 shown in Fig. 1. Fig. 3 is a configuration diagram showing an example of an embodiment of an operation assistance system according to the present disclosure. Hereinafter, the direction in which the attachment AT extends in a top view of the shovel 100 (the upward direction in Fig. 2) may be defined as "front," and directions on the shovel 100 or directions seen from the shovel 100 may be described.

[0014] As shown in FIGS. 1 and 2 , the excavator 100 includes a lower traveling body 1 , an upper rotating body 3 , an attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cabin 10 .

[0015] The undercarriage 1 uses crawlers 1C to travel the excavator 100. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CR is hydraulically driven by a traveling hydraulic motor 1MR. This allows the undercarriage 1 to travel independently.

[0016] The upper rotating body 3 is mounted on the lower traveling body 1 so as to be rotatable (freely rotatable) via the rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by a rotating hydraulic motor 2M.

[0017] The boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down about a rotation axis that extends in the left-right direction. The arm 5 is attached to the tip of the boom 4 so as to be able to rotate about a rotation axis that also extends in the left-right direction. The bucket 6 is attached to the tip of the arm 5 so as to be able to rotate about a rotation axis that also extends in the left-right direction.

[0018] The bucket 6 is an example of an end attachment, and is used for, for example, excavation work, slope work, ground leveling work, and the like.

[0019] The bucket 6 is attached to the tip of the arm 5 in a manner that allows it to be appropriately replaced depending on the work content of the shovel 100. In other words, instead of the bucket 6, a bucket of a different type from the bucket 6, such as a relatively large bucket, a slope bucket, or a dredging bucket, may be attached to the tip of the arm 5. Also, a type of end attachment other than a bucket, such as a mixer, breaker, or crusher, may be attached to the tip of the arm 5. Also, a spare attachment such as a quick coupling or a tiltrotator may be provided between the arm 5 and the end attachment.

[0020] The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0021] The cabin 10 is a control room where an operator sits and operates the excavator 100. The cabin 10 is mounted on the front left side of the upper rotating body 3, for example.

[0022] The excavator 100 may be equipped with a communication device 60 and may be capable of communicating with the remote operation support device 200 via a predetermined communication line NW.

[0023] The communication line NW may include, for example, a local area network (LAN) at a work site. The communication line NW may also include a wide area network (WAN). Examples of wide area networks include a mobile communication network terminated at a base station, a satellite communication network using a communication satellite, and the Internet. The communication line NW may also include, for example, a short-distance communication line based on a wireless communication standard such as Wi-Fi or Bluetooth (registered trademark).

[0024] For example, the excavator 100 operates driven elements such as the lower running body 1 (i.e., a pair of left and right crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6 in response to operations by an operator seated in the cabin 10.

[0025] Furthermore, instead of or in addition to being configured to be operable by an operator inside the cabin 10, the shovel 100 may be configured to be remotely operable from outside the shovel 100. When the shovel 100 is remotely operated, the interior of the cabin 10 may be unmanned. Furthermore, when the shovel 100 is exclusively for remote operation, the cabin 10 may be omitted. The following description will be given on the assumption that the operation of the operator includes at least one of operation of the operating device 26 by the operator inside the cabin 10 and remote operation by an external operator.

[0026] 3, for example, remote operation utilizes an operation support system SYS in which an excavator 100 to be operated and a remote operation support device 200 for an operator to operate the excavator 100 are communicably connected via a communication line NW. Specifically, remote operation includes a mode in which the excavator 100 is operated by operation input related to the actuator of the excavator 100 performed by the remote operation support device 200 that can communicate with the excavator 100 via the communication line NW.

[0027] The remote operation support device 200 is provided, for example, in a management center or the like that externally manages the work of the shovel 100. The remote operation support device 200 may also be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the work status of the shovel 100 from the vicinity of the shovel 100.

[0028] The shovel 100 transmits, for example, via the communication device 60, to the remote operation support device 200 an image (hereinafter referred to as a "peripheral image") showing the surroundings including the area in front of the shovel 100, based on an image captured by an imaging device 40 mounted on the shovel 100. The shovel 100 may also transmit the captured image output by the imaging device 40 to the remote operation support device 200 via the communication device 60, and the remote operation support device 200 may process the captured image received from the shovel 100 to generate a peripheral image. The remote operation support device 200 then displays the peripheral image showing the surroundings including the area in front of the shovel 100 on its own display device. Various information images (information screens) displayed on the output device 50 (display device) installed inside the cabin 10 of the shovel 100 may also be displayed on the display device of the remote operation support device 200. This allows the operator using the remote operation support device 200 to remotely operate the shovel 100 while checking, for example, the contents of an image, information screen, or the like showing the surroundings of the shovel 100 displayed on the display device. The excavator 100 then operates the actuators in response to signals indicating the content of the remote operation received from the remote operation support device 200 via the communication device 60, thereby operating driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6. In this way, the operation support system SYS can realize remote operation of the excavator 100 using the remote operation support device 200.

[0029] Furthermore, remote control may include, for example, a mode in which the shovel 100 is operated by an external voice input, gesture input, or the like to the shovel 100 by a person (e.g., a worker) present around the shovel 100. Specifically, the shovel 100 recognizes voices uttered by surrounding workers, gestures made by the workers, or the like, through a voice input device (e.g., a microphone) or a gesture input device (e.g., an imaging device) mounted on the shovel 100. Then, the shovel 100 may operate actuators in accordance with the content of the recognized voices, gestures, or the like, to drive driven elements such as the lower traveling body 1 (left and right crawlers 1C), upper rotating body 3, boom 4, arm 5, and bucket 6.

[0030] Furthermore, the excavator 100 may automatically operate the actuators regardless of the operation by the operator. This allows the excavator 100 to realize a function of automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, i.e., a so-called "automatic driving function" or "machine control (MC) function."

[0031] The automatic driving function includes, for example, a semi-automatic driving function (operation assistance type MC function). The semi-automatic driving function is a function that automatically operates driven elements (actuators) other than the driven element (actuator) to be operated in response to an operator's operation. The automatic driving function may also include a fully automatic driving function (fully automatic MC function). The fully automatic driving function is a function that automatically operates at least some of the multiple driven elements (actuators) without operator operation. When the fully automatic driving function is enabled in the excavator 100, the interior of the cabin 10 may be unmanned. The semi-automatic driving function, the fully automatic driving function, etc. also include, for example, a rule-based automatic driving function. The rule-based automatic driving function is an automatic driving function in which the operation content of the driven element (actuator) to be operated automatically is determined according to pre-defined rules. The semi-automatic driving function, the fully automatic driving function, etc. may also include an autonomous driving function. The autonomous driving function is an automatic driving function in which the excavator 100 autonomously makes various decisions and determines the operation content of the driven element (actuator) that is the target of automatic driving based on the results of those decisions.

[0032] Furthermore, the work of the shovel 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 200 may be provided. The remote monitoring support device is, for example, the remote operation support device 200. This allows a monitor, who is a user of the remote monitoring support device, to monitor the status of the work of the shovel 100 while checking a peripheral image displayed on a display device of the remote monitoring support device. Furthermore, for example, if the monitor determines it is necessary from a safety standpoint, the monitor can intervene in the operation by the operator or automatic operation of the shovel 100 and bring the shovel 100 to an emergency stop by making a predetermined input using an input device of the remote monitoring support device.

[0033] Next, the configuration of the shovel 100 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of the shovel 100. In Fig. 4, paths through which mechanical power is transmitted are indicated by double lines, paths through which high-pressure hydraulic oil that drives the hydraulic actuator HA flows are indicated by solid lines, paths through which pilot pressure is transmitted are indicated by dashed lines, and paths through which electrical signals are transmitted are indicated by dotted lines.

[0034] The shovel 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0035] 4 , the hydraulic drive system of the excavator 100 includes hydraulic actuators HA that hydraulically drive each of the driven elements, such as the lower traveling structure 1 (left and right crawlers 1CL, 1CR), upper rotating structure 3, boom 4, arm 5, and bucket 6, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0036] The hydraulic actuators HA include traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9. Note that in the excavator 100, some or all of the hydraulic actuators HA may be replaced with electric actuators. In other words, the excavator 100 may be a hybrid excavator or an electric excavator.

[0037] The engine 11 is a prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 is mounted, for example, on the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target rotation speed under direct or indirect control by a controller 30 (described later), for example, and drives the main pump 14 and the pilot pump 15. Note that instead of or in addition to the engine 11, another prime mover (for example, an electric motor) may be mounted on the excavator 100.

[0038] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

[0039] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the regulator 13 adjusts the tilt angle of the swash plate of the main pump 14, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0040] The control valve 17 drives the hydraulic actuators HA in response to an operator's operation of the operating device 26, the details of remote operation, or an operation command corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (directional control valves) that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0041] As shown in FIG. 4 , the operating system of the excavator 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

[0042] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above. The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic devices as pilot pressure.

[0043] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements, thereby enabling the operator to operate the driven elements that are the targets of the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0044] For example, as shown in FIG. 4 , the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32 and is connected to the control valve 17 via a pilot line 27 connected to the outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0045] Alternatively, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0046] Furthermore, the control valves (directional control valves) that are built into the control valve 17 and drive the hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17 (i.e., to the electromagnetic solenoid type control valve).

[0047] As described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output a control command to the electric actuators or a driver that drives the electric actuators, depending on the operation content of the operation device 26 and the remote operation content specified by the remote operation signal. Furthermore, when the excavator 100 is remotely operated, the operation device 26 may be omitted.

[0048] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via a shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, for example, the controller 30 can supply pilot pressure according to an operation command corresponding to the automatic driving function from the hydraulic control valve 31 to the control valve 17, thereby realizing operation of the excavator 100 using the automatic driving function.

[0049] Furthermore, the controller 30 may control the hydraulic control valve 31 to realize remote operation of the shovel 100. Specifically, the controller 30 outputs, via the communication device 60, a control signal corresponding to the content of remote operation specified in a remote operation signal received from the remote operation assistance device 200 to the hydraulic control valve 31. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of remote operation to the control valve 17, thereby realizing operation of the shovel 100 based on remote operation by the operator.

[0050] In addition, if the operating device 26 is electric, the controller 30 can supply pilot pressure corresponding to the operation content (operation signal) of the operating device 26 directly to the control valve 17 from the hydraulic control valve 31, thereby realizing operation of the shovel 100 based on the operation of the operator.

[0051] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having the higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, a shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to a secondary pilot line 27A of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives a hydraulic actuator HA that is the target of operation of the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6) regardless of the operating state of the operating device 26 by the operator, thereby realizing an automatic driving function or a remote control function.

[0052] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the automatic operation function and remote control function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0053] As shown in FIG. 4 , the user interface system of the excavator 100 includes an operation device 26 , an output device 50 , and an input device 52 .

[0054] The output device 50 outputs various information to the user of the shovel 100 (e.g., the operator of the cabin 10 or an external remote operator) and people in the vicinity of the shovel 100 (e.g., workers or drivers of work vehicles).

[0055] For example, the output device 50 includes lighting equipment, a display device, etc. that output various types of information visually. The lighting equipment is, for example, a warning light (indicator lamp), etc. The display device is, for example, a liquid crystal display, an organic EL (electroluminescence) display, etc. For example, as shown in FIG. 2 , the lighting equipment and the display device may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting equipment and the display device may be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the excavator 100.

[0056] The output device 50 may also include a sound output device that outputs various types of information auditorily. Examples of sound output devices include a buzzer and a speaker. The sound output device may be provided, for example, inside or outside the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the excavator 100. The output device 50 may also include a device that outputs various types of information tactilely, such as by vibrating the operator's seat.

[0057] The input device 52 accepts various inputs from the user of the excavator 100. A signal corresponding to the input accepted by the input device 52 is taken into the controller 30. For example, as shown in FIG. 2 , the input device 52 is provided inside the cabin 10 and accepts inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3 and accept inputs from a worker or the like in the vicinity of the excavator 100.

[0058] For example, the input device 52 includes a mechanical input device that accepts input from a user through mechanical operation. The mechanical input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0059] The input device 52 may also include a voice input device that accepts voice input from the user. The voice input device may include, for example, a microphone. The input device 52 may also include a gesture input device that accepts gesture input from the user. The gesture input device may include, for example, an imaging device that captures images of gestures made by the user. The input device 52 may also include a biometric input device that accepts biometric input from the user. The biometric input may include, for example, input of biometric information such as the user's fingerprint or iris.

[0060] As shown in FIG. 4 , the communication system of the shovel 100 according to this embodiment includes a communication device 60 .

[0061] The communication device 60 connects to an external communication line NW and communicates with devices provided separately from the shovel 100. The devices provided separately from the shovel 100 may include devices external to the shovel 100, as well as a portable terminal device (mobile terminal) brought into the cabin 10 by the user of the shovel 100. The communication device 60 may include, for example, a mobile communication module conforming to standards such as 4G (4th Generation) and 5G (5th Generation). The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module. When there are multiple connectable communication lines NW, the communication device 60 may include multiple communication devices according to the types of the communication lines NW.

[0062] For example, the communication device 60 communicates with the remote operation support device 200 within the work site and external devices such as the remote operation support device 200 via a local communication line established at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line established at the work site or a local network using Wi-Fi 6. The communication device 60 may also communicate with the remote operation support device 200 outside the work site and external devices such as the remote operation support device 200 via a wide-area communication line including the work site, i.e., a wide-area network.

[0063] 4, the control system of the shovel 100 includes a controller 30. The control system of the shovel 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S9.

[0064] The controller 30 performs various controls related to the shovel 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, as shown in FIG. 4 , the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus BS1.

[0065] The auxiliary storage device 30A is a non-volatile storage means that stores the installed program as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory. For example, when a program startup instruction is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0066] The CPU 30C executes, for example, a program loaded into the memory device 30B and, in accordance with the instructions of the program, realizes various functions of the controller 30. The interface device 30D functions, for example, as a communication interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of communication interfaces in accordance with the types of communication lines to be connected.

[0067] The interface device 30D also functions as an external interface for reading data from a recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool connected to a connector installed inside the cabin 10 via a detachable cable. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may also be downloaded from another computer external to the excavator 100 via the communication device 60 and installed in the auxiliary storage device 30A.

[0068] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the shovel 100.

[0069] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure by the operating pressure sensor 29, which corresponds to the operating state of each driven element (hydraulic actuator HA) in the operating device 26, is input to the controller 30.

[0070] It should be noted that if the operating device 26 is electric or if the operating device 26 is omitted, the operating pressure sensor 29 is omitted. In these cases, the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operation signal received from the operating device 26.

[0071] The imaging device 40 captures an image of the surroundings of the shovel 100. The imaging device 40 is, for example, a monocular camera. The imaging device 40 may also be a three-dimensional camera (3D camera) that can acquire not only two-dimensional image information but also three-dimensional information including information about the distance to an object shown in the image and the depth of the image, such as a stereo camera, a ToF (Time of Flight) camera, or a depth camera.

[0072] 2, the imaging device 40 includes cameras 40F, 40B, 40L, and 40R. Camera 40F images the area in front of the upper revolving body 3. Camera 40B images the area behind the upper revolving body 3. Camera 40L images the area to the left of the upper revolving body 3. Camera 40R images the area to the right of the upper revolving body 3. This allows the imaging device 40 to capture an image of the entire circumference of the excavator 100, i.e., a range spanning a 360-degree angular direction, when viewed from above the excavator 100. Hereinafter, cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X."

[0073] The output data of the imaging device 40 (camera 40X) is taken into the controller 30 via a one-to-one communication line or an in-vehicle network. This allows the controller 30 to grasp the situation around the shovel 100, for example, based on the output data of the camera 40X. Note that some or all of the cameras 40B, 40L, and 40R may be omitted. Furthermore, the shovel 100 may be provided with a ranging sensor (also referred to as a "distance sensor") capable of acquiring information indicating the distance to an object in the shovel 100's vicinity, instead of or in addition to the imaging device 40. The ranging sensor is, for example, a LiDAR (Light Detecting and Ranging), a millimeter-wave radar, an ultrasonic sensor, or the like.

[0074] The sensor S1 is attached to the boom 4 and measures the attitude of the boom 4. The sensor S1 outputs measurement data representing the attitude of the boom 4. The attitude of the boom 4 is, for example, the attitude angle around the rotation axis of the base end of the boom 4, which corresponds to the connection point between the boom 4 and the upper rotating body 3 (hereinafter referred to as the "boom angle"). The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may apply to the sensors S2 to S4 below. The sensor S1 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may apply to the sensors S2 and S3 below. The output of the sensor S1 (measurement data representing the attitude of the boom 4) is input to the controller 30. This allows the controller 30 to grasp the attitude of the boom 4.

[0075] The sensor S2 is attached to the arm 5 and measures the posture of the arm 5. The sensor S2 outputs measurement data representing the posture of the arm 5. The posture of the arm 5 is, for example, the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The output of the sensor S2 (measurement data representing the posture of the arm 5) is input to the controller 30. This enables the controller 30 to grasp the posture of the arm 5.

[0076] The sensor S3 is attached to the bucket 6 and measures the attitude of the bucket 6. The sensor S3 outputs measurement data that indicates the attitude of the bucket 6. The attitude of the bucket 6 is, for example, the attitude angle (hereinafter referred to as the "bucket angle") around the rotation axis of the base end of the bucket 6 that corresponds to the connection part with the arm 5. The output of the sensor S3 (measurement data that indicates the attitude of the bucket 6) is input to the controller 30. This enables the controller 30 to grasp the attitude of the bucket 6.

[0077] The sensor S4 measures the attitude state of the machine body (e.g., the upper rotating body 3) of the shovel 100. The sensor S4 outputs measurement data representing the attitude state of the machine body of the shovel 100. The attitude state of the machine body of the shovel 100 is, for example, the inclination state of the machine body with respect to a predetermined reference plane (e.g., a horizontal plane). For example, the sensor S4 is attached to the upper rotating body 3 and measures the inclination angles of the shovel 100 about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft inclination angle" and the "lateral inclination angle"). The output of the sensor S4 (measurement data representing the attitude state of the machine body of the shovel 100) is taken into the controller 30. This allows the controller 30 to grasp the attitude state (inclination state) of the machine body (upper rotating body 3).

[0078] The sensor S5 is attached to the upper rotating body 3 and measures the rotation state of the upper rotating body 3. The sensor S5 outputs measurement data representing the rotation state of the upper rotating body 3. The sensor S5 measures, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the sensor S5 (measurement data representing the rotation state of the upper rotating body 3) is input to the controller 30. This allows the controller 30 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0079] The controller 30 can grasp (estimate) the position of the tip of the attachment AT (bucket 6) based on the outputs of the sensors S1 to S5. If the sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocity around three axes, the rotation state of the upper rotating body 3 (for example, the rotation angular velocity) may be detected based on the detection signal of the sensor S4. In this case, the sensor S5 may be omitted.

[0080] The sensor S6 measures the position of the shovel 100. The sensor S6 may measure the position in world (global) coordinates, or may measure the position in local coordinates at the work site. In the former case, the sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, the sensor S6 is a transceiver that communicates with equipment that serves as a reference for the position at the work site and is capable of outputting a signal corresponding to the position relative to the reference. The output of the sensor S6 is taken into the controller 30.

[0081] Sensor S7 measures the pressure (cylinder pressure) in the oil chamber of boom cylinder 7. Sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of boom cylinder 7, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side. The output of sensor S7 (i.e., measurement data of the cylinder pressure of boom cylinder 7) is taken into controller 30.

[0082] The sensor S8 measures the pressure (cylinder pressure) in the oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of the arm cylinder 8, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side of the arm cylinder 8. The output of the sensor S8 (i.e., the measurement data of the cylinder pressure of the arm cylinder 8) is taken into the controller 30.

[0083] The sensor S9 measures the pressure (cylinder pressure) of the oil chamber of the bucket cylinder 9. The sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of the bucket cylinder 9, and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side of the bucket cylinder 9. The output of the sensor S9 (i.e., the measurement data of the cylinder pressure of the bucket cylinder 9) is taken into the controller 30.

[0084] The controller 30 can grasp the load state acting on the attachment AT based on the outputs of the sensors S7 to S9. The load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from the soil on the ground being worked on, the weight of the soil contained in the bucket 6, and the like. Note that some or all of the sensors S1 to S9 may be omitted depending on necessity. The shovel 100 may be equipped with other sensors capable of grasping the state of the shovel 100. For example, the shovel 100 may be equipped with a direction sensor capable of detecting its own direction. The direction sensor is, for example, an electronic compass including a geomagnetic sensor.

[0085] Next, the configuration of the remote operation support device 200 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of the remote operation support device 200.

[0086] The functions of the remote operation support device 200 are realized by any hardware or a combination of any hardware and software. For example, as shown in Fig. 5, the remote operation support device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed calculation device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. These are connected by a bus BS2.

[0087] The external interface 201 functions as an interface for reading data from and writing data to the recording medium 201 A. The recording medium 201 A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, and the like.

[0088] This allows the remote operation assistance device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions. Note that the remote operation assistance device 200 may also obtain various data and programs used in processing from an external device through the communication interface 206.

[0089] The auxiliary storage device 202 stores various installed programs as well as files and data necessary for various processes. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state disk (SSD), and a flash memory. When a program startup instruction is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a dynamic random access memory (DRAM) or an SRAM.

[0090] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the remote operation support device 200 in accordance with the programs. The high-speed calculation device 205 works in conjunction with the CPU 204 to perform calculation processing at a relatively high speed. The high-speed calculation device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. Note that the high-speed calculation device 205 may be omitted depending on the required calculation processing speed.

[0091] The communication interface 206 is used as an interface for communicatively connecting with an external device. This allows the remote operation assistance device 200 to communicate with an external device such as the excavator 100 through the communication interface 206. The communication interface 206 may have multiple types of communication interfaces depending on the communication method between the device and the remote operation assistance device.

[0092] The input device 207 accepts various inputs from the user. The input device 207 includes a remote operation operation device for remotely operating the shovel 100. The input device 207 includes, for example, an input device (mechanical input device) that accepts mechanical operation input from the user. The remote operation operation device may be a mechanical input device. The mechanical input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0093] The input device 207 may also include a voice input device capable of accepting voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice. The input device 207 may also include a gesture input device capable of accepting gesture input from the user. The gesture input device includes, for example, a camera capable of capturing images of the user's gestures. The input device 207 may also include a biometric input device capable of accepting biometric input from the user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about the user's fingerprint or iris.

[0094] The display device 208 displays an information screen or an operation screen for the user of the remote operation support device 200. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. The sound output device 209 conveys various pieces of information to the user of the remote operation support device 200 by sound. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, etc.

[0095] Next, the restricted area setting function of the excavator 100 will be described with reference to FIGS. 6 to 11 in addition to FIGS. 1 to 5.

[0096] Fig. 6 is a functional block diagram showing an example of a restricted area setting function of the shovel 100. Fig. 7 is a diagram showing an example of the environment around the shovel 100. Fig. 8 is a diagram showing an example of an example question assigned to the language model LM. Fig. 9 is a diagram showing an example of a combination of prompts and outputs of the language model. Fig. 10 is a diagram showing another example of the environment around the shovel 100. Fig. 11 is a diagram showing an example of a display of the restricted area RA of the shovel 100.

[0097] 6, the controller 30 of the shovel 100 has, for example, an information acquisition unit 301, an instruction acquisition unit 302, a language generation unit 303, a prompt generation unit 304, a calling unit 305, an area setting unit 306, an operation control unit 307, and a display control unit 308. These units of the controller 30 represent the functions of the controller 30 that are realized, for example, by a program installed in an auxiliary storage device 30A being loaded into a memory device 30B and executed by a CPU 30C. In addition, a language model LM is provided outside the shovel 100.

[0098] The language model LM is, for example, a large language model (LLM). The language model LM is implemented in an external device (for example, a server device) that is communicably connected to the excavator 100 via the communication device 60. The language model LM is, for example, GPT-4.

[0099] The information acquisition unit 301 acquires, for example, the detection results of the sensors S1 to S9. The detection results of the sensors S1 to S9 include information about the posture of the attachment AT and the environment around the excavator 100. Specifically, the information acquisition unit 301 acquires, as information about the posture of the attachment AT, the detection results of the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, and the rotation sensor S5, for example.

[0100] Furthermore, the information acquisition unit 301 acquires, for example, from the imaging device 40 or a distance measurement sensor, detection results of objects present around the shovel 100 as information about the environment around the shovel 100. Furthermore, the information acquisition unit 301 may detect objects around the shovel 100 by, for example, acquiring image data from the imaging device 40 or acquiring the distance and direction to the objects around the shovel 100 from a distance measurement sensor.

[0101] Specifically, the information acquisition unit 301 detects a monitored object from an image captured by the imaging device 40 by arbitrarily applying known image processing techniques such as semantic segmentation or machine learning. The monitored object includes, for example, a person such as a worker. The monitored object may also include other obstacles present around the shovel 100. Examples of the other obstacles include specific moving objects present at the work site of the shovel 100, such as other work machines or work vehicles. Examples of the other obstacles may also include specific stationary objects present at the work site of the shovel 100, such as utility poles, fences, and traffic cones (also referred to as traffic cones (registered trademark)). Examples of the other obstacles may also include specific topographical features present at the work site of the shovel 100, such as ditches, holes, and piles of earth and sand.

[0102] Furthermore, the information acquisition unit 301 may acquire, for example, information related to a construction drawing of a work site of the shovel 100, as information related to the environment around the shovel 100. The information related to the construction drawing includes, for example, information related to the shape, dimensions, and position of objects to be installed at the work site, such as the road RD shown in FIG. 7 and the buried pipe UP, electric wire EW, and slope shoulder TS shown in FIG. 10. The information related to the construction drawing is acquired by the information acquisition unit 301 via the communication interface 206, for example, and stored in the auxiliary storage device 202.

[0103] The information acquisition unit 301 outputs the acquired information to, for example, the language generation unit 303 and the area setting unit 306 .

[0104] The instruction acquisition unit 302 acquires instructions (hereinafter simply referred to as "instructions") regarding the setting of the restricted area RA, which are input in natural language by the operator. The restricted area RA is an area in the work site where the entry or operating speed of the excavator 100 is restricted. Specifically, in the examples shown in FIGS. 7 and 10, the restricted area RA includes restricted areas RA1 to RA10 that are defined by virtual safety barriers SB1 to SB10 as boundaries. The safety barriers SB1 to SB10 are also referred to as "E-FENCE (registered trademark)," "virtual fences," or "safety fences."

[0105] Instructions input from the operator are accepted through the input device 52 installed in the cabin 10 in the case of an operator in the cabin 10, and are accepted through the input device 207 of the remote operation support device 200 in the case of an operator in a remote operation.

[0106] An instruction in natural language is, for example, an instruction input in natural language by voice from an operator. In this case, the instruction acquisition unit 302 can acquire text (sentence) data corresponding to the instruction in natural language by applying a known voice recognition technology based on the voice input data received by the input device 52 or the input device 207.

[0107] Furthermore, the instruction in natural language may be an instruction input as text by an operator using the input device 52 or the input device 207, which is capable of inputting characters, such as a keyboard or a touch panel. In this case, the instruction acquisition unit 302 can acquire text (sentence) data received by the input device 52 or the input device 207 as an instruction in natural language.

[0108] The verbalization unit 303 verbalizes, in natural language, the information relating to the posture of the attachment AT and the environment around the shovel 100 input from the information acquisition unit 301. The information relating to the environment around the shovel 100 verbalized by the verbalization unit 303 includes, for example, information relating to the construction drawings of the work site of the shovel 100, as described above.

[0109] The verbalization unit 303 verbalizes, in natural language, for example, the boom angle, arm angle, and bucket angle, the location of objects to be monitored around the shovel 100, and information on construction drawings acquired by the information acquisition unit 301. Specifically, the verbalization unit 303 verbalizes, for example, the information acquired by the information acquisition unit 301 by applying it to a predefined text template.

[0110] A text template representing the posture of the attachment AT is specified in a format such as "The boom angle is aaa degrees, the arm angle is bbb degrees, and the bucket angle is ccc degrees," where "aaa" is the boom angle, "bbb" is the arm angle, and "ccc" is the bucket angle. A text template representing the location of an object to be monitored is specified in a format such as "There is an eee at ddd," or "There is an eee at ddd," where "ddd" is the position information of the object to be monitored and "eee" is the type or name of the object to be monitored. The same applies to information on construction drawings. The position information of an object is, for example, a direction and distance relative to the excavator 100, or coordinates.

[0111] 7, for example, the verbalization unit 303 verbalizes, for a person P detected by the imaging device 40, "There are people at an azimuth angle of 285 degrees, a distance of 5 meters, and at an azimuth angle of 350 degrees, a distance of 6 meters," using an azimuth angle of 0° in front of the shovel 100. Also, for example, as shown in FIG. 10, for a buried pipe UP shown in a construction drawing, the verbalization unit 303 verbalizes, "There is a buried pipe 5 meters to the left, 3 meters underground."

[0112] The prompt generation unit 304 generates a prompt to be input to the language model LM based on the instruction in natural language acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303. Specifically, the prompt generation unit 304 generates a prompt for setting a restriction area RA corresponding to the instruction acquired by the instruction acquisition unit 302, based on the information on the posture of the attachment AT of the shovel 100 and the surrounding environment verbalized by the verbalization unit 303.

[0113] For example, the prompt generation unit 304 generates a plurality of example problems and assigns them in advance to the language model LM via the calling unit 305. An example problem is defined by a combination of information about the posture of the attachment AT of the shovel 100 and the surrounding environment, which are prerequisites (i.e., constraints), instructions for the example problem, and a correct answer to be output. This allows the language model LM to understand (learn) the output format for the prompt instructions.

[0114] For example, as shown in FIG. 8 , a plurality of example problems represented by combinations of the environment around the shovel 100, instructions from the user (i.e., the operator), and control information (control commands) for the shovel 100 are given. As a result, as shown in FIG. 9 , a control command for generating a virtual safety barrier SB and setting a restricted area RA can be output from the language model LM in response to a prompt generated by the prompt generation unit 304, i.e., an instruction from the operator based on the environment around the shovel 100. Furthermore, by instructing the user to ask again if the instruction cannot be understood, it is possible to prevent a situation in which an inappropriate area is set as the restricted area RA. The number of example problems given is arbitrary, but it is preferable that the number be greater than the types of control commands corresponding to the operations to be performed by the shovel 100. This is because the language model LM can understand all control commands.

[0115] The calling unit 305 calls the language model LM via, for example, a predetermined API (Application Programming Interface), inputs the prompt generated by the prompt generating unit 304 to the language model LM, and obtains the output (answer).

[0116] The area setting unit 306 sets the restricted area RA based on the output of the language model LM acquired by the calling unit 305. Specifically, the area setting unit 306 generates a virtual safety barrier SB in accordance with the control command output from the language model LM to set the restricted area RA.

[0117] 7 , the information acquisition unit 301 acquires information about the environment around the excavator 100, such as a plurality of cones CN to the front right, a road RD behind, a plurality of people P to the front left, and the direction and distance of a dump truck DT to the right. The verbalization unit 303 verbalizes the information acquired by the information acquisition unit 301 and inputs the verbalized information to the prompt generation unit 304.

[0118] Furthermore, the instruction acquisition unit 302 acquires instructions in natural language from the user of the shovel 100 and inputs them to the prompt generation unit 304. The prompt generation unit 304 inputs a prompt generated based on the input information and instructions to the calling unit 305. The calling unit 305 inputs the prompt generated by the prompt generation unit 304 to the language model LM, and obtains a control command from the language model LM as an output for the area setting unit 306 to set a restricted area RA.

[0119] As a result, in response to a user instruction such as "Set a barrier in the cone area," the area setting unit 306 generates a safety barrier SB1 using a convex hull based on the positions of multiple cones CN, as shown in Fig. 7. The area setting unit 306 also sets the area surrounded by the safety barrier SB1 as a restricted area RA1.

[0120] Furthermore, in response to a user instruction such as "Set up a barrier based on the two front cones," the area setting unit 306 generates, as the safety barrier SB2, a vertical plane parallel to the line connecting the two front cones CN among the multiple cones CN shown in Fig. 7. Furthermore, the area setting unit 306 sets, as the restricted area RA2, an area on the opposite side of the excavator 100 with the safety barrier SB2 as the boundary.

[0121] Furthermore, in response to a user instruction such as "Set up a barrier based on the cone on the right," the area setting unit 306 generates, as a safety barrier SB3, a vertical plane parallel to the line connecting the two right-side cones CN among the multiple cones CN shown in Fig. 7. Furthermore, the area setting unit 306 sets, as a restricted area RA3, an area on the opposite side of the excavator 100 with the safety barrier SB3 as the boundary.

[0122] Furthermore, in response to a user instruction such as "Put up a barrier at the back cone," the area setting unit 306 generates a vertical surface parallel to the line connecting the two backmost cones CN among the multiple cones CN shown in Fig. 7 as a safety barrier SB4. The area setting unit 306 also sets the area on the opposite side of the excavator 100 with the safety barrier SB4 as the boundary as a restricted area RA4.

[0123] Furthermore, in response to a user instruction such as "Keep the shovel off the road," the area setting unit 306 generates a vertical surface that is parallel to the road RD and adjacent to the near side edge of the road RD as the safety barrier SB5 shown in Fig. 7. The area setting unit 306 also sets the area on the opposite side of the shovel 100 with the safety barrier SB5 as the boundary as the restricted area RA5.

[0124] Furthermore, in response to a user instruction such as "Don't let the backhoe hit the person," the area setting unit 306 generates a cylindrical or hemispherical surface surrounding the person P shown in Fig. 7 as the safety barrier SB6. Furthermore, the area setting unit 306 sets the area inside the safety barrier SB6 as the restricted area RA6.

[0125] Furthermore, in response to a user instruction such as "I want to load earth and sand without hitting the dump truck," the area setting unit 306 generates a rectangular parallelepiped surrounding the dump truck DT as the safety barrier SB7 shown in Fig. 7. Furthermore, the area setting unit 306 sets the area inside the safety barrier SB7 as the restricted area RA7.

[0126] Furthermore, in response to a user instruction such as "put up a barrier at the depth of the buried pipe," the area setting unit 306 generates a horizontal plane at a depth D from the ground surface GS, which is adjacent to and above the buried pipe UP shown in Fig. 10, as a safety barrier SB8. Furthermore, the area setting unit 306 sets the area below the safety barrier SB8 as a restricted area RA8.

[0127] Furthermore, in response to a user instruction such as "put up a barrier at the height of the electric wire," the area setting unit 306 generates a horizontal plane adjacent to and below the electric wire EW shown in Fig. 10 at a height H from the ground surface GS as a safety barrier SB9. Furthermore, the area setting unit 306 sets the area above the safety barrier SB9 as a restricted area RA9.

[0128] Furthermore, in response to a user instruction such as "Put up a barrier to prevent people from falling off the slope," the area setting unit 306 generates a vertical surface adjacent to the near side of the slope TS shown in Fig. 10 as a safety barrier SB10. The area setting unit 306 also sets the area on the opposite side of the excavator 100 with the safety barrier SB10 as the boundary as a restricted area RA10.

[0129] In this way, in this example, the controller 30 sets the restricted area RA of the shovel 100 based on the result of having the language model LM interpret the operator's instructions in natural language and information about the environment around the shovel 100 that has been verbalized in natural language. This allows the controller 30 to restrict the entry or operating speed of the shovel 100 in accordance with the operator's instructions in natural language that are tailored to the environment around the shovel 100.

[0130] The area setting unit 306 may set the restricted area RA of the shovel 100 based on the result of having the language model LM interpret the instruction acquired by the instruction acquisition unit 302, the information acquired by the information acquisition unit 301, and the posture of the shovel 100. Specifically, the area setting unit 306 applies the posture of the attachment AT acquired by the information acquisition unit 301 to a control command obtained from the language model LM based on the user's instruction to "put up a barrier at the height of the boom." This enables the area setting unit 306 to generate a safety barrier SB at the height of the boom 4 based on the posture of the attachment AT, and to set the upper side of the safety barrier SB as the restricted area RA.

[0131] The operation control unit 307 stops the operation of the shovel 100 before the shovel 100 enters the restricted area RA set by the area setting unit 306, or limits the operation speed of the shovel 100 when the shovel 100 enters the restricted area RA. Specifically, the operation control unit 307 stops the operation of the shovel 100 or limits the operation speed by outputting a control signal to the hydraulic control valve 31 or the hydraulic control valve 33 based on information on the position of the shovel 100, the attitude of the attachment AT, and the restricted area RA.

[0132] The display control unit 308 causes the display device 50A to display the restricted area RA set by the area setting unit 306. Specifically, the display control unit 308 causes the display device 50A to display an image indicating the restricted area RA superimposed on the image of the imaging device 40 displayed on the display device 50A, for example, by outputting a control signal to the display device 50A. An example of the screen 41 displayed on the display device 50A will be described below with reference to FIG. 11 .

[0133] 11 is a diagram showing a display example of the restricted area RA of the shovel 100. The screen 41 displayed on the display device 50A includes display areas 41A to 41E. The display areas 41A to 41E are arranged in a vertical line starting from the top.

[0134] Display area 41A is disposed in the upper part of screen 41. Fixed display content is displayed in display area 41A regardless of the control mode selected by controller 30. Display area 41A includes information display areas 41a to 41e and 41g to 41k.

[0135] The information display area 41a displays the current date and time. The information display area 41b displays the currently selected travel mode of the shovel 100. The information display area 41c displays an image representing the currently attached end attachment. The information display area 41d displays information relating to the fuel consumption rate (fuel efficiency) of the shovel 100. The information display area 41d includes, for example, an information display area 41d1 that displays the lifetime average fuel efficiency or the section average fuel efficiency, and an information display area 41d2 that displays the instantaneous fuel efficiency. The information display area 41e displays information representing the control state of the engine 11.

[0136] The information display area 41g displays the current temperature state of the coolant in the engine 11. The information display area 41h displays the remaining amount of fuel stored in the fuel tank. The information display area 41i displays the work mode corresponding to the rotation speed of the engine 11. The information display area 41j displays the remaining amount of urea water stored in the urea water tank. The information display area 41k displays the temperature state of the hydraulic oil in the hydraulic drive system.

[0137] A peripheral image display area 41n is displayed in the display areas 41B and 41C. An image (hereinafter referred to as a "peripheral image") showing the state of the shovel 100's surroundings is displayed in the peripheral image display area 41n, based on an image captured by the imaging device 40. The peripheral image display area 41n includes peripheral image display areas 41n1 to 41n3.

[0138] The peripheral image display area 41n1 is displayed in the display area 41B so as to be adjacent to and below the information display area 41d included in the display area 41A.

[0139] In this example, the peripheral image display area 41n1 displays an overhead image FV of the area around the shovel 100 viewed from above, which is generated based on the image captured by the imaging device 40. The peripheral image display area 41n1 also displays an shovel image GE that simulates the shovel 100 viewed from above. The shovel image GE and the overhead image FV are arranged in the peripheral image display area 41n1 so that their positional relationship matches the positional relationship between the shovel 100 and the imaging range included in the overhead image FV.

[0140] The peripheral image display areas 41n2 and 41n3 are displayed in the display area 41C adjacent to and below the peripheral image display area 41n1. The peripheral image display areas 41n2 and 41n3 are arranged adjacent to the left and right parts of the display area 41C, respectively, with respect to the center in the horizontal direction.

[0141] In this example, the peripheral image display area 41n2 displays a rearward image BM that shows the situation behind the excavator 100, and the peripheral image display area 41n3 displays a rightward image RM that shows the situation to the right of the excavator 100. The rearward image BM and the rightward image RM correspond to images captured by the camera 40B and the camera 40R, respectively.

[0142] In this example, safety barriers SB3 and SB5 indicating the boundaries of the restricted areas RA3 and RA5 are displayed superimposed on the overhead image FV in the peripheral image display area 41n1. Also, a safety barrier SB5 indicating the boundary of the restricted area RA5 is displayed superimposed on the rearward image BM in the peripheral image display area 41n2, and a safety barrier SB3 indicating the boundary of the restricted area RA3 is displayed superimposed on the right-hand image RM in the peripheral image display area 41n3.

[0143] The display area 41D includes information display areas 41f and 41m. The information display area 41f is arranged below and adjacent to the peripheral image display area 41n2. The cumulative operating time of the engine 11 is displayed in the information display area 41f. The information display area 41m is arranged below the peripheral image display area 41n3 and adjacent to the right of the information display area 41f. The operating status of the air conditioner is displayed in the information display area 41m. The information display area 41m includes information display areas 41m1 to 41m4. The information display area 41m1 displays the position of the air outlet currently being used to blow air from the air conditioner. The information display area 41m2 displays the current operating mode of the air conditioner. The information display area 41m3 displays the current set temperature of the air conditioner. The information display area 41m4 displays the current set air volume of the air conditioner.

[0144] Display area 41E is disposed at the bottom of screen 41. Fixed display content is displayed in display area 41E regardless of the control mode selected by controller 30. Specifically, display area 41E displays tab group 41q, which are operation elements for selecting one control mode to be applied to controller 30 from a plurality of control modes. For example, the operator can operate tab group 41q by using touch panel 80 as input device 52. Alternatively, the operator may be able to operate tab group 41q by using switches associated with display device 50A as input device 52.

[0145] Tab group 41q includes tabs 41q1 to 41q6. Tabs 41q1 to 41q6 are arranged in a row from left to right. Tab 41q1 is an operation icon for configuring settings related to screen 41. Tabs 41q2 to 41q5 are operation icons corresponding to four of the multiple control modes. This allows the operator to select one control mode to be applied to controller 30 from the multiple control modes by using touch panel 80 or the like to select and confirm one of tabs 41q2 to 41q5.

[0146] Next, an example of the flow of a process for setting the restricted area RA of the shovel 100 will be described with reference to Fig. 12. Fig. 12 is a flow chart showing an example of the flow of a process for setting the restricted area RA of the shovel 100. In this example, it is assumed that the controller 30 includes the units shown in Fig. 6.

[0147] 12 and executes process P1 for acquiring the instruction text, when an instruction in the operator's natural language is input to the input device 52 or the input device 207. In process P1, the instruction acquisition unit 302 acquires the instruction text in the operator's natural language from the input device 52 or the input device 207.

[0148] Next, the controller 30 executes process P2 to acquire information regarding the attitude of the attachment AT and the environment around the shovel 100. In this process P2, the information acquisition unit 301 calculates the attitude of the attachment AT based on the detection results of, for example, the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, the rotation sensor S5, the sensor S6, etc. The information acquisition unit 301 also detects objects around the shovel 100, the direction and distance to the objects, the position coordinates of the objects, etc., based on, for example, the output of the imaging device 40 and a distance measurement sensor. The information acquisition unit 301 also acquires, for example, information regarding construction drawings of the work site where the shovel 100 will be working.

[0149] Next, the controller 30 executes a process P3 for verbalizing the acquired information. In this process P3, the verbalization unit 303 verbalizes the information acquired by the information acquisition unit 301 in natural language.

[0150] Next, the controller 30 executes process P4, which translates the instruction text acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303. In process P4, the prompt generation unit 304 also functions as a translation unit that translates the instruction text and verbalized information into the standard language of the language model LM, i.e., a specific language corresponding to the language model LM. Specifically, if the standard language of the language model LM is English and the instruction text and verbalized information are in a language other than English, the prompt generation unit 304 translates the instruction text and verbalized information into English by machine translation. Note that process P4 may be omitted.

[0151] Next, the controller 30 executes a process P5 for generating a prompt. In this process P5, the prompt generator 304 generates a prompt to be input to the language model LM based on the instruction text and verbalized information, or machine translations thereof.

[0152] Next, the controller 30 executes a process P6 for calling the language model LM. In this process P6, the calling unit 305 calls the language model LM, inputs the prompt generated by the prompt generating unit 304 in the previous process P5 to the language model LM, and inputs the control command obtained from the language model LM to the area setting unit 306.

[0153] Next, the controller 30 executes a process P7 for setting a restricted area RA. In this process P7, the area setting unit 306 sets the restricted area RA based on the control command obtained from the language model LM.

[0154] Next, the controller 30 executes process P8 for restricting the operation of the shovel 100 and displaying a screen. In process P8, the operation control unit 307 outputs a control signal to the hydraulic control valve 31 or the like to stop the operation of the shovel 100 immediately before the shovel 100 enters the restricted area RA, or to restrict the operating speed of the shovel 100 when the shovel 100 enters the restricted area RA. In addition, the display control unit 308 outputs a control signal to the display device 50A to display a safety barrier SB, which is the boundary of the restricted area RA, on the screen 41 of the display device 50A. Thereafter, the controller 30 ends the process flow shown in FIG. 12 .

[0155] As described above, the shovel 100, which is an embodiment of a construction machine according to the present disclosure, includes a work attachment AT, an information acquisition unit 301, a verbalization unit 303, an instruction acquisition unit 302, and an area setting unit 306. The information acquisition unit 301 acquires information related to the posture of the attachment AT and the surrounding environment. The verbalization unit 303 verbalizes the information acquired by the information acquisition unit 301 in natural language. The instruction acquisition unit 302 acquires instructions in natural language from an operator. The area setting unit 306 sets a restricted area RA in which entry or movement speed at the work site is restricted. The area setting unit 306 then sets the restricted area RA based on the result of having the language model LM interpret the instruction acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303, or based on the result of having the language model LM interpret the instruction acquired by the instruction acquisition unit 302 and the information acquired by the information acquisition unit 301.

[0156] With this configuration, the shovel 100 of this embodiment can set a restricted area RA in which the entry or operating speed of the shovel 100 is restricted based on instructions in natural language from the operator. This allows the operator to set the restricted area RA more easily than in the case where the operator manually sets the restricted area RA while checking the environment around the shovel 100. Therefore, the shovel 100 of this embodiment allows a diverse range of personnel, such as inexperienced operators and operators of foreign nationality, to perform work while ensuring safety.

[0157] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the information acquisition unit 301 acquires information about the attitude of the attachment AT and the surrounding environment based on the outputs of a position sensor that acquires position information of the shovel 100, an attitude sensor that detects the attitude of the attachment AT, and an external sensor that detects objects around the shovel 100. In this embodiment, the position sensor includes sensor S6, the attitude sensor includes sensors S1 to S5, and the external sensor includes an imaging device 40 and a ranging sensor.

[0158] With this configuration, the shovel 100 of this embodiment can obtain information regarding the posture of the attachment AT based on the output of the posture sensor, and can obtain information regarding the environment surrounding the shovel 100 based on the output of the position sensor and the external sensor.

[0159] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the area setting unit 306 sets the restricted area RA based on the results of having the language model LM interpret the instructions acquired by the instruction acquisition unit 302 and the information on the object detected by the external sensor.

[0160] With this configuration, the operator of the excavator 100 can set the restricted areas RA1 to RA7 by giving instructions in natural language based on the positions of the cone CN, person P, road RD, dump truck DT, etc. detected by the external sensor, as shown in Figure 7.

[0161] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the area setting unit 306 sets the restricted area RA based on the result of having the language model LM interpret the instructions acquired by the instruction acquisition unit 302, the position information of the shovel 100 acquired by the position sensor, and the attitude of the attachment AT acquired by the attitude sensor.

[0162] With this configuration, the operator of the shovel 100 can set the restricted area RA by giving instructions in natural language based on the height of each part of the attachment AT obtained based on the posture of the attachment AT.

[0163] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, when information on multiple objects is acquired based on the detection results of an external sensor and an instruction to set a restricted area RA for the multiple objects is acquired by the instruction acquisition unit 302, the area setting unit 306 sets a restricted area RA using a convex hull based on the positions of the multiple objects.

[0164] With this configuration, the operator of the shovel 100 can give instructions in natural language to set a restricted area RA1 using a convex hull based on the positions of multiple cones CN detected by the external sensor of the shovel 100, for example, as shown in Figure 7.

[0165] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, when two objects are detected by an external sensor and an instruction to set a restricted area RA based on those two objects is acquired by the instruction acquisition unit 302, the area setting unit 306 sets a restricted area RA on the opposite side of the shovel 100 based on the straight line connecting the two objects.

[0166] With this configuration, the operator of the shovel 100 can give instructions in natural language to set restricted areas RA2, RA3, and RA4, for example, based on a straight line connecting two cones CN detected by the external sensor of the shovel 100, as shown in Figure 7.

[0167] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, when a specific object is recognized based on the detection results of the external sensor and an instruction to set a restricted area RA for the specific object is acquired by the instruction acquisition unit 302, the area setting unit 306 sets a restricted area RA in an area that includes the specific object.

[0168] With this configuration, the operator of the shovel 100 can give instructions in natural language to set restricted areas RA6, RA7 in an area that includes a person P and a dump truck DT detected by the external sensor of the shovel 100, for example, as shown in Figure 7.

[0169] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, when position information of a movable object is acquired based on the detection results of an external sensor and an instruction to set a restricted area RA for the movable object is acquired by the instruction acquisition unit 302, the area setting unit 306 sets a restricted area RA based on the position information of the movable object.

[0170] With this configuration, the operator of the excavator 100 can give instructions in natural language, and when, for example, a person P, a cone CN, a dump truck DT, etc. shown in Figure 7 move, the restricted areas RA1 to RA4, RA6, and RA7 set for those objects can be moved along with those objects.

[0171] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the position information of the shovel 100 is acquired by a position sensor, and the attitude of the attachment AT is acquired by an attitude sensor. In this case, when the instruction acquisition unit 302 acquires an instruction to set a restricted area RA based on the height of the shovel 100 or the attachment AT, the area setting unit 306 sets the restricted area RA above or below the height of the shovel 100 or the attachment AT.

[0172] With this configuration, the operator of the excavator 100 can give instructions in natural language to set a restricted area RA on the upper or lower side, for example, based on the height of the boom 4, arm 5, bucket 6, etc., which are part of the attachment AT.

[0173] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, when an instruction to expand or reduce the set restricted area RA is acquired by the instruction acquisition unit 302, the area setting unit 306 expands or reduces the restricted area RA in accordance with the instruction.

[0174] With this configuration, the operator of the shovel 100 can give instructions in natural language to expand or shrink the restricted area RA surrounding the target object, such as the restricted areas RA1, RA6, and RA7 shown in Figure 7.

[0175] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the area setting unit 306 fixes the restricted area RA when an instruction to fix the set restricted area RA is acquired by the instruction acquisition unit 302.

[0176] With this configuration, even if a restricted area RA1 is set in an area surrounded by multiple cones CN as shown in Figure 7 and then the cone CN is moved, it is possible to maintain the restricted area RA1 that was set based on the position information of the cone CN before the movement.

[0177] Furthermore, in the shovel 100, which is an embodiment of the work machine according to the present disclosure, the area setting unit 306 deletes the restricted area RA when an instruction to delete the set restricted area RA is acquired by the instruction acquisition unit 302.

[0178] With this configuration, when the operator of the excavator 100 wants to delete a restricted area RA after setting it, he or she can give an instruction in natural language, which is then acquired by the instruction acquisition unit 302, and the restricted area RA is deleted by the area setting unit 306. Therefore, it is possible to easily delete a restricted area RA after it has been set.

[0179] Furthermore, in the shovel 100 which is an embodiment of the work machine according to the present disclosure, the information acquired by the information acquisition unit 301 includes information on construction drawings.

[0180] 10, the information acquisition unit 301 can acquire information such as the buried pipe UP, the electric wire EW, the slope shoulder TS, etc., which are difficult to detect by an external sensor such as the imaging device 40, from the information on the construction drawing. As a result, the operator of the excavator 100 can set the restricted areas RA8, RA9, RA10 based on the buried pipe UP, the electric wire EW, the slope shoulder TS, etc., by giving instructions in natural language.

[0181] Moreover, the shovel 100, which is an embodiment of the work machine according to the present disclosure, further includes a prompt generation unit 304 that causes a language model LM to interpret the instructions acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303. The prompt generation unit 304 also functions as a translation unit that translates the instructions acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303 into a specific language corresponding to the language model LM and causes the language model LM to interpret the instructions.

[0182] This configuration can improve the accuracy of the output of the language model LM. More specifically, if the specific language corresponding to the language model LM is English, the prompt generation unit 304 translates instructions and information in a language other than English into English and causes the language model LM to interpret them. As a result, the accuracy of the output of the language model LM in response to a prompt input from the prompt generation unit 304 to the language model LM via the calling unit 305 can be improved.

[0183] Moreover, the shovel 100 , which is an embodiment of the work machine according to the present disclosure, further includes a display device 50A that displays the restricted area RA set by the area setting unit 306 .

[0184] With this configuration, the operator of the shovel 100 can visually understand the restricted area RA set around the shovel 100 by checking the screen 41 of the display device 50A. Specifically, in the example shown in FIG. 7 , restricted areas RA3, RA5 are set to the right and rear of the shovel 100. In this case, as shown in FIG. 11 , the display device 50A displays virtual safety barriers SB3, SB5 that are boundaries of the restricted areas RA3, RA5 by superimposing them on the overhead image FV, the right image RM, the rear image BM, etc., based on the image captured by the imaging device 40. As a result, the operator of the shovel 100 can intuitively grasp the restricted areas RA3, RA5 set to the right and rear of the shovel 100 based on the display on the screen 41 of the display device 50A.

[0185] The operation support system SYS of this embodiment also includes an information acquisition unit 301, a verbalization unit 303, an instruction acquisition unit 302, and an area setting unit 306. The information acquisition unit 301 acquires information regarding the posture of the attachment AT of the shovel 100, which is a work machine, and the environment around the shovel 100. The verbalization unit 303 verbalizes the information acquired by the information acquisition unit 301 in natural language. The instruction acquisition unit 302 acquires instructions in natural language from the operator of the shovel 100. The area setting unit 306 sets the restricted area RA in which the entry or operating speed of the shovel 100 is restricted. The area setting unit 306 sets the restricted area RA based on a result of having a language model LM interpret the instruction acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303, or based on a result of having the language model LM interpret the instruction acquired by the instruction acquisition unit 302 and the information acquired by the information acquisition unit 301.

[0186] With this configuration, the operation support system SYS of this embodiment can set a restricted area RA in which the entry or operating speed of the shovel 100 is restricted based on instructions in natural language from the operator, just like the shovel 100 of this embodiment. This allows the operator to set the restricted area RA more easily than in the case where the operator sets the restricted area RA manually while checking the environment around the shovel 100. Therefore, the operation support system SYS of this embodiment allows a diverse range of human resources, such as inexperienced operators and operators of foreign nationality, to perform work while ensuring safety.

[0187] Preferred embodiments of the present disclosure have been described above. However, the invention according to the present disclosure is not limited to the above-described embodiments. Various modifications, substitutions, etc. may be applied to the above-described embodiments without departing from the scope of the invention according to the present disclosure. Furthermore, the features described with reference to the above-described embodiments may be combined as appropriate as long as there is no technical contradiction. For example, the above-described restricted area setting function may be applied to work machines other than excavators. Examples of other work machines include bulldozers, mobile cranes, etc.

[0188] This application claims priority based on Japanese Patent Application No. 2024-070004, filed on April 23, 2024, the entire contents of which are incorporated herein by reference.

[0189] 40 Imaging device (external sensor), 50A Display device, 100 Excavator (work machine), 301 Information acquisition unit, 303 Language generation unit, 302 Instruction acquisition unit, 304 Prompt generation unit (translation unit), 306 Area setting unit, AT Attachment, CN Cone (object), DT Dump truck (object), LM Language model, P Person (object), RA Restricted area, RA1-RA10 Restricted areas, RD Road (object), S1 Boom angle sensor (attitude sensor), S2 Arm angle sensor (attitude sensor), S3 Bucket angle sensor (attitude sensor), S4 Machine body inclination sensor (attitude sensor), S6 Sensor (position sensor), SYS Operation support system.

Claims

1. A work machine comprising: a work attachment; an information acquisition unit that acquires information about the posture of the attachment and the surrounding environment; a language conversion unit that converts the information acquired by the information acquisition unit into natural language; an instruction acquisition unit that acquires instructions in natural language from an operator; and an area setting unit that sets a restricted area in which entry or movement speed is restricted at a work site, wherein the area setting unit sets the restricted area based on the result of interpreting the instructions acquired by the instruction acquisition unit and the information converted by the language conversion unit using a language model, or based on the result of interpreting the instructions acquired by the instruction acquisition unit using the language model and the information acquired by the information acquisition unit.

2. A work machine as described in claim 1, wherein the information acquisition unit acquires information about the attitude of the attachment and the surrounding environment based on the outputs of a position sensor that acquires position information of the work machine, an attitude sensor that detects the attitude of the attachment, and an external sensor that detects objects around the work machine.

3. The work machine according to claim 2, wherein the area setting unit sets the restricted area based on the results of interpreting the instructions acquired by the instruction acquisition unit and information about objects detected by the external sensor using the language model.

4. A work machine as described in claim 2, wherein the area setting unit sets the restricted area based on the result of interpreting the instructions acquired by the instruction acquisition unit using the language model, the position information of the work machine acquired by the position sensor, and the attitude of the attachment acquired by the attitude sensor.

5. A work machine as described in claim 3, wherein when information on multiple objects is acquired based on the detection results of the external sensor and an instruction to set the restricted area for the multiple objects is acquired by the instruction acquisition unit, the area setting unit sets the restricted area using a convex hull based on the positions of the multiple objects.

6. A work machine as described in claim 3, wherein when two objects are detected by the external sensor and an instruction to set the restricted area based on the two objects is acquired by the instruction acquisition unit, the area setting unit sets the restricted area on the opposite side of the work machine based on a straight line connecting the two objects.

7. A work machine as described in claim 3, wherein the area setting unit sets the restricted area to an area that includes a specific object when a specific object is recognized based on the detection results of the external sensor and an instruction to set the restricted area for the specific object is acquired by the instruction acquisition unit.

8. A work machine as described in claim 3, wherein the area setting unit sets the restricted area based on the position information of a movable object when position information of the movable object is acquired based on the detection results of the external sensor and an instruction to set the restricted area for the movable object is acquired by the instruction acquisition unit.

9. A work machine according to claim 4, wherein the area setting unit sets the restricted area above or below the height of the work machine or the attachment when position information of the work machine is acquired by the position sensor, the attitude of the attachment is acquired by the attitude sensor, and an instruction to set the restricted area based on the height of the work machine or the attachment is acquired by the instruction acquisition unit.

10. A work machine as described in claim 1, wherein when an instruction to expand or reduce the set restricted area is acquired by the instruction acquisition unit, the area setting unit expands or reduces the restricted area in accordance with the instruction.

11. The work machine according to claim 1, wherein the area setting unit fixes the restricted area when an instruction to fix the set restricted area is acquired by the instruction acquisition unit.

12. The work machine according to claim 1, wherein the area setting unit deletes the restricted area when an instruction to delete the set restricted area is acquired by the instruction acquisition unit.

13. The work machine according to claim 1, wherein the information acquired by the information acquisition unit includes information on construction drawings.

14. The work machine according to claim 1, further comprising a translation unit that translates the instructions acquired by the instruction acquisition unit and the information verbalized by the verbalization unit into a specific language corresponding to the language model and causes the language model to interpret the instructions.

15. The work machine according to claim 1, further comprising a display device that displays the restricted area set by the area setting unit.

16. An operation assistance system comprising: an information acquisition unit that acquires information regarding the posture of a work machine attachment and the environment around the work machine; a language conversion unit that converts the information acquired by the information acquisition unit into natural language; an instruction acquisition unit that acquires instructions in natural language from an operator of the work machine; and an area setting unit that sets a restricted area in which the intrusion or operating speed of the work machine is restricted, wherein the area setting unit sets the restricted area based on the result of interpreting, using a language model, the instructions acquired by the instruction acquisition unit and the information converted into language by the language conversion unit, or based on the result of interpreting, using the language model, the instructions acquired by the instruction acquisition unit and the information acquired by the information acquisition unit.

Images