Work machine, operation support system

Abstract

The present disclosure provides a work machine capable of setting a restricted area according to a natural language-based instruction. The work machine (shovel 100) is provided with a work attachment (AT), an information acquisition unit (301), a verbalization unit (303), an instruction acquisition unit (302), and an area setting unit (306). The information acquisition unit acquires information about the posture of the attachment and the environment around the attachment. The verbalization unit verbalizes the information acquired by the information acquisition unit in natural language. The instruction acquisition unit acquires an instruction in natural language from an operator. The area setting unit sets a restricted area (RA1-RA10) in which intrusion or movement speed is restricted in a work site. The area setting unit sets the restricted area according to a result obtained by causing a language model (LM) to analyze the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit, or according to a result obtained by causing the language model to analyze the instruction acquired by the instruction acquisition unit and the information acquired by the information acquisition unit.

Description

Technical Field

[0001] This disclosure relates to a work machine and an operation support system. Background Technology

[0002] Previously, there was a display device for an excavator that properly conveyed instructions from external workers to the operator (see Patent Document 1 below).

[0003] The display device for an excavator described in Patent Document 1 includes: a communication unit; a display unit; a voice acquisition unit for acquiring voice from inside the cab; and a voice-to-text conversion unit for converting the voice acquired by the voice acquisition unit into text information. The display unit displays, in chronological order, transmission information acquired by the communication unit and transmitted from an external terminal, and text information converted by the voice-to-text conversion unit.

[0004] With this structure, the display device of the excavator in Patent Document 1 can display external instructions received by the communication unit as text information in chronological order. Furthermore, if the excavator operator responds to external instructions with voice, the response is converted into text information, displayed on the display device, and transmitted to the outside, thus making it easier to respond.

[0005] Previous technical documents Patent documents Patent Document 1: International Publication No. 2020 / 080501 Summary of the Invention

[0006] The technical problem to be solved by the invention However, in the technology described in Patent Document 1, the operator of the excavator needs to manually set the restricted area where the excavator's intrusion or movement speed is limited.

[0007] This disclosure provides a work machine and operation support system capable of setting restricted areas based on natural language-based instructions.

[0008] means for solving technical problems One aspect of this disclosure provides a work machine comprising: an auxiliary device for work; an information acquisition unit for acquiring information related to the posture of the auxiliary device and the surrounding environment; a speech processing unit for processing the information acquired by the information acquisition unit into natural language; an instruction acquisition unit for acquiring instructions given by an operator in natural language; and a zone setting unit for setting a restricted area in the work site where intrusion or movement speed is limited. The zone setting unit sets the restricted area based on a result obtained by parsing the instructions acquired by the instruction acquisition unit and the information processed by the speech processing unit using a language model, or based on a result obtained by parsing the instructions acquired by the instruction acquisition unit using the language model and the information acquired by the information acquisition unit.

[0009] Another aspect of this disclosure provides an operation support system comprising: an information acquisition unit for acquiring information related to the posture of an accessory device of a work machine and the surrounding environment of the work machine; a speech processing unit for processing the information acquired by the information acquisition unit into natural language; an instruction acquisition unit for acquiring instructions given in natural language by the operator of the work machine; and a zone setting unit for setting a restricted zone in which the intrusion or movement speed of the work machine is restricted. The zone setting unit sets the restricted zone based on a result obtained by parsing the instructions acquired by the instruction acquisition unit and the speech processing unit into natural language using a language model, or based on a result obtained by parsing the instructions acquired by the instruction acquisition unit and the information acquired by the information acquisition unit using the language model.

[0010] Invention Effects According to the above-described manner of this disclosure, it is possible to provide a working machine and operation support system capable of setting restricted areas based on natural language-based instructions. Attached Figure Description

[0011] Figure 1 This is a side view showing an example of an excavator.

[0012] Figure 2 This is a top view showing an example of an excavator.

[0013] Figure 3 This is a diagram illustrating an example of an operational support system.

[0014] Figure 4 This is a diagram illustrating an example of the hardware structure of an excavator.

[0015] Figure 5 This is a diagram illustrating an example of the hardware structure of a remote operation support device.

[0016] Figure 6This is a function block diagram illustrating an example of the restricted area setting function of an excavator.

[0017] Figure 7 This is an example of the environment surrounding an excavator.

[0018] Figure 8 This is a diagram illustrating an example of a problem assigned to a language model.

[0019] Figure 9 This is a diagram illustrating an example of a combination of prompts and outputs from a language model.

[0020] Figure 10 This is another example of the environment surrounding the excavator.

[0021] Figure 11 This is a diagram showing an example of the restricted area of ​​an excavator.

[0022] Figure 12 This is a flowchart illustrating an example of the processing related to setting the restricted area of ​​an excavator. Detailed Implementation

[0023] Hereinafter, embodiments of the operating machinery and operation support system involved in this disclosure will be described with reference to the accompanying drawings. The embodiments described below are illustrative and do not limit the invention. All features and combinations thereof in the embodiments of this disclosure are not necessarily the essence of the invention. In addition, in the various drawings, the same or corresponding structures are sometimes labeled with the same or corresponding symbols, and repeated descriptions are omitted.

[0024] Figure 1 This is a side view showing an example of an embodiment of the working machinery involved in this disclosure, namely an excavator 100. Figure 2 yes Figure 1 The top view of the excavator 100 shown. Figure 3 This is a structural diagram illustrating an example of an implementation of the operation support system according to this disclosure. Hereinafter, the direction in which the auxiliary device AT extends when viewed from above the excavator 100 will sometimes be indicated. Figure 2 The direction of the excavator 100 or the direction observed from the excavator 100 is specified as "front" (upward direction).

[0025] like Figure 1 , Figure 2 As shown, the excavator 100 includes a lower traveling body 1; an upper slewing body 3; an auxiliary device AT including a boom 4, a stick 5 and a bucket 6; and a cab 10.

[0026] The lower traveling body 1 uses tracks 1C to move the excavator 100. Tracks 1C include a left track 1CL and a right track 1CR. Track 1CL is hydraulically driven by a travel hydraulic motor 1ML. Similarly, track 1CR is hydraulically driven by a travel hydraulic motor 1MR. Thus, the lower traveling body 1 can move autonomously.

[0027] The upper rotating body 3 is rotatably (flexibly) mounted on the lower traveling body 1 via the rotating mechanism 2. For example, the upper rotating body 3 is hydraulically driven by the rotating mechanism 2 via the rotating hydraulic motor 2M, thereby rotating relative to the lower traveling body 1.

[0028] The boom 4 is mounted at the center of the front of the upper slewing body 3 in a manner that allows it to pitch around a rotation axis in the left-right direction. The stick 5 is mounted at the front end of the boom 4 in a manner that allows it to rotate around a rotation axis in the left-right direction. The bucket 6 is mounted at the front end of the stick 5 in a manner that allows it to rotate around a rotation axis in the left-right direction.

[0029] Bucket 6 is an example of an end-attachment, such as for excavation, ramp work, or ground clearing operations.

[0030] The bucket 6 is mounted on the front end of the boom 5 in a manner that allows for appropriate replacement to correspond to the work performed by the excavator 100. That is, a different type of bucket, such as a relatively large bucket, a slope bucket, or a dredging bucket, can be installed at the front end of the boom 5 instead of the bucket 6. Furthermore, end attachments other than buckets, such as mixers, hydraulic breakers, or shredders, can also be installed at the front end of the boom 5. Additionally, spare auxiliary devices such as quick-connect couplings or tilting rotators can be provided between the boom 5 and the end attachments.

[0031] The boom 4, stick 5, and bucket 6 are hydraulically driven by the boom cylinder 7, stick cylinder 8, and bucket cylinder 9, respectively.

[0032] The cab 10 is a control room for the operator to sit in and operate the excavator 100. The cab 10 is, for example, mounted on the front left side of the upper rotating body 3.

[0033] The excavator 100 can be equipped with a communication device 60 and can communicate with the remote operation support device 200 via a specified communication line NW.

[0034] Communication lines (NW) can include, for example, local area networks (LANs) at work sites. Furthermore, communication lines (NW) can also include wide area networks (WANs). Wide area networks include, for example, mobile communication networks with base stations as endpoints, satellite communication networks utilizing communication satellites, and the Internet. Additionally, communication lines (NW) can also include, for example, short-range communication lines based on wireless communication standards such as WiFi or Bluetooth (registered trademark).

[0035] For example, the excavator 100 causes the driven components such as the lower walking body 1 (i.e., the left and right pairs of tracks 1CL and 1CR), the upper slewing body 3, the boom 4, the stick 5, and the bucket 6 to move in response to the operation of the operator sitting in the cab 10.

[0036] Furthermore, the excavator 100 can be configured to be remotely operated from outside the excavator 100 (remote operation) instead of being operated by an operator sitting in the cab 10, or, in addition to being operable by an operator sitting in the cab 10, it can also be remotely operated from outside the excavator 100. In the case of remote operation of the excavator 100, the cab 10 can be unmanned. Furthermore, if the excavator 100 is dedicated to remote operation, the cab 10 can be omitted. Hereinafter, the description will assume that the operator's operation includes at least one of the following: operation of the operating device 26 by the operator in the cab 10 and remote operation by an external operator.

[0037] For example, such as Figure 3 As shown, remote operation utilizes an operation support system SYS. In this system SYS, the excavator 100 to be operated and a remote operation support device 200 for the operator to operate the excavator 100 are connected via a communication line NW to enable communication. Specifically, remote operation includes operating the excavator 100 by means of operation inputs related to the actuators of the excavator 100 made by the remote operation support device 200, which can communicate with the excavator 100 via the communication line NW.

[0038] The remote operation support device 200 may be installed, for example, in a management center that manages the operation of the excavator 100 from the outside. Furthermore, the remote operation support device 200 may also be a portable operating terminal, in which case the operator can remotely operate the excavator 100 while directly monitoring its operating status from its vicinity.

[0039] For example, the excavator 100 transmits an image (hereinafter referred to as "peripheral image") showing the surrounding conditions of the excavator 100, based on the camera image output by its own camera device 40, to the remote operation support device 200 via the communication device 60. Alternatively, the excavator 100 may transmit the camera image output by the camera device 40 to the remote operation support device 200 via the communication device 60, and the remote operation support device 200 may process the camera image received from the excavator 100 and generate the peripheral image. Then, the remote operation support device 200 displays the peripheral image showing the surrounding conditions of the excavator 100, including its front and surrounding areas, on its own display device. Furthermore, various information images (information screens) displayed on the output device 50 (display device) provided inside the excavator 100's cab 10 can also be displayed on the display device of the remote operation support device 200. Therefore, an operator using the remote operation support device 200 can remotely operate the excavator 100 while simultaneously checking the displayed content such as images and information screens showing the surrounding conditions of the excavator 100 on the display device. Then, the excavator 100 activates its actuators based on signals indicating the remote operation content received from the remote operation support device 200 via the communication device 60, thereby actuating driven components such as the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6. Thus, the operation support system SYS enables remote operation of the excavator 100 using the remote operation support device 200.

[0040] Furthermore, remote operation may include methods such as operating the excavator 100 based on external voice or gesture input from people (e.g., operators) present around the excavator 100. Specifically, the excavator 100 recognizes voice or gestures from nearby operators using a voice input device (e.g., a microphone) or gesture input device (e.g., a camera). Moreover, the excavator 100 can activate actuators based on the recognized voice or gestures to drive driven components such as the lower walking body 1 (left and right tracks 1C), upper slewing body 3, boom 4, stick 5, and bucket 6.

[0041] Furthermore, the excavator 100 can automatically operate its actuators regardless of the operator's actions. Thus, the excavator 100 can achieve the function of automatically operating at least a portion of the driven components, such as the lower traveling body 1, the upper rotating body 3, and the auxiliary device AT, i.e., the so-called "automatic operation function" or "machine control (MC) function".

[0042] Automatic operation functions may include, for example, semi-automatic operation functions (operation support type MC functions). Semi-automatic operation functions are functions that automatically activate driven components (actuators) other than the driven components (actuators) of the workpiece in response to operator input. Furthermore, automatic operation functions may also include fully automatic operation functions (fully automatic type MC functions). Fully automatic operation functions are functions that automatically activate at least a portion of multiple driven components (actuators) without operator input. In the excavator 100, when the fully automatic operation function is active, the interior of the cab 10 can be unmanned. Moreover, semi-automatic or fully automatic operation functions may include, for example, rule-based automatic operation functions. Rule-based automatic operation functions are automatic operation functions that automatically determine the action content of the driven components (actuators) that are the workpieces of the automatic operation according to predefined rules. Furthermore, semi-automatic or fully automatic operation functions may also include autonomous operation functions. The autonomous operation function is an automatic operation function in the following manner: the excavator 100 autonomously makes various judgments and determines the action content of the driven element (actuator) that becomes the object of automatic operation based on its judgment results.

[0043] Furthermore, the operation of the excavator 100 can be remotely monitored. In this case, a remote monitoring support device with the same functions as the remote operation support device 200 can be installed. The remote monitoring support device is, for example, the remote operation support device 200. Thus, the monitor, as the user of the remote monitoring support device, can monitor the operating status of the excavator 100 while simultaneously checking the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, if deemed necessary from a safety perspective, the monitor can intervene in the operator's operation of the excavator 100 or its automatic operation by making specified inputs through the input device of the remote monitoring support device, causing the excavator 100 to stop urgently.

[0044] Next, refer to Figure 4 The structure of the excavator 100 is described. Figure 4 This is a block diagram illustrating an example of the structure of an excavator 100. Additionally, in Figure 4 In the diagram, the path for transmitting mechanical power is represented by a double line, the path for the high-pressure working oil driving the hydraulic actuator HA is represented by a solid line, the path for transmitting pilot pressure is represented by a dashed line, and the path for transmitting electrical signals is represented by a dotted line.

[0045] The excavator 100 includes various components such as a hydraulic drive system related to the hydraulic drive of the driven component, an operating system related to the operation of the driven component, a user interface system related to information exchange with the user, a communication system related to external communication, and a control system related to various controls.

[0046] like Figure 4 As shown above, the hydraulic drive system of the excavator 100 includes hydraulic actuators HA that hydraulically drive the driven components such as the lower traveling body 1 (left and right tracks 1CL and 1CR), the upper slewing body 3, the boom 4, the stick 5, and the bucket 6. Furthermore, the hydraulic drive system of the excavator 100 according to this embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0047] The hydraulic actuator HA includes a travel hydraulic motor 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, a stick cylinder 8, and a bucket cylinder 9, etc. Furthermore, in the excavator 100, some or all of the hydraulic actuator HA can be replaced with electric actuators. That is, the excavator 100 can be a hybrid excavator or an electric excavator.

[0048] Engine 11 is the prime mover of excavator 100 and the main power source in the hydraulic drive system. Engine 11 is, for example, a diesel engine that uses diesel fuel. Engine 11 is, for example, mounted at the rear of the upper rotating body 3. Engine 11 rotates at a constant target speed under the direct or indirect control of controller 30 (described later) to drive main pump 14 and pilot pump 15. Alternatively, other prime movers (e.g., electric motors) may be mounted on excavator 100 instead of engine 11.

[0049] The regulator 13 controls (adjusts) the discharge volume of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swashplate of the main pump 14 (hereinafter referred to as "tilt angle") according to the control command from the controller 30.

[0050] The main pump 14 supplies working oil to the control valve 17 via a high-pressure hydraulic line. Similar to the engine 11, the main pump 14 is mounted, for example, at the rear of the upper rotating body 3. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable-capacity hydraulic pump. As described above, under the control of the controller 30, the main pump 14 adjusts the piston stroke length by adjusting the tilt angle of the swashplate via the adjuster 13, thereby controlling the discharge flow rate or discharge pressure.

[0051] Control valve 17 drives hydraulic actuators HA according to operator input to operating device 26, remote operation commands, or operation instructions corresponding to automatic operation functions. Control valve 17 is, for example, mounted in the central part of the upper rotating body 3. As described above, control valve 17 is connected to main pump 14 via high-pressure hydraulic lines and selectively supplies working oil from main pump 14 to each hydraulic actuator according to operator input or operation instructions corresponding to automatic operation functions. Specifically, control valve 17 includes multiple control valves (directional valves) that control the flow rate and direction of working oil supplied from main pump 14 to each hydraulic actuator HA.

[0052] like Figure 4 As shown, the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0053] Pilot pump 15 supplies pilot pressure to various hydraulic devices via pilot line 25. Similar to engine 11, pilot pump 15 is, for example, mounted at the rear of upper rotating body 3. Pilot pump 15 is, for example, a fixed displacement hydraulic pump, driven by engine 11 as described above. Alternatively, pilot pump 15 may be omitted. In this case, the relatively high-pressure working oil discharged from main pump 14, after being reduced to a relatively low pressure by a pre-defined pressure reducing valve, can be supplied as pilot pressure to various hydraulic devices.

[0054] The operating device 26 is located near the operator's seat in the cab 10 and is used by the operator to operate various driven components. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive each driven component, thereby enabling the operator to operate the driven components that are driven by the hydraulic actuators HA. The operating device 26 includes pedal devices or lever devices for operating each driven component (hydraulic actuator HA).

[0055] For example, such as Figure 4 As shown, the operating device 26 is a hydraulically piloted type. Specifically, the operating device 26 utilizes working oil supplied from the pilot pump 15 via the pilot line 25 and its branch pilot line 25A to output pilot pressure corresponding to the operation to the secondary side pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the shuttle valve 32. Thus, pilot pressure corresponding to the operation of various driven components (hydraulic actuators HA) in the operating device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by the operator or others.

[0056] Furthermore, the operating device 26 can also be electrically powered. In this case, the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as the "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. Then, the controller 30 outputs a control command corresponding to the content of the operation signal to the hydraulic control valve 31, that is, a control signal corresponding to the operation content of the operating device 26. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0057] Furthermore, the control valve (directional valve) built into the control valve 17 that drives each hydraulic actuator HA can also be a solenoid type. In this case, the operating signal output from the operating device 26 can be directly input to the control valve 17 (i.e., a solenoid type control valve).

[0058] Furthermore, as described above, part or all of the hydraulic actuator HA can be replaced by an electric actuator. In this case, the controller 30 can output control commands corresponding to the operation content of the operating device 26 or the remote operation content specified by the remote operation signal to the electric actuator or the driver that drives the electric actuator. Moreover, in the case of remotely operating the excavator 100, the operating device 26 can be omitted.

[0059] Hydraulic control valves 31 are provided for each driven component (hydraulic actuator HA) of the operating device 26, and for each driving direction of the driven component (hydraulic actuator HA) (e.g., the lifting and lowering direction of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA used to drive the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6. The hydraulic control valves 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and configured to change their flow area (i.e., the cross-sectional area through which the working oil can flow). Thus, the hydraulic control valves 31 can output a predetermined pilot pressure to the secondary pilot line 27B using the working oil supplied to the pilot pump 15 through the pilot line 25B. Therefore, the hydraulic control valves 31 can indirectly act on the control valve 17 with a predetermined pilot pressure corresponding to the control signal from the controller 30 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Therefore, for example, the controller 30 can supply pilot pressure from the hydraulic control valve 31 to the control valve 17 corresponding to the operation command corresponding to the automatic operation function, thereby realizing the operation of the excavator 100 based on the automatic operation function.

[0060] Furthermore, the controller 30 can also control the hydraulic control valve 31 to achieve remote operation of the excavator 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 200 to the hydraulic control valve 31 via the communication device 60. As a result, the controller 30 can supply pilot pressure corresponding to the content of the remote operation from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the excavator 100 based on the operator's remote operation.

[0061] Furthermore, when the operating device 26 is electric, the controller 30 can directly supply pilot pressure from the hydraulic control valve 31 to the control valve 17 corresponding to the operation content (operation signal) of the operating device 26, thereby realizing the operation of the excavator 100 based on the operator's operation.

[0062] The shuttle valve 32 has two inlet ports and one outlet port, and outputs working oil with the higher pilot pressure of the pilot pressure input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) of the operating device 26, and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA used to drive the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the aforementioned lever or pedal device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve in the control valve 17 via the pilot line 27. The corresponding control valve refers to the control valve of the hydraulic actuator HA, which drives the lever or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 can respectively apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve independently of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Thus, the controller 30 can control the movement of the driven components (lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6) independently of the operator's operating state of the operating device 26, thereby realizing automatic operation or remote operation functions.

[0063] A hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to change its flow area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. Thus, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to a level lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply the desired pilot pressure to the pilot port of the control valve within the control valve 17, regardless of the operation of the operating device 26. Thus, the controller 30 can, for example, more appropriately realize the automatic operation function or remote operation function of the excavator 100 by controlling the hydraulic control valve 33 in addition to controlling the hydraulic control valve 31.

[0064] like Figure 4 As shown, the user interface system of the excavator 100 includes an operating device 26, an output device 50, and an input device 52.

[0065] The output device 50 outputs various information to users of the excavator 100 (e.g., the operator of the cab 10 or an external remote operator) or people around the excavator 100 (e.g., workers or drivers of work vehicles).

[0066] For example, output device 50 includes lighting equipment or display devices that visually output various information. Lighting equipment may be, for example, warning lights (indicator lights). Display devices may be, for example, liquid crystal displays (LCDs) or organic EL (electroluminescence) displays. For example, such as... Figure 2 As shown, lighting equipment or display devices can be installed inside the cab 10 and output various information to the operator inside the cab 10 in a visual manner. Furthermore, lighting equipment or display devices can also be installed on the side of the upper rotating body 3, for example, and output various information to workers around the excavator 100 in a visual manner.

[0067] Furthermore, the output device 50 may also include a sound output device that outputs various information audibly. The sound output device may include, for example, a buzzer or a loudspeaker. The sound output device may be installed either inside or outside the cab 10, and output various information audibly to the operator inside the cab 10 or to people (workers, etc.) around the excavator 100. Furthermore, the output device 50 may also include a device that outputs various information tactilely, such as through vibration of the control seat.

[0068] Input device 52 receives various inputs from the user of excavator 100. Signals corresponding to the inputs received through input device 52 are fed into controller 30. For example, such as... Figure 2 As shown, the input device 52 is installed inside the cab 10 and receives input from the operator inside the cab 10. Alternatively, the input device 52 may be installed on the side of the upper rotating body 3 and receive input from personnel around the excavator 100.

[0069] For example, input device 52 includes a mechanical input device that accepts mechanically operated input from a user. The mechanical input device may include a touch panel mounted on the display device, a touchpad disposed around the display device, a push-button switch, a lever, a toggle switch, a rotary switch disposed on the operating device 26 (lever device), etc.

[0070] Furthermore, the input device 52 may also include a voice input device that accepts the user's voice input. The voice input device may include, for example, a microphone. The input device 52 may also include a gesture input device that accepts the user's gesture input. The gesture input device may include, for example, a camera device that captures the state of the user's gestures. The input device 52 may also include a biometric input device that accepts the user's biometric input. Biometric input may include, for example, the input of the user's fingerprint, iris, or other biometric information.

[0071] like Figure 4 As shown, the communication system of the excavator 100 involved in this embodiment includes a communication device 60.

[0072] The communication device 60 connects to an external communication line NW and communicates with a device separately installed from the excavator 100. This separate device from the excavator 100 may include, in addition to a device located external to the excavator 100, a portable terminal device (mobile terminal) brought into the cab 10 by the user of the excavator 100. The communication device 60 may include, for example, a mobile communication module based on standards such as 4G (4th Generation) or 5G (5th Generation). Furthermore, the communication device 60 may also include, for example, a satellite communication module. Additionally, the communication device 60 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. Furthermore, in the case of multiple connectable communication lines NW, the communication device 60 may include multiple communication devices depending on the type of communication line NW.

[0073] For example, the communication device 60 communicates with external devices such as the remote operation support device 200 within the work site via a local communication line established at the work site. The local communication line may be, for example, a mobile communication line based on local 5G (so-called local 5G) established at the work site or a local area network based on WiFi 6. Furthermore, the communication device 60 can also communicate with the remote operation support device 200 located outside the work site via a wide-area communication line (i.e., a wide area network) that includes the work site.

[0074] like Figure 4 As shown, the control system of the excavator 100 includes a controller 30. Furthermore, the control system of the excavator 100 according to this embodiment includes an operating pressure sensor 29, a camera device 40, and sensors S1 to S9.

[0075] The controller 30 performs various controls related to the excavator 100. The functions of the controller 30 can be implemented using any hardware or any combination of hardware and software. For example, ... Figure 4 As shown, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D connected via a bus BS1.

[0076] Auxiliary storage device 30A is a non-volatile storage unit that stores the installed program and necessary files or data. Auxiliary storage device 30A is, for example, EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory. For example, when a program start instruction is present, memory device 30B loads the program from auxiliary storage device 30A so that CPU 30C can read the program. Memory device 30B is, for example, SRAM (Static Random Access Memory).

[0077] CPU 30C, for example, executes a program loaded into memory device 30B and implements various functions of controller 30 according to the program's commands. Interface device 30D, for example, functions as a communication interface for connecting to communication lines inside excavator 100. Interface device 30D may also include multiple different types of communication interfaces depending on the type of communication line to be connected.

[0078] Furthermore, the interface device 30D functions as an external interface for reading data from or writing data to the storage medium. The storage medium can be, for example, a special tool connected to a connector located inside the cab 10 via a detachable cable. The storage medium can also be a common storage medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thus, programs implementing various functions of the controller 30 can be provided, for example, via a portable storage medium and installed in the auxiliary storage device 30A of the controller 30. Furthermore, the program can be downloaded from another computer located outside the excavator 100 via the communication device 60 and installed in the auxiliary storage device 30A.

[0079] In addition, some of the functions of controller 30 can also be implemented by other controllers (control devices). That is, the functions of controller 30 can be implemented in a distributed manner by multiple controllers mounted on excavator 100.

[0080] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot-operated device 26, that is, the pilot pressure corresponding to the operating state of each driven component (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure corresponding to the operating state of each driven component (hydraulic actuator HA) in the operating device 26, detected by the operating pressure sensor 29, is input into the controller 30.

[0081] Furthermore, when the operating device 26 is electrically powered or when the operating device 26 is omitted, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating status of each driven component operated by the operating device 26 based on the operating signals received from the operating device 26.

[0082] The camera device 40 captures images of the area surrounding the excavator 100. The camera device 40 may be, for example, a monocular camera. Furthermore, the camera device 40 may also be, for example, a stereo camera, a ToF (Time of Flight) camera, or a depth camera, capable of acquiring not only two-dimensional image information but also three-dimensional information including information related to the distance to objects projected in the image or the depth of the image—a 3D camera.

[0083] For example, such as Figure 2 As shown, the camera device 40 includes cameras 40F, 40B, 40L, and 40R. Camera 40F captures the front of the upper rotating body 3. Camera 40B captures the rear of the upper rotating body 3. Camera 40L captures the left side of the upper rotating body 3. Camera 40R captures the right side of the upper rotating body 3. Thus, the camera device 40 can capture the entire circumference of the excavator 100 when viewed from above, that is, the range spanning 360 degrees in the angular direction centered on the excavator 100. Hereinafter, cameras 40F, 40B, 40L, and 40R will sometimes be collectively referred to or individually as "camera 40X".

[0084] The output data of the camera device 40 (camera 40X) is received by the controller 30 via a one-to-one communication line or a vehicle network. Thus, for example, the controller 30 can monitor the surrounding conditions of the excavator 100 based on the output data of the camera 40X. Furthermore, some or all of the cameras 40B, 40L, and 40R can be omitted. Alternatively, a distance sensor (also called a "distance sensor") capable of acquiring information indicating the distance between the excavator 100 and surrounding objects can be installed on the excavator 100 instead of the camera device 40; or, in addition to the camera device 40, a distance sensor capable of acquiring information indicating the distance between the excavator 100 and surrounding objects can be installed on the excavator 100. The distance sensor can be, for example, a LiDAR (Light Detecting and Ranging) sensor, a millimeter-wave radar, or an ultrasonic sensor.

[0085] Sensor S1 is mounted on boom 4 and measures the posture state of boom 4. Sensor S1 outputs measurement data representing the posture state of boom 4. The posture state of boom 4 is, for example, the posture angle (hereinafter referred to as "boom angle") of the base end of the connection between boom 4 and upper rotating body 3 about the rotation axis. Sensor S1 may include, for example, a rotary potentiometer, rotary encoder, accelerometer, angular accelerometer, six-axis sensor, IMU (Inertial Measurement Unit), etc. The same applies to sensors S2 to S4. Furthermore, sensor S1 may include a cylinder sensor that detects the extension and retraction position of boom cylinder 7. The same applies to sensors S2 and S3. The output of sensor S1 (measurement data representing the posture state of boom 4) is input to controller 30. Thus, controller 30 can grasp the posture state of boom 4.

[0086] Sensor S2 is mounted on the boom 5 and measures the attitude state of the boom 5. Sensor S2 outputs measurement data representing the attitude state of the boom 5. The attitude state of the boom 5 is, for example, the attitude angle (hereinafter referred to as "boom angle") of the base end of the boom 5 connected to the boom 4 about the axis of rotation. The output of sensor S2 (the measurement data representing the attitude state of the boom 5) is input to controller 30. Thus, controller 30 can grasp the attitude state of the boom 5.

[0087] Sensor S3 is mounted on bucket 6 and measures the posture of bucket 6. Sensor S3 outputs measurement data representing the posture of bucket 6. The posture of bucket 6 is, for example, the posture angle (hereinafter referred to as "bucket angle") of the base end of the connection between bucket 6 and stick 5 about the axis of rotation. The output of sensor S3 (the measurement data representing the posture of bucket 6) is input to controller 30. Thus, controller 30 can grasp the posture of bucket 6.

[0088] Sensor S4 measures the posture of the excavator 100's body (e.g., the upper rotating body 3). Sensor S4 outputs measurement data representing the posture of the excavator 100's body. The posture of the excavator 100's body is, for example, its tilt relative to a predetermined reference plane (e.g., a horizontal plane). For example, sensor S4 is mounted on the upper rotating body 3 and measures the tilt angles (hereinafter referred to as "forward tilt angle" and "left-right tilt angle") of the excavator 100 about two axes in the forward and backward directions. The output of sensor S4 (the measurement data representing the posture of the excavator 100's body) is input to controller 30. Thus, controller 30 can grasp the posture (tilt state) of the body (upper rotating body 3).

[0089] Sensor S5 is mounted on the upper rotating body 3 and measures the rotation state of the upper rotating body 3. Sensor S5 outputs measurement data indicating the rotation state of the upper rotating body 3. Sensor S5, for example, measures the rotational angular velocity or rotational angle of the upper rotating body 3. Sensor S5 may include, for example, a gyroscope sensor, a resolver, a rotary encoder, etc. The output of sensor S5 (measurement data indicating the rotation state of the upper rotating body 3) is input to controller 30. Thus, controller 30 can grasp the rotation state of the upper rotating body 3, such as the rotational angle.

[0090] The controller 30 can determine (estimate) the position of the front end (bucket 6) of the auxiliary device AT based on the outputs of sensors S1 to S5. Furthermore, if sensor S4 includes a gyroscope sensor capable of detecting angular velocities around three axes, a 6-axis sensor, an IMU, etc., the rotational state (e.g., rotational angular velocity) of the upper rotating body 3 can be detected based on the detection signal from sensor S4. In this case, sensor S5 can be omitted.

[0091] Sensor S6 measures the position of excavator 100. Sensor S6 can measure the position in world (global) coordinates or in local coordinates at the work site. In the former case, sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, sensor S6 is a transceiver capable of communicating with a device serving as a reference for the position at the work site and outputting a signal corresponding to the position relative to the reference. The output of sensor S6 is fed into controller 30.

[0092] Sensor S7 measures the pressure (cylinder pressure) of the oil chamber of boom cylinder 7. Sensor S7 may include, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of boom cylinder 7 and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side. The output of sensor S7 (i.e., the measured data of the cylinder pressure of boom cylinder 7) is input to controller 30.

[0093] Sensor S8 measures the pressure (cylinder pressure) of the oil chamber in the boom cylinder 8. Sensor S8 may include, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of the boom cylinder 8 and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side of the boom cylinder 8. The output of sensor S8 (i.e., the measured cylinder pressure data of the boom cylinder 8) is input to controller 30.

[0094] Sensor S9 measures the pressure (cylinder pressure) of the oil chamber of bucket cylinder 9. Sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of bucket cylinder 9 and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side of bucket cylinder 9. The output of sensor S9 (i.e., the measured data of the cylinder pressure of bucket cylinder 9) is input to controller 30.

[0095] The controller 30 can determine the load state acting on the auxiliary device AT based on the outputs of sensors S7 to S9. The load acting on the auxiliary device AT includes, for example, the reaction force of sand and soil from the working surface acting on the bucket 6, or the weight of the sand and soil contained in the bucket 6. Furthermore, some or all of sensors S1 to S9 may be omitted as needed. Other sensors capable of determining the state of the excavator 100 may also be mounted on the excavator 100. For example, the excavator 100 may have a direction sensor capable of detecting its own orientation. The direction sensor may be, for example, an electronic compass including a geomagnetic sensor.

[0096] Next, refer to Figure 5 The structure of the remote operation support device 200 will be described. Figure 5 This is a block diagram illustrating an example of the structure of the remote operation support device 200.

[0097] The functions of the remote operation support device 200 are implemented through any hardware or any combination of hardware and software. For example, such as Figure 5 As shown, the remote operation support device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed computing device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. They are connected via a bus BS2.

[0098] External interface 201 functions as an interface for reading data from or writing data to storage medium 201A. Storage medium 201A includes, for example, floppy disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray Discs), SD memory cards, and USB storage devices.

[0099] Therefore, the remote operation support device 200 can read various data used in the processing through the storage medium 201A and store it in the auxiliary storage device 202, or install programs that implement various functions. In addition, the remote operation support device 200 can also obtain various data or programs used in the processing from external devices through the communication interface 206.

[0100] Auxiliary storage device 202 stores various installed programs and files or data required for various processes. Auxiliary storage device 202 may include, for example, HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. When a program startup instruction is present, memory device 203 reads the program from auxiliary storage device 202 and stores it. Memory device 203 may include, for example, DRAM (Dynamic Random Access Memory) or SRAM.

[0101] CPU 204 executes various programs loaded from auxiliary storage device 202 into memory device 203, and implements various functions related to remote operation support device 200 according to the programs. High-speed computing device 205 works in conjunction with CPU 204 to perform computational processing at a relatively high speed. High-speed computing device 205 may include, for example, a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array). Alternatively, depending on the required computational processing speed, high-speed computing device 205 may be omitted.

[0102] The communication interface 206 serves as an interface for communication with external devices. Thus, the remote operation support device 200 can communicate with external devices such as the excavator 100 via the communication interface 206. Furthermore, the communication interface 206 can have multiple types of communication interfaces depending on the communication method with the device to be connected.

[0103] Input device 207 accepts various inputs from the user. Input device 207 includes a remote operation device for remotely operating the excavator 100. Input device 207 may include, for example, an input device (mechanical input device) that accepts mechanical operation inputs from the user. The remote operation device may be a mechanical input device. Mechanical input devices may include, for example, buttons, toggle switches, joysticks, keyboards, mice, touch panels mounted on display device 208, or touchpads separate from display device 208.

[0104] Furthermore, the input device 207 may include a voice input device capable of accepting voice input from a user. The voice input device may include, for example, a microphone capable of collecting the user's voice. The input device 207 may also include a gesture input device capable of accepting gesture input from a user. The gesture input device may include, for example, a camera capable of capturing the state of the user's gestures. The input device 207 may also include a biometric input device capable of accepting biometric input from a user. The biometric input device may include, for example, a camera capable of acquiring image data containing information related to the user's fingerprint or iris.

[0105] Display device 208 displays information screens or operation screens to the user of remote operation support device 200. Display device 208 may be, for example, a liquid crystal display (LCD) or an organic EL (Electroluminescence) display. Sound output device 209 transmits various information to the user of remote operation support device 200 via sound. Sound output device 209 may be, for example, a buzzer, alarm, or speaker.

[0106] Next, besides Figures 1-5 In addition, also refer to Figures 6 to 11 This section explains the function of setting the restricted area for the excavator 100.

[0107] Figure 6 This is a function block diagram illustrating an example of the restricted area setting function of the excavator 100. Figure 7 This is a diagram showing an example of the environment surrounding the excavator 100. Figure 8 This is a diagram illustrating an example of how a language model (LM) is assigned. Figure 9 This is a diagram illustrating an example of a combination of prompts and outputs from a language model. Figure 10 This is another example of the environment surrounding the excavator 100. Figure 11 This is a diagram showing an example of the restricted area RA of the excavator 100.

[0108] For example, such as Figure 6 As shown, the controller 30 of the excavator 100 includes an information acquisition unit 301, an instruction acquisition unit 302, a speech recognition unit 303, a prompt generation unit 304, a recall unit 305, a region setting unit 306, an action control unit 307, and a display control unit 308. Each of these units in the controller 30 represents a function of the controller 30, implemented, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it via the CPU 30C. Furthermore, a language model LM is provided externally to the excavator 100.

[0109] The language model LM is, for example, a large-scale language model (LLM). The language model LM is installed on an external device (e.g., a server device) that is connected to the excavator 100 via a communication device 60 in a manner that enables mutual communication. The language model LM is, for example, GPT-4.

[0110] The information acquisition unit 301 acquires, for example, the detection results of sensors S1 to S9. The detection results of sensors S1 to S9 include information related to the posture of the auxiliary device AT and the surrounding environment of the excavator 100. Specifically, the information acquisition unit 301 acquires, for example, the detection results of the boom angle sensor S1, the stick angle sensor S2, the bucket angle sensor S3, the body tilt sensor S4, and the swing sensor S5 as information related to the posture of the auxiliary device AT.

[0111] Furthermore, the information acquisition unit 301 acquires, for example, the detection results of objects existing around the excavator 100 from the camera device 40 or the ranging sensor as information related to the environment around the excavator 100. Additionally, the information acquisition unit 301 can detect objects around the excavator 100 by acquiring image data from the camera device 40 or by acquiring the distance and direction of objects around the excavator 100 from the ranging sensor.

[0112] Specifically, the information acquisition unit 301 detects objects of surveillance from the camera images captured by the camera device 40, for example, by appropriately applying known image processing techniques such as semantic segmentation or machine learning. Objects of surveillance include, for example, workers. Furthermore, objects of surveillance may include other obstacles present around the excavator 100. Other obstacles include, for example, other moving objects present at the excavator 100's work site, such as other operating machinery or vehicles. Other obstacles may include, for example, specific fixed objects present at the excavator 100's work site, such as utility poles, fences, and traffic cones (also known as colored traffic cones (registered trademark)). Furthermore, other obstacles may include specific terrain shapes present at the excavator 100's work site, such as ditches, holes, and piles of sand.

[0113] Furthermore, the information acquisition unit 301 can, for example, acquire information related to the construction drawings of the excavator 100's work site as information related to the surrounding environment of the excavator 100. The information related to the construction drawings includes, for example, information related to the construction drawings. Figure 7 The road RD shown Figure 10 The diagram shows the shape, size, and location of objects such as the buried pipe UP, the electrical wire EW, and the slope shoulder TS, which are installed at the work site. Information related to the construction drawings is acquired by the information acquisition unit 301 via the communication interface 206 and stored in the auxiliary storage device 202.

[0114] The information acquisition unit 301 outputs the acquired information to, for example, the language unit 303 and the region setting unit 306.

[0115] The instruction acquisition unit 302 acquires instructions (hereinafter referred to as "instructions") related to the setting of the restricted area RA, input by the operator in natural language. The restricted area RA is the area in the work site where the intrusion of the excavator 100 or its operating speed is restricted. Specifically, in Figure 7 and Figure 10 In the example shown, the restricted area RA includes the restricted areas RA1 to RA10 defined by virtual security barriers SB1 to SB10. Security barriers SB1 to SB10 are sometimes also referred to as "E-FENCE (registered trademark)," "virtual fence," or "security fence."

[0116] Input from the operator's instructions is received by the input device 52 provided in the cab 10 when the operator is in the cab 10, and by the input device 207 of the remote operation support device 200 when the operator is operating remotely.

[0117] Instructions given in natural language, for example, are instructions input in natural language by the operator's voice. In this case, the instruction acquisition unit 302 can acquire text (article) data corresponding to the instructions given in natural language by applying known speech recognition technology based on the voice input data received by the input device 52 or the input device 207.

[0118] Furthermore, instructions given in natural language can also refer to instructions input by the operator using text input via input device 52 or input device 207, which are capable of character input such as keyboards or touch panels. In this case, the instruction acquisition unit 302 can acquire the text (article) data received by input device 52 or input device 207 as instructions given in natural language.

[0119] The language translation unit 303 translates the information input from the information acquisition unit 301 related to the posture of the auxiliary device AT and the environment surrounding the excavator 100 into natural language. As described above, the information related to the environment surrounding the excavator 100 translated by the language translation unit 303 includes, for example, information related to the construction drawings of the excavator 100's work site.

[0120] The languageization unit 303, for example, translates information acquired by the information acquisition unit 301, such as boom angle, stick angle, bucket angle, the configuration of monitored objects around the excavator 100, and construction drawing information, into natural language. Specifically, the languageization unit 303 translates information acquired by the information acquisition unit 301 into a template of a predefined text.

[0121] The template for text indicating the posture of the auxiliary device AT is, for example, setting "aaa" for the boom angle, "bbb" for the stick angle, and "ccc" for the bucket angle, thus specifying it in the form of "Boom angle is aaa degrees, stick angle is bbb degrees, bucket angle is ccc degrees," etc. The template for text indicating the configuration of the monitored object is, for example, setting "ddd" for the position information of the monitored object, and "eee" for the type or name of the monitored object, thus specifying it in the form of "eee is at ddd." or "eee is at ddd." The same applies to information on the construction drawings. The position information of the object is, for example, the direction and distance or coordinates relative to the excavator 100.

[0122] like Figure 7 As shown, regarding the person P detected by the camera device 40, the speech unit 303, for example, uses an azimuth angle of 0° in front of the excavator 100 to speech-code it as "There is a person at an azimuth angle of 285 degrees and a distance of 5 meters, and at an azimuth angle of 350 degrees and a distance of 6 meters." Furthermore, as... Figure 10 As shown, for example, regarding the buried pipe UP recorded in the construction drawing, the language section 303 languageizes it as "There is a buried pipe 5 meters to the left and 3 meters underground."

[0123] The prompt word generation unit 304 generates prompt words input to the language model LM based on the natural language-based instructions acquired by the instruction acquisition unit 302 and the information expressed by the languageization unit 303. Specifically, the prompt word generation unit 304 generates prompt words for setting a restriction area RA corresponding to the instructions acquired by the instruction acquisition unit 302, based on information expressed by the languageization unit 303 related to the posture of the excavator 100's auxiliary device AT and the surrounding environment.

[0124] For example, the prompt generation unit 304 generates multiple example questions and pre-assigns them to the language model LM via the calling unit 305. Each example question is defined by a combination of information related to the preconditions (i.e., constraints), namely the posture of the excavator 100's auxiliary device AT and the surrounding environment, the instructions of the example question, and the correct answer to be output. Thus, the language model LM is able to understand (learn) the output format for the instructions given by the prompt words.

[0125] For example, such as Figure 8 As shown, several examples are given, each representing a combination of the excavator 100's surrounding environment, user (i.e., operator) instructions, and the excavator 100's control information (control commands). Thus, as... Figure 9As shown, the language model LM can output control instructions to set a virtual safety barrier SB for the restricted area RA, based on the prompt words generated by the prompt word generation unit 304 and the operator's instructions regarding the surrounding environment of the excavator 100. Furthermore, by asking follow-up questions when the instructions are unclear, the possibility of setting an inappropriate area as the restricted area RA can be suppressed. The number of example questions provided is arbitrary, but preferably more than the number of control instructions corresponding to the actions that the excavator 100 should perform. This is because the language model LM can understand all control instructions.

[0126] The calling unit 305 calls the language model LM through a specified API (Application Programming Interface) for example, and inputs the prompt words generated by the prompt word generation unit 304 into the language model LM to obtain its output (answer).

[0127] The region setting unit 306 sets the restricted region RA based on the output of the language model LM obtained by the calling unit 305. Specifically, the region setting unit 306 sets the restricted region RA by generating a virtual security barrier SB according to the control instructions output from the language model LM.

[0128] Specifically, in Figure 7 In the example shown, the information acquisition unit 301 acquires information about the surrounding environment of the excavator 100, including multiple traffic cones CN to the right front, the road RD to the rear, multiple people P to the left front, and the dump truck DT to the right. The speech generation unit 303 speech-izes the information acquired by the information acquisition unit 301 and inputs it into the prompt word generation unit 304.

[0129] Furthermore, the instruction acquisition unit 302 acquires the instructions given by the user of the excavator 100 in natural language and inputs them into the prompt word generation unit 304. The prompt word generation unit 304 generates a prompt word based on the input information and instructions and inputs it into the calling unit 305. The calling unit 305 inputs the prompt word generated by the prompt word generation unit 304 into the language model LM, and obtains the control instructions, as output by the region setting unit 306, for setting the restriction region RA.

[0130] As a result, the zone setting unit 306, for example, responds to the user's instruction to "set a barrier in the cone area." Figure 7 As shown, a safety barrier SB1 is generated based on the convex hull of the positions of multiple cones CN. Furthermore, the area setting unit 306 sets the area surrounded by the safety barrier SB1 as the restricted area RA1.

[0131] Furthermore, the area setting unit 306 generates, for example, a barrier based on the user's instruction to "set up a barrier with the two cones in front of me as a reference." Figure 7 The vertical plane parallel to the straight line connecting the two cones CN shown serves as safety barrier SB2. Furthermore, the area setting unit 306 sets the area on the side opposite to the excavator 100 as restricted area RA2, with safety barrier SB2 as the boundary.

[0132] Furthermore, the area setting unit 306 generates, for example, a barrier based on the user's instruction to "set the barrier with the right-hand cone as a reference." Figure 7 The vertical plane parallel to the straight line connecting the two right-side cones CN shown in the diagram serves as safety barrier SB3. Furthermore, the area setting unit 306 sets the area on the side opposite to the excavator 100 as restricted area RA3, with safety barrier SB3 as the boundary.

[0133] Furthermore, the area setting unit 306, for example, generates a barrier based on the user's instruction to "set a barrier at the position of the inner cone." Figure 7 The vertical plane parallel to the straight line connecting the two inner cones CN shown in the diagram serves as a safety barrier SB4. Furthermore, the area setting unit 306 sets the area on the side opposite to the excavator 100 as a restricted area RA4, with the safety barrier SB4 as the boundary.

[0134] Furthermore, the area setting unit 306 generates, for example, a system based on the user's instruction to "prevent the excavator from entering the road." Figure 7 The vertical plane parallel to the road RD and adjacent to the side edge of the road RD is used as a safety barrier SB5. Furthermore, the area setting unit 306 sets the area on the side opposite to the excavator 100 as a restricted area RA5 with the safety barrier SB5 as the boundary.

[0135] Furthermore, the area setting unit 306 generates a surrounding area based, for example, on the user's instruction to "prevent the backhoe excavator from hitting people." Figure 7 The cylindrical or hemispherical surface of the person P shown serves as the safety barrier SB6. Furthermore, the area setting unit 306 sets the area inside the safety barrier SB6 as the restricted area RA6.

[0136] Furthermore, the area setting unit 306 generates a surrounding area based, for example, on the user's instruction that "I want to load soil and sand without hitting the dump truck." Figure 7 The cuboid of the dump truck DT shown serves as the safety barrier SB7. Furthermore, the area setting unit 306 sets the area inside the safety barrier SB7 as the restricted area RA7.

[0137] Furthermore, the area setting unit 306, for example, generates a barrier based on the user's instruction to "set a barrier at the depth of the buried pipe." Figure 10 The horizontal plane at a depth D above the ground surface GS adjacent to the buried pipe UP is designated as safety barrier SB8. Furthermore, the area setting unit 306 designates the area below safety barrier SB8 as restricted area RA8.

[0138] Furthermore, the zone setting unit 306 generates, for example, a barrier based on the user's instruction to "set up a barrier at the height of the power line." Figure 10 The horizontal plane at a height H above the ground surface GS adjacent to the lower part of the wire EW shown is designated as safety barrier SB9. Furthermore, the area setting unit 306 designates the area above safety barrier SB9 as restricted area RA9.

[0139] Furthermore, the area setting unit 306 generates, for example, a barrier based on the user's instruction, "Set up a barrier to prevent falling off the slope shoulder." Figure 10 The vertical surface adjacent to the front side of the slope shoulder TS shown is used as a safety barrier SB10. Furthermore, the area setting unit 306 sets the area on the side opposite to the excavator 100 as a restricted area RA10 with the safety barrier SB10 as the boundary.

[0140] Thus, in this example, the controller 30 sets a restricted area RA for the excavator 100 based on the results of parsing the language model LM's instructions to the operator based on natural language and information related to the surrounding environment of the excavator 100 that has been verbalized in natural language. Therefore, the controller 30 can limit the intrusion or movement speed of the excavator 100 according to the operator's natural language-based instructions that match the surrounding environment of the excavator 100.

[0141] Furthermore, the area setting unit 306 can also set the restricted area RA of the excavator 100 based on the result of parsing the instructions acquired by the instruction acquisition unit 302 using the language model LM, the information acquired by the information acquisition unit 301, and the posture of the excavator 100. Specifically, the area setting unit 306, for example, applies the posture of the auxiliary device AT acquired by the information acquisition unit 301 to the control command obtained from the language model LM based on the user's instruction "Set up a barrier at the height of the boom." Thus, the area setting unit 306 can generate a safety barrier SB at the height of the boom 4 based on the posture of the auxiliary device AT, and set the upper side of the safety barrier SB as the restricted area RA.

[0142] The motion control unit 307 stops the excavator 100 from moving before it enters the restricted area RA set by the area setting unit 306, or limits the speed of the excavator 100 when it enters the restricted area RA. Specifically, the motion control unit 307 outputs a control signal to the hydraulic control valve 31 or the hydraulic control valve 33 based on the position of the excavator 100, the posture of the auxiliary device AT, and information about the restricted area RA, thereby stopping the movement of the excavator 100 or limiting its speed.

[0143] The display control unit 308 displays the restricted area RA set by the area setting unit 306 on the display device 50A. Specifically, the display control unit 308, for example, outputs a control signal to the display device 50A to overlay the image representing the restricted area RA onto the image displayed on the camera device 40 on the display device 50A. Hereinafter, refer to Figure 11 An example of a screen 41 displayed on the display device 50A will be described.

[0144] Figure 11 This diagram illustrates a display example of the restricted area RA of the excavator 100. The screen 41 displayed on the display device 50A includes display areas 41A to 41E. The display areas 41A to 41E are arranged sequentially from top to bottom in the vertical direction.

[0145] Display area 41A is located at the upper part of screen 41. In display area 41A, regardless of the control mode selected by controller 30, fixed display content is displayed. Display area 41A includes information display areas 41a-41e and 41g-41k.

[0146] The current date and time are displayed in information display area 41a. The currently selected operating mode of the excavator 100 is displayed in information display area 41b. An image representing the currently installed termination attachment is displayed in information display area 41c. Information related to the fuel consumption rate (fuel consumption) of the excavator 100 is displayed in information display area 41d. Information display area 41d includes, for example, an information display area 41d1 displaying life-cycle average fuel consumption or interval average fuel consumption, and an information display area 41d2 displaying instantaneous fuel consumption. Information display area 41e displays information indicating the control status of the engine 11.

[0147] The information display area 41g displays the current temperature of the coolant in engine 11. The information display area 41h displays the remaining fuel in the fuel tank. The information display area 41i displays the operating mode corresponding to the engine speed of engine 11. The information display area 41j displays the remaining urea solution in the urea solution tank. The information display area 41k displays the temperature of the hydraulic drive system's working oil.

[0148] The peripheral image display area 41n is displayed in display areas 41B and 41C. The peripheral image display area 41n displays an image (hereinafter referred to as "peripheral image") representing the state of the periphery of the excavator 100 based on the camera image of the camera device 40. The peripheral image display area 41n includes peripheral image display areas 41n1 to 41n3.

[0149] The peripheral image display area 41n1 is displayed in the display area 41B in such a way that it is adjacent to the information display area 41d contained in the display area 41A.

[0150] In this example, a bird's-eye view image FV, generated from the camera image of the camera device 40, showing the perimeter of the excavator 100 from a top-down perspective, is displayed in the peripheral image display area 41n1. Furthermore, an excavator image GE, which simulates a top-down view of the excavator 100, is also displayed in the peripheral image display area 41n1. The excavator image GE and the bird's-eye view image FV are arranged in the peripheral image display area 41n1 such that their positional relationship corresponds to the positional relationship between the excavator 100 and the camera range included in the bird's-eye view image FV.

[0151] The peripheral image display areas 41n2 and 41n3 are displayed in the display area 41C below the peripheral image display area 41n1. The peripheral image display areas 41n2 and 41n3 are arranged adjacent to the left and right portions of the display area 41C with the center in the left-right direction as the reference.

[0152] In this example, the rear image BM, representing the state behind the excavator 100, is displayed in the peripheral image display area 41n2, and the right image RM, representing the state to the right of the excavator 100, is displayed in the peripheral image display area 41n3. The rear image BM and the right image RM correspond to the camera images of camera 40B and camera 40R, respectively.

[0153] In this example, the bird's-eye view image FV, overlaid on the surrounding image display area 41n1, displays safety barriers SB3 and SB5, representing the boundaries of restricted areas RA3 and RA5. Furthermore, the rear image BM, overlaid on the surrounding image display area 41n2, displays safety barrier SB5, representing the boundary of restricted area RA5, and the right image RM, overlaid on the surrounding image display area 41n3, displays safety barrier SB3, representing the boundary of restricted area RA3.

[0154] The display area 41D includes information display areas 41f and 41m. Information display area 41f is positioned below the surrounding image display area 41n2. Information display area 41f displays the cumulative operating time of engine 11. Information display area 41m is positioned below the surrounding image display area 41n3 and to the right of information display area 41f. Information display area 41m displays the operating status of the air conditioner. Information display area 41m includes information display areas 41m1 to 41m4. Information display area 41m1 displays the current location of the air outlet used in the air supply from the air conditioner. Information display area 41m2 displays the current operating mode of the air conditioner. Information display area 41m3 displays the current set temperature of the air conditioner. Information display area 41m4 displays the current set airflow of the air conditioner.

[0155] Display area 41E is located at the lower part of screen 41. In display area 41E, fixed display content is shown regardless of the control mode selected by controller 30. Specifically, display area 41E displays a tab group 41q containing the operational elements for selecting a control mode applicable to controller 30 from multiple control modes. For example, the operator can operate tab group 41q using touch panel 80 as input device 52. Furthermore, the operator can operate tab group 41q using a switch attached to display device 50A as input device 52.

[0156] Tab group 41q includes tabs 41q1 to 41q6. Tabs 41q1 to 41q6 are arranged sequentially from left to right. Tab 41q1 is an operation icon used for settings related to screen 41. Tabs 41q2 to 41q5 are operation icons corresponding to four of the multiple control modes. Thus, by using the touch panel 80 or similar means to select and confirm the operation of any of the tabs 41q2 to 41q5, the operator can select a control mode suitable for the controller 30 from multiple control modes.

[0157] Next, refer to Figure 12 An example of the processing procedure for setting the restricted area RA of excavator 100 will be explained. Figure 12 This is a flowchart illustrating an example of the process for setting the restricted area RA of an excavator 100. In this example, the controller 30 has... Figure 6 The parts shown are the premise.

[0158] For example, if an operator's instructions based on natural language are input into input device 52 or input device 207, the controller 30 of the excavator 100 will start... Figure 12The processing flow shown executes a process P1 that retrieves the text of the instruction. In this process P1, the instruction retrieval unit 302 retrieves the text of the operator's instruction given in natural language from the input device 52 or the input device 207.

[0159] Next, the controller 30 executes processing P2, which acquires information related to the posture of the auxiliary device AT and the surrounding environment of the excavator 100. In this processing P2, the information acquisition unit 301 calculates the posture of the auxiliary device AT, for example, based on the detection results from the boom angle sensor S1, stick angle sensor S2, bucket angle sensor S3, body tilt sensor S4, slewing sensor S5, and sensor S6. Furthermore, the information acquisition unit 301 detects, for example, objects around the excavator 100, their direction and distance to the objects, and the position coordinates of the objects, based on the output of the camera device 40 or the distance sensor. Additionally, the information acquisition unit 301 acquires, for example, information from the construction drawing of the work site where the excavator 100 is operating.

[0160] Next, the controller 30 executes a process P3 that converts the acquired information into language. In this process P3, the language conversion unit 303 converts the information acquired by the information acquisition unit 301 into language using natural language.

[0161] Next, the controller 30 executes processing P4, which involves the translation instruction acquisition unit 302 acquiring the text of the instruction and the languageization unit 303 translating the information. In this processing P4, the prompt word generation unit 304 also functions as a translation unit that translates the instruction text and languageization information into the standard language of the language model LM, i.e., the specific language corresponding to the language model LM. Specifically, if the standard language of the language model LM is English and the instruction text and languageization information are in a language other than English, the prompt word generation unit 304 translates the instruction text and languageization information into English using machine translation. Alternatively, processing P4 can be omitted.

[0162] Next, the controller 30 executes the prompt word generation process P5. In this process P5, the prompt word generation unit 304 generates prompt words that are input to the language model LM based on the indicated text and language information or their machine translation.

[0163] Next, the controller 30 executes the process P6 of calling the language model LM. In this process P6, the calling unit 305 calls the language model LM, inputs the prompt words generated by the prompt word generation unit 304 in the previous process P5 into the language model LM, and inputs the control commands obtained from the language model LM into the region setting unit 306.

[0164] Next, the controller 30 executes the process P7 of setting the restricted area RA. In this process P7, the area setting unit 306 sets the restricted area RA according to the control instructions obtained from the language model LM.

[0165] Next, the controller 30 executes process P8, which restricts the movement of the excavator 100 and displays the screen. In process P8, the motion control unit 307 stops the excavator 100 before it enters the restricted area RA, or outputs a control signal to the hydraulic control valve 31 to limit the speed of the excavator 100 if it has already entered the restricted area RA. Furthermore, the display control unit 308 outputs a control signal to the display device 50A, displaying the boundary of the restricted area RA, i.e., the safety barrier SB, on the screen 41 of the display device 50A. Then, the controller 30 terminates the process. Figure 12 The processing flow is shown below.

[0166] As explained above, the excavator 100, which is an embodiment of the working machinery involved in this disclosure, includes a working accessory AT, an information acquisition unit 301, a speech recognition unit 303, an instruction acquisition unit 302, and a zone setting unit 306. The information acquisition unit 301 acquires information related to the posture of the accessory AT and the surrounding environment. The speech recognition unit 303 translates the information acquired by the information acquisition unit 301 into natural language. The instruction acquisition unit 302 acquires instructions given by the operator in natural language. The zone setting unit 306 sets a restricted area RA in the work site where intrusion or movement speed is limited. Furthermore, the zone setting unit 306 sets the restricted area RA based on the result obtained by parsing the instructions acquired by the instruction acquisition unit 302 and the speech recognition unit 303 using a language model LM, or based on the result obtained by parsing the instructions acquired by the instruction acquisition unit 302 using a language model LM and the information acquired by the information acquisition unit 301.

[0167] With this structure, the excavator 100 according to this embodiment can set a restricted area RA that limits the intrusion or operating speed of the excavator 100 based on the operator's natural language-based instructions. Therefore, compared to manually setting the restricted area RA while the operator checks the environment around the excavator 100, the restricted area RA can be set more easily. Thus, the excavator 100 according to this embodiment allows various personnel, including inexperienced operators and foreign operators, to work safely.

[0168] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the information acquisition unit 301 acquires information related to the posture of the auxiliary device AT and the surrounding environment based on the outputs of a position sensor that acquires the position information of the excavator 100, a posture sensor that detects the posture of the auxiliary device AT, and an external sensor that detects objects around the excavator 100. In this embodiment, the position sensor includes sensor S6, the posture sensor includes sensors S1 to S5, and the external sensor includes a camera device 40 or a ranging sensor.

[0169] With this structure, the excavator 100 according to this embodiment can acquire information related to the posture of the auxiliary device AT based on the output of the posture sensor, and acquire information related to the environment around the excavator 100 based on the output of the position sensor and the external sensor.

[0170] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 sets the restricted area RA based on the result obtained by the language model LM parsing the instructions acquired by the instruction acquisition unit 302 and the information of objects detected by external sensors.

[0171] Through this structure, such as Figure 7 As shown, the operator of the excavator 100 can set the restricted areas RA1 to RA7 by giving instructions in natural language, based on the positions of the cone CN, person P, road RD, dump truck DT, etc. detected by external sensors.

[0172] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 sets the restricted area RA based on the result obtained by parsing the instruction acquired by the instruction acquisition unit 302 by the language model LM, the position information of the excavator 100 acquired by the position sensor, and the posture of the auxiliary device AT acquired by the posture sensor.

[0173] With this structure, the operator of the excavator 100 can set the restricted area RA by giving instructions in natural language, based on the height of each part of the auxiliary device AT obtained from the posture of the auxiliary device AT.

[0174] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 acquires information about multiple objects based on the detection results of external sensors, and when the instruction acquisition unit 302 acquires an instruction to set a restriction area RA for these multiple objects, the restriction area RA is set based on the convex hull of the position of these multiple objects.

[0175] With this structure, the operator of the excavator 100 can give instructions using natural language, for example, such as... Figure 7 As shown, the restricted area RA1 can be set by the convex hull of the positions of multiple cones CN detected by external sensors of the excavator 100.

[0176] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, when the area setting unit 306 detects two objects by an external sensor and the instruction acquisition unit 302 acquires an instruction to set a restriction area RA based on the two objects, the restriction area RA is set on the side opposite to the excavator 100 based on the straight line connecting the two objects.

[0177] With this structure, the operator of the excavator 100 can give instructions using natural language, for example, such as... Figure 7 As shown, the restricted areas RA2, RA3, and RA4 can be set based on the straight line connecting the two cones CN detected by the external sensor of the excavator 100.

[0178] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 identifies a specific object based on the detection results of external sensors and obtains an instruction from the instruction acquisition unit 302 to set a restriction area RA for the specific object, and sets a restriction area RA in the area containing the specific object.

[0179] With this structure, the operator of the excavator 100 can give instructions using natural language, for example, such as... Figure 7 As shown, it is possible to set restricted areas RA6 and RA7 for areas containing people P or dump truck DT detected by external sensors of excavator 100.

[0180] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 sets the restriction area RA based on the location information of the movable object when it obtains the position information of the movable object based on the detection results of the external sensor and the instruction acquisition unit 302 obtains an instruction to set a restriction area RA for the movable object.

[0181] With this structure, the operator of the excavator 100 can give instructions using natural language, for example, in Figure 7 When the person P, cone CN, dump truck DT, etc., are moving, the restriction areas RA1 to RA4, RA6, and RA7 set on these objects can move together with these objects.

[0182] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the position information of the excavator 100 is obtained by a position sensor, and the posture of the auxiliary device AT is obtained by a posture sensor. At this time, if the instruction acquisition unit 302 obtains an instruction to set a restriction area RA based on the height of the excavator 100 or the auxiliary device AT, the area setting unit 306 sets the restriction area RA on the upper side or the lower side based on the height of the excavator 100 or the auxiliary device AT.

[0183] With this structure, the operator of the excavator 100 can give instructions in natural language, for example, by setting a restricted area RA on the upper or lower side based on the height of the boom 4, stick 5, bucket 6, etc., which are part of the auxiliary device AT.

[0184] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 expands or shrinks the restricted area RA according to the instruction obtained by the instruction acquisition unit 302 when the instruction is obtained to expand or shrink the restricted area RA.

[0185] With this structure, the operator of the excavator 100 can give instructions using natural language, for example, such as... Figure 7 As shown in the restricted areas RA1, RA6, and RA7, the restricted area RA surrounding the object can be expanded or reduced.

[0186] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 fixes the restricted area RA when the instruction acquisition unit 302 obtains an instruction to fix the set restricted area RA.

[0187] Through this structure, for example, such as Figure 7 As shown, even if the cone CN moves after setting the restriction area RA1 in an area surrounded by multiple cones CN, the restriction area RA1 set according to the position information of the cone CN before the movement can be maintained.

[0188] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the area setting unit 306 deletes the restricted area RA when the instruction acquisition unit 302 obtains an instruction to delete the set restricted area RA.

[0189] With this structure, when the operator of the excavator 100 deletes a restricted area RA after setting it, the operator can give a natural language instruction, which is then received by the instruction acquisition unit 302, and the restricted area RA is deleted by the area setting unit 306. Therefore, it is possible to easily delete the set restricted area RA.

[0190] Furthermore, in the excavator 100, which is an embodiment of the working machinery involved in this disclosure, the information acquired by the information acquisition unit 301 includes information from the construction drawings.

[0191] With this structure, the information acquisition unit 301, for example, Figure 10 As shown, information such as buried pipes UP, electrical wires EW, and slope shoulders TS, which are difficult to detect by external sensors such as camera device 40, can be obtained from the information in the construction drawings. As a result, the operator of the excavator 100 can set the restricted areas RA8, RA9, and RA10 based on buried pipes UP, electrical wires EW, and slope shoulders TS by giving instructions in natural language.

[0192] Furthermore, the excavator 100, as an embodiment of the working machinery involved in this disclosure, also includes a prompt word generation unit 304. This prompt word generation unit 304 enables the language model LM to parse the instructions acquired by the instruction acquisition unit 302 and the information languageized by the languageization unit 303. The prompt word generation unit 304 also functions as a translation unit, which translates the instructions acquired by the instruction acquisition unit 302 and the information languageized by the languageization unit 303 into a specific language corresponding to the language model LM and enables the language model LM to parse it.

[0193] This structure improves the accuracy of the language model LM's output. More specifically, when the specific language corresponding to the language model LM is English, the prompt word generation unit 304 translates instructions or information in languages ​​other than English into English and parses them for the language model LM. As a result, the accuracy of the language model LM's output of prompt words input from the prompt word generation unit 304 to the language model LM via the calling unit 305 is improved.

[0194] Furthermore, the excavator 100, which is an embodiment of the working machinery involved in this disclosure, also includes a display device 50A, which displays the restricted area RA set by the area setting unit 306.

[0195] With this structure, the operator of the excavator 100 can visually understand the restricted area RA set around the excavator 100 by checking the screen 41 of the display device 50A. Specifically, in Figure 7 In the example shown, restricted areas RA3 and RA5 are set to the right and rear of the excavator 100. At this time, as... Figure 11As shown, the display device 50A overlays virtual safety barriers SB3 and SB5, which serve as the boundaries of restricted areas RA3 and RA5, onto the bird's-eye view FV, right-side image RM, and rear-side image BM, etc., based on the image from the camera device 40. As a result, the operator of the excavator 100 can intuitively grasp the restricted areas RA3 and RA5 set on the right and rear of the excavator 100 based on the display on the screen 41 of the display device 50A.

[0196] Furthermore, the operation support system SYS of this embodiment includes an information acquisition unit 301, a speech recognition unit 303, an instruction acquisition unit 302, and a region setting unit 306. The information acquisition unit 301 acquires information related to the posture of the auxiliary device AT of the excavator 100 (which is a working machine) and the surrounding environment of the excavator 100. The speech recognition unit 303 translates the information acquired by the information acquisition unit 301 into natural language. The instruction acquisition unit 302 acquires instructions given in natural language by the operator of the excavator 100. The region setting unit 306 sets a restriction region RA that limits the intrusion or operating speed of the excavator 100. The region setting unit 306 sets the restriction region RA based on the result of parsing the instructions acquired by the instruction acquisition unit 302 and the speech recognition unit 303 using a language model LM, or based on the result of parsing the instructions acquired by the instruction acquisition unit 302 and the information acquired by the information acquisition unit 301 using a language model LM.

[0197] With this structure, the operation support system SYS according to this embodiment, similar to the excavator 100 of this embodiment, can set a restriction zone RA that limits the intrusion or movement speed of the excavator 100 based on the operator's natural language-based instructions. Therefore, compared to manually setting the restriction zone RA while the operator checks the environment around the excavator 100, the restriction zone RA can be set more easily. Thus, with the operation support system SYS according to this embodiment, various personnel, including inexperienced operators and foreign operators, can ensure safe operation.

[0198] The preferred embodiments of this disclosure have been described above. However, this disclosure is not limited to the embodiments described above. Various modifications and substitutions can be applied to the above embodiments without departing from the scope of this disclosure. Furthermore, the features described with reference to the above embodiments can be appropriately combined as long as they are not technically contradictory. For example, the above-described restricted area setting function can also be applied to other operating machinery besides excavators. Other operating machinery includes, for example, bulldozers, mobile cranes, etc.

[0199] This application claims priority based on Japanese Patent Application No. 2024-070004, filed on April 23, 2024, the entire contents of which are incorporated herein by reference.

[0200] Symbol Explanation 40 - Camera device (external sensor), 50A - Display device, 100 - Excavator (operating machinery), 301 - Information acquisition unit, 303 - Language processing unit, 302 - Instruction acquisition unit, 304 - Prompt generation unit (translation unit), 306 - Area setting unit, AT - Auxiliary device, CN - Cone (object), DT - Dump truck (object), LM - Language model, P - Person (object), RA - Restricted area, RA1-RA10 - Restricted area, RD - Road (object), S1 - Boom angle sensor (posture sensor), S2 - Stick angle sensor (posture sensor), S3 - Bucket angle sensor (posture sensor), S4 - Body tilt sensor (posture sensor), S6 - Sensor (position sensor), SYS - Operation support system.

Claims

1. A type of operating machinery, comprising: auxiliary equipment for operation; The information acquisition unit acquires information related to the posture of the auxiliary device and the surrounding environment; The languageization unit translates the information acquired by the information acquisition unit into natural language. The instruction acquisition unit acquires instructions given by the operator in natural language; and The zone setting unit defines restricted areas within the work site where intrusion or movement speed is limited. The region setting unit sets the restricted region based on the result obtained by parsing the instructions obtained by the instruction acquisition unit and the information translated by the languageization unit using the language model, or based on the result obtained by parsing the instructions obtained by the instruction acquisition unit using the language model and the information obtained by the information acquisition unit.

2. The operating machinery according to claim 1, wherein, The information acquisition unit acquires information related to the posture of the auxiliary device and the surrounding environment based on the outputs of the position sensor that acquires the position information of the working machine, the posture sensor that detects the posture of the auxiliary device, and the external sensor that detects objects around the working machine.

3. The operating machinery according to claim 2, wherein, The region setting unit sets the restricted region based on the result obtained by parsing the instructions acquired by the instruction acquisition unit and the information of objects detected by the external sensors using the language model.

4. The operating machinery according to claim 2, wherein, The region setting unit sets the restricted region based on the result obtained by parsing the instructions acquired by the instruction acquisition unit using the language model, the position information of the working machine acquired by the position sensor, and the posture of the auxiliary device acquired by the posture sensor.

5. The operating machinery according to claim 3, wherein, The region setting unit acquires information about multiple objects based on the detection results of the external sensors, and when the instruction acquisition unit acquires an instruction to set a restriction region for the multiple objects, it sets the restriction region based on the convex hull of the positions of the multiple objects.

6. The operating machinery according to claim 3, wherein, When the area setting unit detects two objects by the external sensor and the instruction acquisition unit obtains an instruction to set a restriction area based on the two objects, the area setting unit sets the restriction area on the side opposite to the working machine, using the straight line connecting the two objects as a reference.

7. The operating machinery according to claim 3, wherein, The area setting unit identifies a specific object based on the detection results of the external sensor, and when the instruction acquisition unit receives an instruction to set the restriction area for the specific object, it sets the restriction area in the area containing the specific object.

8. The operating machinery according to claim 3, wherein, The area setting unit obtains the position information of the movable object based on the detection results of the external sensor, and when the instruction acquisition unit obtains an instruction to set the restriction area for the movable object, it sets the restriction area based on the position information of the movable object.

9. The operating machinery according to claim 4, wherein, When the area setting unit obtains the position information of the working machine from the position sensor, the posture of the auxiliary device from the posture sensor, and the instruction acquisition unit obtains an instruction to set the restriction area based on the height of the working machine or the auxiliary device, the area setting unit sets the restriction area on the upper or lower side based on the height of the working machine or the auxiliary device.

10. The operating machinery according to claim 1, wherein, When the instruction acquisition unit receives an instruction to expand or shrink the set restricted area, the area setting unit expands or shrinks the restricted area according to the instruction.

11. The operating machinery according to claim 1, wherein, When the instruction acquisition unit receives an instruction to fix the set restricted area, the area setting unit fixes the restricted area.

12. The operating machinery according to claim 1, wherein, When the instruction acquisition unit receives an instruction to delete the set restricted area, the area setting unit deletes the restricted area.

13. The operating machinery according to claim 1, wherein, The information acquired by the information acquisition unit includes information from construction drawings.

14. The operating machinery according to claim 1, further comprising: The translation unit translates the instructions acquired by the instruction acquisition unit and the information processed by the languageization unit into a specific language corresponding to the language model so that it can be parsed by the language model.

15. The operating machinery according to claim 1, further comprising: A display device that displays the restricted area set by the area setting unit.

16. An operation support system comprising: The information acquisition unit acquires information related to the posture of the auxiliary devices of the working machine and the surrounding environment of the working machine; The languageization unit translates the information acquired by the information acquisition unit into natural language. The instruction acquisition unit acquires instructions given in natural language by the operator of the operating machinery; and The area setting unit sets a restricted area where the intrusion or operating speed of the operating machinery is limited. The region setting unit sets the restricted region based on the result obtained by parsing the instructions obtained by the instruction acquisition unit and the information translated by the languageization unit using the language model, or based on the result obtained by parsing the instructions obtained by the instruction acquisition unit using the language model and the information obtained by the information acquisition unit.

Images