Construction machine, operation support system, information processing device, and program

Abstract

The present invention provides a technology capable of improving the effectiveness of operation of a construction machine by an operator using a natural language instruction. A shovel (100) according to one embodiment of the present invention is provided with: an imaging device (40) for acquiring information indicating the environment around the shovel (100); a language conversion unit (303) for converting the information acquired by the imaging device (40) into a natural language; an instruction acquisition unit (301) that acquires an instruction using a natural language from an operator; and an operation control unit (306) that controls the operation of the shovel (100) on the basis of a result obtained by causing the language model LM1 to interpret the instruction acquired by the instruction acquisition unit (301) and the information language-converted by the language conversion unit (303).

Description

Technical Field

[0001] This application claims priority based on Japanese Patent Application No. 2023-217118 filed on December 22, 2023. The entire contents of the Japanese application are incorporated herein by reference.

[0002] The present invention relates to a construction machine or the like. Background Art

[0003] Conventionally, a technique has been known in which a construction machine is operated according to an instruction in natural language such as voice or text input (see Patent Document 1).

[0004] In Patent Document 1, it is possible to perform voice recognition on an instruction from an operator and cause the construction machine to perform an action corresponding to the content of the voice recognition.

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2000-056827

[0006] However, in Patent Document 1, it is only possible to give instructions for specific actions of a construction machine such as instructions for moving forward, backward, left, or right or instructions for excavation, and it is not possible to give instructions for actions that match the working environment, for example. Summary of the Invention

[0007] Therefore, in view of the above problems, an object of the present invention is to provide a technique capable of improving the effectiveness of operating a construction machine according to an instruction in natural language by an operator.

[0008] To achieve the above object, in one embodiment of the present invention,

[0009] There is provided a construction machine including:

[0010] An environment information acquisition unit that acquires information indicating the environment around the construction machine;

[0011] A verbalization unit that verbalizes the information acquired by the environment information acquisition unit in natural language;

[0012] An instruction acquisition unit that acquires an instruction in natural language from an operator; and

[0013] A control unit that controls the operation of the construction machine based on the result obtained by having a language model interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit.

[0014] Moreover, in another embodiment of the present invention,

[0015] There is provided an operation support system including:

[0016] An environment information acquisition unit that acquires information representing the environment around the construction machine;

[0017] A verbalization unit that verbalizes the information acquired by the environment information acquisition unit in natural language;

[0018] An instruction acquisition unit that acquires an instruction in natural language from an operator; and

[0019] A control unit that controls the operation of the construction machine based on the result obtained by having a language model interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit.

[0020] Moreover, in another embodiment of the present invention,

[0021] There is provided an information processing apparatus including:

[0022] A verbalization unit that verbalizes information representing the environment around the construction machine in natural language;

[0023] An instruction acquisition unit that acquires an instruction in natural language from an operator; and

[0024] A control unit that controls the operation of the construction machine based on the result obtained by having a language model interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit.

[0025] Moreover, in another embodiment of the present invention,

[0026] There is provided a program that causes an information processing apparatus to execute the following steps:

[0027] A verbalization step of verbalizing information representing the environment around the construction machine in natural language;

[0028] An instruction acquisition step of acquiring an instruction in natural language from an operator; and

[0029] A control step of controlling the operation of the construction machine based on the result obtained by having a language model interpret the instruction acquired in the instruction acquisition step and the information verbalized in the verbalization step.

[0030] Advantages of the Invention

[0031] According to the above-described embodiment, it is possible to improve the effectiveness of operating a construction machine according to an instruction in natural language from an operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a side view showing an example of an excavator.

[0033] Figure 2 It is a top view showing an example of an excavator.

[0034] Figure 3 This is a diagram showing an example of a structure related to the remote operation of an excavator.

[0035] Figure 4 This is a diagram showing an example of the hardware structure of an excavator.

[0036] Figure 5 This is a diagram showing an example of the hardware structure of a remote operation support device.

[0037] Figure 6 This is a functional block diagram showing the first example of a functional structure related to the operation support of an excavator.

[0038] Figure 7 This is a diagram showing an example of the surrounding environment of an excavator.

[0039] Figure 8 This is a diagram showing an example of an example given to a language model.

[0040] Figure 9 This is a diagram showing an example of a combination of prompts and outputs of a language model.

[0041] Figure 10 This is a flowchart roughly showing the first example of a process related to the operation support of an excavator.

[0042] Figure 11 This is a functional block diagram showing the second example of a functional structure related to the operation support of an excavator.

[0043] Figure 12 This is a flowchart roughly showing the second example of a process related to the operation support of an excavator.

[0044] Figure 13 This is a functional block diagram showing the third example of a functional structure related to the operation support of an excavator.

[0045] Figure 14 This is a flowchart roughly showing the third example of a process related to the operation support of an excavator.

[0046] Figure 15 This is a functional block diagram showing the fourth example of a functional structure related to the operation support of an excavator.

[0047] Figure 16 This is a flowchart roughly showing the fourth example of a process related to the operation support of an excavator.

[0048] Figure 17 This is a functional block diagram showing the fifth example of a functional structure related to the operation support of an excavator.

[0049] Figure 18It is a flowchart showing the fifth example of processes related to the operation support of an excavator.

[0050] In the figure: 1 - Lower traveling body, 2 - Swing mechanism, 3 - Upper swing body, 4 - Boom, 5 - Arm, 6 - Bucket, 10 - Cab, 11 - Engine, 14 - Main pump, 15 - Pilot pump, 17 - Control valve, 26 - Operating device, 30 - Controller, 31 - Hydraulic control valve, 40 - Imaging device, 50 - Output device, 52 - Input device, 60 - Communication device, 100 - Excavator, 200 - Remote operation support device, 207 - Input device, 208 - Display device, 301 - Instruction acquisition unit, 302 - Object detection unit, 303 - Verbalization unit, 304 - Prompt generation unit, 304A - Prompt generation unit, 304B - Prompt generation unit, 305 - Call unit, 306 - Motion control unit, 307 - Urgency determination unit, 307A - Instruction classifier, 308 - Language model selection unit, 309 - Notification unit, AT - Attachment device, HA - Hydraulic actuator, LM1 - Language model, LM2 - Language model, SYS - Remote operation support system. Detailed implementation manners

[0051] Hereinafter, the implementation manners will be described with reference to the drawings.

[0052] [Outline of the excavator]

[0053] Refer to Figures 1 - 3 The outline of the excavator 100 according to the present implementation manner will be described.

[0054] Figure 1 It is a side view showing an example of the excavator 100. Figure 2 It is a top view showing an example of the excavator 100. Figure 3 It is a diagram showing an example of the structure related to the remote operation of the excavator 100. Hereinafter, the direction in which the attachment device AT extends when looking down at the excavator 100 ( Figure 2 the upward direction) is defined as "front" to describe the direction in the excavator 100 or the direction observed from the excavator 100.

[0055] As Figure 1 、 Figure 2 shown, the excavator 100 includes a lower traveling body 1, an upper swing body 3, an attachment device AT including a boom 4, an arm 5, and a bucket 6, and a cab 10.

[0056] The lower traveling body 1 uses the crawlers 1C to make the excavator 100 travel. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The left crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the left crawler 1CL is hydraulically driven by a traveling hydraulic motor 1MR. Thus, the lower traveling body 1 can travel by itself.

[0057] The upper revolving body 3 is mounted on the lower traveling body 1 via a slewing mechanism 2 so as to be rotatable (freely rotatable). For example, the upper revolving body 3 rotates relative to the lower traveling body 1 by hydraulically driving the slewing mechanism 2 with a slewing hydraulic motor 2M.

[0058] The boom 4 is mounted at the front center of the upper revolving body 3 so as to be able to pitch about a rotation axis along the left - right direction. The arm 5 is mounted at the front end of the boom 4 so as to be able to rotate about a rotation axis along the left - right direction. The bucket 6 is mounted at the front end of the arm 5 so as to be able to rotate about a rotation axis along the left - right direction.

[0059] The bucket 6 is an example of an end - attachment device and is used, for example, for excavation work, slope work, or leveling work, etc.

[0060] The bucket 6 is mounted at the front end of the arm 5 so as to be replaceable appropriately according to the work content of the excavator 100. That is, a bucket of a different type from the bucket 6 can be mounted at the front end of the arm 5 instead of the bucket 6. For example, a relatively large large bucket, a slope bucket, a dredging bucket, etc. And, an end - attachment device of a type other than a bucket can also be mounted at the front end of the arm 5. For example, a mixer, a breaker, a crusher, etc. And, a preparatory attachment device such as a quick coupler, a tilt - rotator, etc. can also be provided between the arm 5 and the end - attachment device.

[0061] The boom 4, the arm 5, and the bucket 6 are respectively hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0062] The cab 10 is an operating room for an operator to board and operate the excavator 100. The cab 10 is mounted, for example, on the front left side of the upper revolving body 3.

[0063] The excavator 100 can be equipped with a communication device 60 and can communicate with a remote operation support device 200 via a prescribed communication line NW.

[0064] The communication line NW includes, for example, a local area network (LAN) at the construction site. And, the communication line NW can also include a wide area network (WAN). The wide area network includes, for example, a mobile communication network with a base station as a terminal, a satellite communication network using a communication satellite, the Internet, etc. And, the communication line NW can also include a short - range communication line based on a wireless communication standard such as WiFi or Bluetooth (registered trademark).

[0065] For example, the excavator 100 drives driven components such as the lower traveling body 1 (i.e., a pair of left and right crawlers 1CL and 1CR), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6 according to the operations of the operator riding in the cab 10.

[0066] Moreover, the excavator 100 may be configured to be remotely operable (long-distance operable) from the outside of the excavator 100 instead of being configured to be operable by the operator riding in the cab 10, or, in addition to being configured to be operable by the operator riding in the cab 10, may also be configured to be remotely operable from the outside of the excavator 100. In the case of remotely operating the excavator 100, the inside of the cab 10 may be in an unmanned state. And, in the case where the excavator 100 is dedicated to remote operation, the cab 10 may be omitted. Hereinafter, it is assumed that the operations of the operator include at least one of the operations of the operator in the cab 10 on the operation device 26 and the remote operation of the operator outside.

[0067] For example, as Figure 3 shown, the remote operation uses a remote operation support system SYS. In this remote operation support system SYS, the operation target excavator 100 and the remote operation support device 200 for the operator to operate the excavator 100 are connected via a communication circuit NW so as to be able to communicate. Specifically, the remote operation includes a method of operating the excavator 100 by operation input related to the actuator of the excavator 100 performed by the remote operation support device 200, and the remote operation support device 200 can communicate with the excavator 100 via the communication line NW.

[0068] The remote operation support device 200 is provided, for example, in a management center or the like that manages the work of the excavator 100 from the outside. And, the remote operation support device 200 may also be a portable operation terminal. In this case, the operator can remotely operate the excavator 100 while directly confirming the work status of the excavator 100 from the vicinity of the excavator 100.

[0069] The excavator 100 transmits, for example, an image (hereinafter referred to as "surrounding image") representing the situation of the surroundings including the front of the excavator 100 based on the captured image output by the on-board camera device 40 to the remote operation support device 200 via the communication device 60. Further, the excavator 100 may also transmit the captured image output by the camera device 40 to the remote operation support device 200 via the communication device 60, and the remote operation support device 200 processes the captured image received from the excavator 100 and generates a surrounding image. Then, the remote operation support device 200 displays the surrounding image representing the situation of the surroundings including the front of the excavator 100 on its own display device. Similarly, various information images (information screens) displayed on the output device 50 (display device) inside the cab 10 of the excavator 100 can also be displayed on the display device of the remote operation support device 200. Thus, an operator using the remote operation support device 200 can remotely operate the excavator 100 while confirming the display content such as the image or information screen representing the situation around the excavator 100 displayed on the display device. Then, the excavator 100 causes the actuator to operate according to the signal representing the content of the remote operation received from the remote operation support device 200 via the communication device 60, thereby driving the driven components such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6. Thus, the remote operation support system SYS can achieve remote operation of the excavator 100 using the remote operation support device 200.

[0070] Further, the remote operation may also include a method of operating the excavator 100 according to an external sound input or gesture input from a person (e.g., a worker) around the excavator 100. Specifically, the excavator 100 identifies the sound emitted by a surrounding worker or the gesture made by a worker, etc. through an on-board sound input device (e.g., a microphone) or gesture input device (e.g., a camera device) etc. Then, the excavator 100 can cause the actuator to operate according to the content of the identified sound or gesture, etc., thereby driving the driven components such as the lower traveling body 1 (left and right crawlers 1C), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6.

[0071] Further, the excavator 100 can also cause the actuator to automatically operate regardless of the operation content of the operator. Thus, the excavator 100 can achieve a function of automatically operating at least a part of the driven components such as the lower traveling body 1, the upper slewing body 3, and the attachment AT, that is, the so-called "automatic operation function" or "Machine Control (MC) function".

[0072] The automatic operation function includes, for example, a semi-automatic operation function (operation support type MC function). The semi-automatic operation function is a function that automatically operates a driven component (actuator) other than the driven component (actuator) of the operation target according to the operation of the operator. And, the automatic operation function may include a full-automatic operation function (full-automatic type MC function). The full-automatic operation function is a function that automatically operates at least a part of a plurality of driven components (actuators) without the operation of the operator. In the excavator 100, when the full-automatic operation function is effective, the inside of the cab 10 can be in a state without an operator. And, the semi-automatic operation function or the full-automatic operation function, etc. include, for example, a rule-based automatic operation function. The rule-based automatic operation function is an automatic operation function in the following manner: the operation content of the driven component (actuator) of the automatic operation target is automatically determined according to a preset rule. And, the semi-automatic operation function or the full-automatic operation function, etc. may also include an autonomous operation function. The autonomous operation function is an automatic operation function in the following manner: the excavator 100 makes various judgments autonomously and determines the operation content of the driven component (actuator) of the automatic operation target according to the judgment result.

[0073] And, the operation of the excavator 100 can also be remotely monitored. At this time, a remote monitoring support device having the same function as the remote operation support device 200 can be provided. The remote monitoring support device is, for example, the remote operation support device 200. Thus, a monitor who is a user of the remote monitoring support device can monitor the operation status of the excavator 100 while confirming the surrounding image displayed on the display device of the remote monitoring support device. And, for example, when it is judged to be necessary from the viewpoint of safety, the monitor can intervene in the operation or automatic operation of the excavator 100 by making a prescribed input using the input device of the remote monitoring support device and can make the excavator 100 stop urgently.

[0074] [Structure of Excavator]

[0075] Next, refer to Figure 4 to describe the structure of the excavator 100.

[0076] Figure 4 is a block diagram showing an example of the structure of the excavator 100.

[0077] In addition, in Figure 4 , the path for transmitting mechanical power is represented by a double line, the path through which the high-pressure working oil for driving the hydraulic actuator HA flows is represented by a solid line, the path for transmitting pilot pressure is represented by a dotted line, and the path for transmitting an electric signal is represented by a dashed line.

[0078] The excavator 100 includes various components such as a hydraulic drive system related to the hydraulic drive of the driven components, an operating system related to the operation of the driven components, a user interface system related to information exchange with the user, a communication system related to communication with the outside, and a control system related to various controls.

[0079] <Hydraulic Drive System>

[0080] As Figure 4 shown, as described above, the hydraulic drive system of the excavator 100 includes a hydraulic actuator HA that hydraulically drives the driven components such as the lower traveling body 1 (left and right crawlers 1C), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, respectively. And, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0081] The hydraulic actuator HA includes traveling hydraulic motors 1ML, 1MR, a slewing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, etc.

[0082] In addition, part or all of the hydraulic actuator HA of the excavator 100 can be replaced with an electric actuator. That is, the excavator 100 can be a hybrid excavator or an electric excavator.

[0083] The engine 11 is the prime mover of the excavator 100 and is the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine fueled by diesel. The engine 11 is, for example, mounted on the rear of the upper slewing body 3. The engine 11 rotates at a constant speed at a preset target speed under the direct or indirect control of the controller 30 described later to drive the main pump 14 and the pilot pump 15.

[0084] In addition, other prime movers (for example, an electric motor) etc. can be mounted on the excavator 100 to replace the engine 11, or, in addition to the engine 11, other prime movers etc. can be mounted on the excavator 100.

[0085] The regulator 13 controls (regulates) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate (hereinafter referred to as "deflection angle") of the main pump 14 according to the control command from the controller 30.

[0086] The main pump 14 supplies working oil to the control valve 17 through a high-pressure hydraulic pipeline. Similar to the engine 11, the main pump 14 is, for example, mounted on the rear of the upper slewing body 3. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement type hydraulic pump. As described above, under the control of the controller 30, the stroke length of the piston is adjusted by adjusting the deflection angle of the swash plate by the regulator 13, thereby controlling the discharge flow rate or discharge pressure.

[0087] The control valve 17 drives the hydraulic actuator HA according to the operation of the operation device 26 by the operator, the content of the remote operation, or the operation instruction corresponding to the automatic operation function. The control valve 17 is mounted, for example, at the center of the upper swing body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies the hydraulic oil supplied from the main pump 14 to each hydraulic actuator according to the operation of the operator or the operation instruction corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (directional control valves) that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0088] <Operating System>

[0089] As Figure 4 shown, the operating system of the excavator 100 includes a pilot pump 15, an operation device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0090] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. Similar to the engine 11, the pilot pump 15 is mounted, for example, at the rear of the upper swing body 3. The pilot pump 15 is, for example, a fixed-displacement hydraulic pump and is driven by the engine 11 as described above.

[0091] Alternatively, the pilot pump 15 may be omitted. In this case, the hydraulic oil with a relatively high pressure discharged from the main pump 14 can be supplied to various hydraulic devices as pilot pressure after being reduced in pressure by a specified pressure reducing valve.

[0092] The operation device 26 is provided near the operator's seat in the cab 10 and is used by the operator to operate various driven components. Specifically, the operation device 26 is used by the operator to operate the hydraulic actuator HA that drives each driven component, and as a result, the operator can operate the driven component that is the driving object of the hydraulic actuator HA. The operation device 26 includes a pedal device or a lever device for operating each driven component (hydraulic actuator HA).

[0093] For example, as Figure 4As shown, the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 uses the working oil supplied from the pilot pump 15 through the pilot pipeline 25 and the pilot pipeline 25A branched therefrom, and outputs a pilot pressure corresponding to the operation content to the secondary-side pilot pipeline 27A. The pilot pipeline 27A is connected to one introduction port of the reciprocating valve 32, and is connected to the control valve 17 via the pilot pipeline 27 connected to the discharge port of the reciprocating valve 32. Thus, a pilot pressure corresponding to the operation content related to various driven components (hydraulic actuators HA) in the operating device 26 can be input to the control valve 17 via the reciprocating valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operator or the like on the operating device 26.

[0094] Also, the operating device 26 can be an electric type. In this case, the pilot pipeline 27A, the reciprocating valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal corresponding to the operation content (hereinafter referred to as "operation signal"), and the operation signal is input to the controller 30. And the controller 30 outputs a control command corresponding to the content of the operation signal, that is, a control signal corresponding to the operation content for the operating device 26, to the hydraulic control valve 31. Thus, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0095] Also, the control valve (directional valve) for driving each hydraulic actuator HA built in the control valve 17 can be an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 can be directly input to the control valve 17 (that is, the electromagnetic solenoid type control valve).

[0096] Also, as described above, part or all of the hydraulic actuators HA can be replaced with electric actuators. In this case, the controller 30 can output a control command corresponding to the operation content of the operating device 26 or the content of the remote operation specified by the remote operation signal to the electric actuator or the driver for driving the electric actuator, etc. And in the case of remotely operating the excavator 100, or, as described later, in the case of operating the excavator 100 using natural language, the operating device 26 can be omitted.

[0097] The hydraulic control valve 31 is provided for each driven component (hydraulic actuator HA) of each operating object of the operating device 26, and is provided for each driving direction (for example, the raising direction and the lowering direction of the boom 4) of the driven component (hydraulic actuator HA). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 can be provided, for example, in the pilot pipe line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow path area (that is, the cross-sectional area through which the working oil can flow). Thus, the hydraulic control valve 31 can output a specified pilot pressure to the secondary-side pilot pipe line 27B by using the working oil of the pilot pump 15 supplied through the pilot pipe line 25B. Therefore, the hydraulic control valve 31 can indirectly actuate a specified pilot pressure corresponding to the control signal from the controller 30 on the control valve 17 through the check valve 32 between the pilot pipe line 27B and the pilot pipe line 27. Therefore, for example, the controller 30 can supply a pilot pressure corresponding to the operation instruction corresponding to the automatic operation function from the hydraulic control valve 31 to the control valve 17, so as to realize the operation of the excavator 100 based on the automatic operation function.

[0098] Moreover, the controller 30 can control the hydraulic control valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 200 to the hydraulic control valve 31 through the communication device 60. Thus, the controller 30 can supply a pilot pressure corresponding to the content of the remote operation from the hydraulic control valve 31 to the control valve 17, so as to realize the operation of the excavator 100 based on the remote operation of the operator.

[0099] Moreover, when the operating device 26 is electric, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the hydraulic control valve 31 to the control valve 17, so as to realize the operation of the excavator 100 based on the operation of the operator.

[0100] The reciprocating valve 32 has two inlet ports and one discharge port, and outputs the working oil with the higher pilot pressure among the pilot pressures input to the two inlet ports to the discharge port. Similarly to the hydraulic control valve 31, the reciprocating valve 32 is provided for each driven component (hydraulic actuator HA) of each operation target of the operation device 26, and is provided for each operation direction of the driven component (hydraulic actuator HA). For example, two reciprocating valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper revolving body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the reciprocating valve 32 is connected to the secondary pilot pipe 27A of the operation device 26 (specifically, the above-mentioned lever device or pedal device included in the operation device 26), and the other is connected to the secondary pilot pipe 27B of the hydraulic control valve 31. The discharge port of the reciprocating valve 32 is connected to the pilot port of the control valve corresponding to the control valve 17 through the pilot pipe 27. The corresponding control valve refers to the control valve for driving the hydraulic actuator HA which is the operation target of the above-mentioned lever device or pedal device connected to one inlet port of the reciprocating valve 32. Therefore, these reciprocating valves 32 can respectively apply the higher pilot pressure among the pilot pressure of the secondary pilot pipe 27A of the operation device 26 and the pilot pressure of the secondary pilot pipe 27B of the hydraulic control valve 31 to the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve regardless of the operation of the operator on the operation device 26 by outputting a pilot pressure higher than the secondary pilot pressure of the operation device 26 from the hydraulic control valve 31. Thus, the controller 30 can control the actions of the driven components (the lower traveling body 1, the upper revolving body 3, the boom 4, the arm 5, and the bucket 6) regardless of the operation state of the operator on the operation device 26, thereby realizing the automatic operation function or the remote operation function.

[0101] The hydraulic control valve 33 is provided in the pilot pipe 27A connecting the operating device 26 and the reciprocating valve 32. The hydraulic control valve 33 is configured to be able to change its flow path area, for example. The hydraulic control valve 33 operates according to the control signal input from the controller 30. Thus, when the operating device 26 is being operated by an operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. And, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 so that it is lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valve 31 and the hydraulic control valve 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17 regardless of the operation content of the operating device 26, for example. Thus, the controller 30 can more appropriately implement the automatic operation function or the remote operation function of the excavator 100 by controlling the hydraulic control valve 33 in addition to controlling the hydraulic control valve 31, for example.

[0102] <User Interface System>

[0103] As Figure 4 shown, the user interface system of the excavator 100 includes an operating device 26, an output device 50, and an input device 52.

[0104] The output device 50 outputs various information to the user of the excavator 100 (for example, the operator in the cab 10 or the operator of remote operation outside), or the people around the excavator 100 (for example, workers or drivers of construction vehicles).

[0105] For example, the output device 50 includes lighting equipment or display devices that output various information visually. The lighting equipment is, for example, a warning light (indicator light), etc. The display device is, for example, a liquid crystal display or an organic EL (Electroluminescence) display, etc. For example, as Figure 2 shown, the lighting equipment or the display device can be provided inside the cab 10 and output various information visually to the operator inside the cab 10, etc. And the lighting equipment or the display device can also be provided on the side of the upper swing body 3, etc., and output various information visually to the workers around the excavator 100, etc., for example.

[0106] Further, the output device 50 may also include a sound output device that outputs various information in an audible manner. The sound output device includes, for example, a buzzer or a speaker. The sound output device may be provided, for example, on at least one of the inside and outside of the cab 10, and outputs various information to the operator inside the cab 10 or people (such as workers) around the excavator 100 in an audible manner.

[0107] Further, the output device 50 may also include a device that outputs various information by tactile means such as vibration of the operator's seat.

[0108] The input device 52 receives various inputs from the user of the excavator 100, and a signal corresponding to the received input is input to the controller 30. For example, as Figure 2 shown, the input device 52 is provided inside the cab 10 and receives inputs from the operator inside the cab 10 and the like. Further, the input device 52 may be provided, for example, on the side surface of the upper swing body 3 or the like, and receives inputs from workers around the excavator 100 and the like.

[0109] For example, the input device 52 includes a mechanical input device that receives inputs from the user based on mechanical operations. The mechanical input device may include a touch panel mounted on the display device, a touchpad provided around the display device, a button switch, a lever, a toggle switch, a rotary switch provided on the operating device 26 (lever device), and the like.

[0110] Further, the input device 52 may also include a sound input device that receives the user's voice input. The sound input device includes, for example, a microphone.

[0111] Further, the input device 52 may also include a gesture input device that receives the user's gesture input. The gesture input device includes, for example, a camera device that captures the state of the gesture made by the user.

[0112] Further, the input device 52 may also include a biological input device that receives the user's biological input. The biological input includes, for example, the input of biological information such as the user's fingerprint and iris.

[0113] <Communication System>

[0114] As Figure 4 shown, the communication system of the excavator 100 according to the present embodiment includes a communication device 60.

[0115] The communication device 60 is connected to the external communication line NW and communicates with a device that is separately installed from the shovel 100. The device that is separately installed from the shovel 100 may include a device located outside the shovel 100, and may also include a portable terminal device (mobile terminal) that the user of the shovel 100 brings into the cab 10. The communication device 60 may include, for example, a device that complies with 4G (4 th Generation (fourth generation)), 5G (5 th The communication device 60 may include a mobile communication module of a specification such as the fifth generation. Furthermore, the communication device 60 may include, for example, a satellite communication module. Furthermore, the communication device 60 may include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. Furthermore, in the case where there are multiple connectable communication lines NW, the communication device 60 may include multiple communication devices according to the type of the communication line NW.

[0116] For example, the communication device 60 communicates with an external device such as the remote operation support device 200 in the construction site through a local communication line constructed at the construction site. The local communication line is, for example, a mobile communication line based on a local 5G (so-called local 5G) constructed at the construction site or a local area network based on WiFi6.

[0117] Furthermore, the communication device 60 may communicate with the remote operation support device 200 or the like located outside the construction site via a communication line in a wide area including the construction site (ie, a wide area network).

[0118] <Control system>

[0119] like Figure 4 As shown, the control system of the shovel 100 includes a controller 30. Furthermore, the control system of the shovel 100 according to the present embodiment includes an operation pressure sensor 29, an imaging device 40, and sensors S1 to S9.

[0120] The controller 30 performs various controls related to the shovel 100 .

[0121] The functions of the controller 30 may be implemented by any hardware or any combination of hardware and software. Figure 3 As shown, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D connected via a bus BS1.

[0122] The auxiliary storage device 30A is a non-volatile storage component that stores the programs to be installed and also stores required files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory, etc.

[0123] For example, in the presence of a program start instruction, the memory device 30B loads the program of the auxiliary storage device 30A so that the CPU 30C can read the program. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0124] The CPU 30C, for example, executes the program loaded into the memory device 30B and realizes various functions of the controller 30 according to the commands of the program.

[0125] The interface device 30D, for example, functions as a communication interface for connecting to the communication line inside the excavator 100. The interface device 30D may also include a plurality of different types of communication interfaces according to the type of the communication line to be connected.

[0126] Also, the interface device 30D functions as an external interface for reading data from a recording medium or writing data to a recording medium. The recording medium is, for example, a dedicated tool connected to a connector provided inside the cab 10 through a detachable cable. Also, the recording medium can, for example, also be a general recording medium such as an SD memory card or a USB (Universal Serial Bus) memory, etc. Thus, the programs for realizing various functions of the controller 30 can, for example, be provided by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Also, the programs can be downloaded from other computers outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.

[0127] In addition, a part of the functions of the controller 30 can also be realized by other controllers (control devices). That is, the functions of the controller 30 can be in a manner of being dispersedly realized by a plurality of controllers mounted on the excavator 100.

[0128] The operation pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot-operated device 26, that is, the pilot pressure corresponding to the operation state of each driven component (hydraulic actuator) in the operation device 26. The detection signal of the pilot pressure corresponding to the operation state related to each driven component (hydraulic actuator HA) in the operation device 26 detected by the operation pressure sensor 29 is input to the controller 30.

[0129] Further, when the operating device 26 is electric, or when the operating device 26 is omitted, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven component operated by the operating device 26 based on the operation signal input from the operating device 26.

[0130] The imaging device 40 captures the situation around the excavator 100.

[0131] The imaging device 40 is, for example, a monocular camera. Also, the imaging device 40 can be, for example, a three-dimensional camera (3D camera) such as a stereo camera, a TOF (Time of Flight) camera, or a depth camera that can not only acquire two-dimensional image information but also acquire three-dimensional information including information related to the distance from the object appearing in the image or the depth of the image.

[0132] For example, as Figure 2 shown, the imaging device 40 includes cameras 40F, 40B, 40L, and 40R. The camera 40F captures the front of the upper swing body 3. The camera 40B captures the rear of the upper swing body 3. The camera 40L captures the left side of the upper swing body 3. The camera 40R captures the right side of the upper swing body 3. Thus, the imaging device 40 can capture the entire circumference centered on the excavator 100 when looking down at the excavator 100, that is, the situation in the angular direction range spanning 360 degrees. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X".

[0133] The output data of the imaging device 40 (camera 40X) is input to the controller 30 via a one-to-one communication line or a vehicle network. Thus, for example, the controller 30 can grasp the situation around the excavator 100 based on the output data of the camera 40X.

[0134] In addition, a part or all of the cameras 40B, 40L, and 40R may be omitted. Also, a distance measuring sensor (also referred to as a "distance sensor") capable of acquiring information indicating the distance between the excavator 100 and surrounding objects may be provided on the excavator 100 instead of the imaging device 40, or a distance measuring sensor capable of acquiring information indicating the distance between the excavator 100 and surrounding objects may be provided on the excavator 100 in addition to the imaging device 40. The distance measuring sensor is, for example, LIDAR (Light Detecting and Ranging), a millimeter-wave radar, or an ultrasonic sensor.

[0135] The sensor S1 is installed on the boom 4 to measure the posture state of the boom 4. The sensor S1 outputs measurement data indicating the posture state of the boom 4. The posture state of the boom 4 is, for example, the posture angle (hereinafter referred to as "boom angle") of the base end of the boom 4 corresponding to the connection portion with the upper swing body 3 around the rotation axis. The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. The same applies to the sensors S2 to S4 hereinafter. Further, the sensor S1 may include a cylinder sensor for detecting the telescopic position of the boom cylinder 7. The same applies to the sensors S2 and S3 hereinafter. The output of the sensor S1 (measurement data indicating the posture state of the boom 4) is input to the controller 30. Thus, the controller 30 can grasp the posture state of the boom 4.

[0136] The sensor S2 is installed on the arm 5 to measure the posture state of the arm 5. The sensor S2 outputs measurement data indicating the posture state of the arm 5. The posture state of the arm 5 is, for example, the posture angle (hereinafter referred to as "arm angle") of the base end of the arm 5 corresponding to the connection portion with the boom 4 around the rotation axis. The output of the sensor S2 (measurement data indicating the posture state of the arm 5) is input to the controller 30. Thus, the controller 30 can grasp the posture state of the arm 5.

[0137] The sensor S3 is installed on the bucket 6 to measure the posture state of the bucket 6. The sensor S3 outputs measurement data indicating the posture state of the bucket 6. The posture state of the bucket 6 is, for example, the posture angle (hereinafter referred to as "bucket angle") of the base end of the bucket 6 corresponding to the connection portion with the arm 5 around the rotation axis. The output of the sensor S3 (measurement data indicating the posture state of the bucket 6) is input to the controller 30. Thus, the controller 30 can grasp the posture state of the bucket 6.

[0138] The sensor S4 measures the posture state of the body of the excavator 100 (for example, the upper swing body 3). The sensor S4 outputs measurement data indicating the posture state of the body of the excavator 100. The posture state of the body of the excavator 100 is, for example, the inclination state of the body with respect to a specified reference plane (for example, the horizontal plane). For example, the sensor S4 is installed on the upper swing body 3 to measure the inclination angles (hereinafter referred to as "front-back inclination angle" and "left-right inclination angle") of the excavator 100 around two axes in the front-back direction and the left-right direction. The output of the sensor S4 (measurement data indicating the posture state of the body of the excavator 100) is input to the controller 30. Thus, the controller 30 can grasp the posture state (inclination state) of the body (upper swing body 3).

[0139] The sensor S5 is installed on the upper swing body 3 to measure the swing state of the upper swing body 3. The sensor S5 outputs measurement data indicating the swing state of the upper swing body 3. The sensor S5 measures, for example, the swing angular velocity or swing angle of the upper swing body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the sensor S5 (measurement data indicating the swing state of the upper swing body 3) is input to the controller 30. Thus, the controller 30 can grasp the swing state such as the swing angle of the upper swing body 3.

[0140] The controller 30 can grasp (infer) the position of the front end (bucket 6) of the attachment device AT based on the outputs of the sensors S1 to S5.

[0141] In addition, when the sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the swing state (e.g., swing angular velocity) of the upper swing body 3 can also be detected based on the detection signal of the sensor S4. In this case, the sensor S5 can be omitted.

[0142] The sensor S6 measures the position of the excavator 100. The sensor S6 can measure the position in the world (global) coordinates or in the local coordinates of the construction site. In the former case, the sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, the sensor S6 is a transceiver that can communicate with a device serving as a reference for the position of the construction site and outputs a signal corresponding to the position relative to the reference. The output of the sensor S6 is input to the controller 30.

[0143] The sensor S7 measures the pressure (cylinder pressure) of the oil chamber of the boom cylinder 7. The sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the boom cylinder 7 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber. The output of the sensor S7 (i.e., measurement data of the cylinder pressure of the boom cylinder 7) is input to the controller 30.

[0144] The sensor S8 measures the pressure (cylinder pressure) of the oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the arm cylinder 8 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of the arm cylinder 8. The output of the sensor S8 (i.e., measurement data of the cylinder pressure of the arm cylinder 8) is input to the controller 30.

[0145] The sensor S9 measures the pressure (cylinder pressure) of the oil chamber of the bucket cylinder 9. The sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber on the rod side of the bucket cylinder 9 and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber on the bottom side of the bucket cylinder 9. The output of the sensor S9 (i.e., the measurement data of the cylinder pressure of the bucket cylinder 9) is input to the controller 30.

[0146] The controller 30 can grasp the load state acting on the attachment device AT based on the outputs of the sensors S7 to S9. The load acting on the attachment device AT includes, for example, the reaction force acting on the bucket 6 from the sand on the ground of the work object or the weight of the sand contained in the bucket 6.

[0147] In addition, some or all of the sensors S1 to S9 may be omitted as needed. Other sensors capable of grasping the state of the excavator 100 may also be mounted on the excavator 100. For example, the excavator 100 may be equipped with an orientation sensor capable of detecting its own orientation. The orientation sensor is, for example, an electronic compass including a geomagnetic sensor.

[0148] [Structure of the remote operation support device]

[0149] Next, refer to Figure 5 to describe the structure of the remote operation support device 200.

[0150] Figure 5 is a block diagram showing an example of the structure of the remote operation support device 200.

[0151] The functions of the remote operation support device 200 are implemented by any hardware or any combination of hardware and software, etc. For example, as Figure 5 shown, the remote operation support device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. They are connected by a bus BS2.

[0152] The external interface 201 functions as an interface for reading data from or writing data to the recording medium 201A. The recording medium 201A includes, for example, a floppy disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. Thus, the remote operation support device 200 can read various data used in processing through the recording medium 201A and store them in the auxiliary storage device 202 or install programs for implementing various functions.

[0153] In addition, the remote operation support device 200 can also obtain various data or programs used in the process from an external device through the communication interface 206.

[0154] The auxiliary storage device 202 stores various installed programs and stores files, data, etc. required for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, etc.

[0155] In the case of a program start instruction, the memory device 203 reads the program from the auxiliary storage device 202 and stores it. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) or an SRAM.

[0156] The CPU 204 executes various programs loaded from the auxiliary storage device 202 into the memory device 203 and implements various functions related to the remote operation support device 200 according to the programs.

[0157] The high-speed computing device 205 cooperates with the CPU 204 to perform arithmetic processing at a relatively high speed. The high-speed computing device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

[0158] In addition, according to the speed of the required arithmetic processing, the high-speed computing device 205 can also be omitted.

[0159] The communication interface 206 serves as an interface for connecting to an external device to enable communication. Thus, the remote operation support device 200 can communicate with an external device such as the excavator 100 through the communication interface 206. Also, the communication interface 206 can have multiple communication interfaces according to the communication method with the connected device, etc.

[0160] The input device 207 receives various inputs from the user. The input device 207 includes a remote operation device for remotely operating the excavator 100.

[0161] The input device 207 includes, for example, an input device (mechanical input device) that accepts mechanical operation inputs from the user. The operation device for remote operation can be a mechanical input device. The mechanical input device includes, for example, buttons, toggle keys, levers, keyboards, mice, touch panels installed on the display device 208, touchpads separately provided from the display device 208, and the like.

[0162] Moreover, the input device 207 may also include a voice input device that can accept voice inputs from the user. The voice input device includes, for example, a microphone that can collect the user's voice.

[0163] Moreover, the input device 207 may also include a gesture input device that can accept gesture inputs from the user. The gesture input device includes, for example, a camera that can capture the state of the user's gestures.

[0164] Moreover, the input device 207 may also include a biological input device that can accept biological inputs from the user. The biological input device includes, for example, a camera that can acquire image data containing information related to the user's fingerprint or iris.

[0165] The display device 208 displays an information screen or an operation screen for the user of the remote operation support device 200. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0166] The voice output device 209 notifies the user of the remote operation support device 200 of various information using voice. The voice output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0167] [First Example of Functional Structure Related to Excavator Operation Support]

[0168] Next, in addition to referring to Figures 1 - 5 in addition to, Figures 6 - 9 a first example of the functional structure related to the operation support of the excavator 100 will be described.

[0169] Figure 6 is a functional block diagram showing a first example of the functional structure related to the operation support of the excavator 100.

[0170] Figure 7 is a diagram showing an example of the surrounding environment of the excavator 100. Figure 8 is a diagram showing an example of the problem given to the language model LM 1. Figure 9 is a diagram showing an example of the combination of the prompts and outputs of the language model LM1.

[0171] As Figure 6As shown, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, and an action control unit 306 as functional units. These functions are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it by the CPU 30C. Also, a language model LM1 is provided outside the excavator 100.

[0172] The language model LM1 is a large language model (LLM: Large Language Models). The language model LM1 is installed in an external device (e.g., a server device) that is connected to the excavator 100 via the communication device 60 so as to be able to communicate with each other. The language model LM1 is, for example, GPT-4.

[0173] The instruction acquisition unit 301 acquires an instruction related to the operation of the excavator 100 (hereinafter, simply referred to as "instruction") input by the operator in natural language.

[0174] In the case of the operator in the cab 10, the input of the instruction from the operator is received by the input device 52 provided in the cab 10, and in the case of the remotely operating operator, it is received by the input device 207 of the remote operation support device 200.

[0175] The instruction using natural language is, for example, an instruction input in natural language by the operator's voice. At this time, the instruction acquisition unit 301 can acquire data of text (article) corresponding to the instruction using natural language by applying known speech recognition technology to the data of the voice input received by the input device 52 or the input device 207.

[0176] Also, the instruction using natural language may be an instruction text-input by the operator using an input device 52 or an input device 207 capable of character input such as a keyboard or a touch panel. At this time, the instruction acquisition unit 301 can acquire data of the text (article) received by the input device 52 or the input device 207 as an instruction in natural language.

[0177] The object detection unit 302 detects a monitoring target object around the excavator 100 based on the output of the imaging device 40 or the distance measuring sensor. For example, the object detection unit 302 detects the monitoring target object from the captured image of the imaging device 40 by arbitrarily applying known image processing technology such as semantic segmentation or machine learning.

[0178] The monitored object, for example, includes a worker or the like. Also, the monitored object may include other obstacles around the excavator 100. Other obstacles include, for example, specific moving objects at the construction site of the excavator 100 such as other construction machinery or construction vehicles. Also, other obstacles may include specific fixed objects at the construction site of the excavator 100 such as utility poles, fences, and traffic cones (also known as color cones (registered trademark)). Also, other obstacles may include specific terrain shapes at the construction site of the excavator 100 such as trenches, pits, and sand piles.

[0179] The verbalization unit 303 verbalizes the environment around the excavator 100 using natural language. Also, the verbalization unit 303 may verbalize the construction drawings of the work object of the excavator 100 using natural language.

[0180] For example, the verbalization unit 303 verbalizes the configuration of the monitored objects detected by the object detection unit 302 around the excavator 100 using natural language. Specifically, the verbalization unit 303 can verbalize in the form of applying the types and positions of the monitored objects detected by the object detection unit 302 to a template of a text that represents the configuration of the monitored objects around the excavator 100. The same applies to the verbalization of the configuration of the objects in the construction drawings described later.

[0181] In the template of the text representing the configuration of the monitored objects, "xxx" is the position information of the monitored object, and "yyy" is the type or name of the monitored object, and is specified, for example, in the form of "There is yyy on xxx." or "There exists yyy on xxx."

[0182] For example, as Figure 7 shown, the object detection unit 302 has detected a person P, traffic cones CN1, CN2, and a sand pile PL in front of the excavator 100.

[0183] In this example, the area around the excavator 100 is divided into a front area RF, a rear area RB, a left area RL, and a right area RR based on the excavator 100. Also, the front area RF is divided into an adjacent area RF1 on the left front, an adjacent area RF2 on the right front, an area RF3 on the left front and farther away than the area RF1, and an area RF4 on the right front and farther away than the area RF2. Hereinafter, the areas RF1, RF2, RF3, and RF4 are verbalized as "left front adjacent area", "right front adjacent area", "left front distant area", and "right front distant area".

[0184] At this time, since person P is included in region RF1, the verbalization unit 303 verbalizes the existence of person P as "There is a person in the left front adjacent region". Also, since the two triangular pyramids CN1 and CN2 are included in region RF2, the verbalization unit 303 verbalizes the existence of the triangular pyramids CN1 and CN2 as "There are two color cones in the right front adjacent region". Also, since the sand pile PL is included in region RF3, the verbalization unit 303 verbalizes the existence of the sand pile PL as "There is a sand pile in the left front distant region".

[0185] Also, in the same way as the case of the objects described in the construction drawings to be described later, the verbalization unit 303 can also verbalize the existence of person P, the triangular pyramids CN1 and CN2, and the sand pile PL using the distance or coordinates based on the excavator 100.

[0186] Also, for example, the verbalization unit 303 verbalizes the arrangement of the objects described in the construction drawings based on the position of the excavator 100 using natural language.

[0187] For example, regarding the buried piping described in the construction drawings, the verbalization unit 303 verbalizes it as "There is piping 1 m underground at a position 5 m in front". Also, regarding the utility pole described in the construction drawings, the verbalization unit 303 verbalizes it as "There is a utility pole at a position 5 m in front and 2 m to the right". Also, the verbalization unit 303 can also verbalize the position of the objects described in the construction drawings using the coordinates based on the excavator 100 or fixed to the construction site.

[0188] The prompt generation unit 304 generates a prompt to be input to the language model LM1 based on the natural language-based instruction obtained by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303. Specifically, the prompt generation unit 304 generates a prompt for outputting control information of the excavator 100 corresponding to the instruction obtained by the instruction acquisition unit 301 on the premise of the information on the surrounding environment of the excavator 100 or the construction drawings verbalized by the verbalization unit 303.

[0189] For example, the prompt generation unit 304 generates a plurality of examples and pre-gives them to the language model LM1 through the call unit 305. The examples are defined by a combination of the information on the surrounding environment of the excavator 100 or the construction drawings as the premise conditions (i.e., constraint conditions), the instruction of the example, and the correct answer to be output. Thereby, the language model LM1 can understand (learn) the output form of the prompt with respect to the instruction.

[0190] For example, as Figure 8 shown, a plurality of examples represented by a combination of the surrounding environment of the excavator 100, the instruction of the user (i.e., the operator), and the control information (control command) of the excavator 100 are given. Thereby, as Figure 9As shown, a control command corresponding to the operation of the excavator 100 can be output from the language model LM1 for the prompt generated by the prompt generation unit 304 (i.e., an instruction from the operator premised on the surrounding environment of the excavator 100). Also, in the case where the instruction cannot be understood, by asking a counter-question for the instruction, it is possible to suppress the situation where the excavator 100 performs an inappropriate operation. The number of given example problems is arbitrary, but it is preferably more than the number of types of control commands corresponding to the operations that the excavator 100 should perform. This is because the language model LM1 can understand all control commands.

[0191] The calling unit 305 calls the language model LM1 through, for example, a prescribed API (Application Programming Interface), inputs the prompt generated by the prompt generation unit 304 into the language model LM1, and obtains its output (reply).

[0192] The motion control unit 306 controls the operation of the excavator 100 according to the output of the language model LM1 obtained by the calling unit 305. Specifically, the motion control unit 306 drives the hydraulic actuator HA of the excavator 100 according to the control command output from the language model LM1, and outputs a control instruction for controlling the operation of the excavator 100 to the hydraulic control valve 31.

[0193] In this way, in this example, the controller 30 controls the operation of the excavator 100 based on the result obtained by having the language model LM1 interpret the instruction of the operator based on natural language and the surrounding environment of the excavator 100 verbalized in natural language. Thereby, the controller 30 can control the operation of the excavator 100 according to the instruction of the operator based on natural language that conforms to the surrounding environment of the excavator 100.

[0194] In addition, part or all of the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, and the motion control unit 306 may be transferred to an information processing device outside the excavator 100. For example, in the case of remotely operating the excavator 100, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, and the motion control unit 306 may be transferred to the remote operation support device 200.

[0195] [First Example of Processing Related to Operation Support of Excavator]

[0196] Next, refer to Figure 10 An explanation will be given of the first example of the processing related to the operation support of the excavator 100.

[0197] Figure 10It is a flowchart schematically showing a first example of processing related to the operation support of the excavator 100. In this example, it is premised on the functional structure related to the operation support of the excavator 100 Figure 6 as follows.

[0198] For example, it is executed when an operator's instruction based on natural language is received through the input device 52 or the input device 207 Figure 10 as shown in the flowchart.

[0199] As Figure 10 shown, in step S102, the instruction acquisition unit 301 acquires the text of the instruction.

[0200] If the processing of step S102 is completed, the controller 30 proceeds to step S104.

[0201] In step S104, the object detection unit 302 detects the monitoring target object around the excavator 100 based on the output of the imaging device 40 or the distance measuring sensor.

[0202] If the processing of step S104 is completed, the controller 30 proceeds to step S106.

[0203] In step S106, the verbalization unit 303 verbalizes the environment and construction drawings around the excavator 100 in natural language.

[0204] If the processing of step S106 is completed, the controller 30 proceeds to step S108.

[0205] In step S108, the prompt generation unit 304 generates a prompt to be input to the language model LM1 based on the results of the processing in steps S104 and S106.

[0206] If the processing of step S108 is completed, the controller 30 proceeds to step S110.

[0207] In step S110, the call unit 305 calls the language model LM1 and inputs the prompt generated in step S108.

[0208] If step S110 is completed and an output is obtained from the language model LM1, the controller 30 proceeds to step S112.

[0209] In step S112, the motion control unit 306 controls the motion of the excavator 100 based on the output of the language model LM1 (i.e., the control command).

[0210] If the processing of step S112 is completed, the controller 30 ends the processing of this flowchart.

[0211] [Second Example of Functional Structure Related to Operation Support of Excavator]

[0212] Next, in addition to reference Figures 1 - 5 in addition, reference is also made to Figure 11 to describe a second example of the functional structure related to the operation support of the excavator 100.

[0213] Hereinafter, the same or corresponding structures as those in the first example of the above functional structure are denoted by the same reference numerals, and the description will be centered on the parts different from those in the first example of the above functional structure.

[0214] Figure 11 is a functional block diagram showing a second example of the functional structure related to the operation support of the excavator 100.

[0215] As Figure 11 shown, similar to the first example above, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, and an action control unit 306 as functional units. And, different from the first example above, the controller 30 includes an urgency determination unit 307. These functions are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it by the CPU 30C.

[0216] The urgency determination unit 307 determines the urgency of the instruction acquired by the instruction acquisition unit 301. The urgency of the instruction indicates the degree to which the action of the excavator 100 corresponding to the instruction should be urgently implemented.

[0217] For example, the urgency determination unit 307 determines the high or low urgency according to the presence or absence of a specified character string (hereinafter, for convenience, referred to as "urgency word") included in the text acquired by the instruction acquisition unit 301. The urgency words are, for example, "dangerous", "precarious", "stop", etc. At this time, the urgency determination unit 307 determines that the urgency is high when the text acquired by the instruction acquisition unit 301 includes an urgency word, and determines that the urgency is low when it does not include it.

[0218] If it is determined that the urgency is high, the urgency determination unit 307 outputs a notice of high urgency together with the text of the instruction acquired by the instruction acquisition unit 301 to the action control unit 306.

[0219] If a notification with a high urgency level is input from the urgency determination unit 307, the operation control unit 306 causes the excavator 100 to perform a prescribed operation (hereinafter, for convenience, referred to as "emergency operation") corresponding to the instruction determined to have a high urgency level. The emergency operation is, for example, an operation to suddenly stop the excavator 100 (hereinafter, referred to as "sudden stop operation"). In addition, the emergency operation may include, in addition to the sudden stop operation of the excavator 100, an operation to avoid danger by deliberately causing the excavator 100 to continue operating (hereinafter, referred to as "danger avoidance operation"). At this time, the operation control unit 306 determines whether to perform a sudden stop operation or a danger avoidance operation based on the words included in the text of the instruction.

[0220] On the other hand, if it is determined that the urgency level is low, the urgency determination unit 307 outputs the notification with a low urgency level and the text of the instruction acquired by the instruction acquisition unit 301 to the prompt generation unit 304.

[0221] The prompt generation unit 304 generates a prompt to be input to the language model LM1 based on the natural language instruction acquired by the instruction acquisition unit 301 and input through the urgency determination unit 307 and the information verbalized by the verbalization unit 303.

[0222] Similar to the first example described above, the invocation unit 305 invokes the language model LM1, inputs the prompt generated by the prompt generation unit 304 into the language model LM1, and obtains its output (reply).

[0223] In the case where it is determined by the urgency determination unit 307 that the urgency level is low, similar to the first example described above, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1 acquired by the invocation unit 305.

[0224] In this way, in this example, when the urgency level of the operator's instruction is relatively high, the controller 30 does not apply the language model LM1, but directly causes the excavator 100 to perform an emergency operation corresponding to the instruction. As a result, when the urgency level of the operator's instruction is high, the controller 30 can avoid the delay caused by using the language model LM1, and thus cause the excavator 100 to perform an emergency operation quickly.

[0225] In addition, a part or all of the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the invocation unit 305, the operation control unit 306, and the urgency determination unit 307 may also be transferred to an information processing device outside the excavator 100. For example, in the case of remotely operating the excavator 100, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the invocation unit 305, the operation control unit 306, and the urgency determination unit 307 may be transferred to the remote operation support device 200.

[0226] [Second Example of Processing Related to Operation Support of Excavator]

[0227] Next, referring to Figure 12 a second example of the processing related to the operation support of the excavator 100 will be described.

[0228] Figure 12 is a flowchart schematically showing a second example of the processing related to the operation support of the excavator 100. In this example, it is premised on the functional structure related to the operation support of the Figure 11 excavator 100.

[0229] Figure 12 The flowchart of

[0230] such as Figure 12 shown is executed when an instruction from an operator based on natural language is received through the input device 52 or the input device 207. Figure 10 As

[0231] shown, the processing of step S202 is the same as the processing of step S102 of

[0232] so the description thereof is omitted.

[0233] If the processing of step S202 is completed, the controller 30 proceeds to step S204.

[0234] In step S204, the urgency determination unit 307 determines whether the urgency of the instruction obtained in step S202 is relatively high or relatively low based on the text of the instruction.

[0235] If the processing of step S204 is completed, the controller 30 proceeds to step S206. Figure 10

[0236] In step S206, the controller 30 determines whether the urgency of the instruction obtained in step S202 is relatively high based on the determination result of step S204. If the urgency of the instruction is relatively low, the controller 30 proceeds to step S208, and if the urgency of the instruction is relatively high, the controller 30 proceeds to step S218.

[0237] The processing of steps S208, S210, S212, S214, and S216 is the same as the processing of steps S104, S106, S108, S110, and S112 of the above

[0238] [Third Example of Functional Structure Related to Operation Support of Excavator]

[0239] Next, in addition to referring to Figures 1 - 5 in addition, also refer to Figure 13 A third example of the functional structure related to the operation support of the excavator 100 will be described.

[0240] Hereinafter, the same symbols will be assigned to the structures that are the same as or corresponding to the first or second example of the above functional structure, and the description will be centered on the parts that are different from the first or second example of the above functional structure.

[0241] Figure 13 It is a functional block diagram showing a third example of the functional structure related to the operation support of the excavator 100.

[0242] As Figure 13 shown, similar to the second example above, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, an action control unit 306, and an urgency determination unit 307 as functional units. And, different from the first and second examples above, the controller 30 includes a language model LM2 and a language model selection unit 308. These functions are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it by the CPU 30C.

[0243] The language model LM2 is a language model with a relatively smaller scale than the language model LM1.

[0244] In addition, the language model LM1 can also be installed in an information processing device different from the controller 30 mounted on the excavator 100.

[0245] If it is determined that the urgency is low, the urgency determination unit 307 outputs a notification of low urgency together with the text of the instruction obtained by the instruction acquisition unit 301 to the language model selection unit 308.

[0246] When it is determined by the urgency determination unit 307 that the urgency of an instruction from the operator is low, the language model selection unit 308 selects which of the language models LM1 and LM2 to input the text of the instruction obtained by the instruction acquisition unit 301. Specifically, when the content of the instruction obtained by the instruction acquisition unit 301 is relatively simple, the language model selection unit 308 selects the language model LM2, and when the content of the instruction is relatively complex or difficult to understand, the language model selection unit 308 selects the language model LM1.

[0247] For example, the language model selection unit 308 selects either language model LM1 or LM2 according to the length of the text of the instruction obtained by the instruction acquisition unit 301. The length of the text of the instruction is defined, for example, according to the number of characters or words in the text. Specifically, when the length of the text of the instruction is longer than a specified reference, the language model selection unit 308 selects language model LM1, and in other cases, selects language model LM2. That the length of the text is longer than the specified reference may mean that the length of the text is equal to or longer than the specified reference, or that the length of the text is longer than the specified reference.

[0248] In addition, the language model selection unit 308 selects either language model LM1 or LM2 according to the rarity (rarity) of the words included in the text of the instruction obtained by the instruction acquisition unit 301. Specifically, when the number of words with a high rarity (hereinafter referred to as "rare words") included in all the words included in the text of the instruction is larger than a specified reference, the language model selection unit 308 selects language model LM1, and in other cases, selects language model LM2. Rare words are predefined. For example, by applying text matching technology, the language model selection unit 308 determines the presence or absence of rare words included in the text of the instruction and the number of rare words.

[0249] In addition, the language model selection unit 308 may select either language model LM1 or LM2 by considering both the length of the text of the instruction and the rarity of the words included in the text. At this time, the language model selection unit 308 may select language model LM1 when both the first condition indicating that the length of the text of the instruction is longer than a specified reference and the second condition indicating that the number of rare words included in the text of the instruction is larger than a specified reference are satisfied, or may select language model LM1 when either condition is satisfied.

[0250] The language model selection unit 308 outputs the text of the instruction obtained by the instruction acquisition unit 301 and information indicating one of the selected language models LM1 and LM2 to the prompt generation unit 304.

[0251] The prompt generation unit 304 generates a prompt to be input to language model LM1 or language model LM2. The prompt generation unit 304 includes prompt generation units 304A and 304B.

[0252] When the language model LM1 is selected by the language model selection unit 308, the prompt generation unit 304A generates a prompt to be input to language model LM1 according to the instruction obtained by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0253] When the language model LM2 is selected by the language model selection unit 308, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the instruction obtained by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303. The form of the prompt for the language model LM2 may be the same as or different from that of the language model LM1.

[0254] The prompt generation unit 304 outputs the prompt generated by the prompt generation unit 304A to the call unit 305, and outputs the prompt generated by the prompt generation unit 304B to the language model LM2.

[0255] Taking the prompt input from the prompt generation unit 304B as input, the language model LM2 outputs control information (control commands) for the excavator 100 in the same way as the language model LM1.

[0256] If a notification of high urgency is input from the urgency determination unit 307, the action control unit 306 causes the excavator 100 to perform an emergency action corresponding to the instruction, in the same way as in the second example above.

[0257] Furthermore, if the output of the language model LM1 is input from the call unit 305, the action control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1, in the same way as in the first example above.

[0258] Furthermore, if the output of the language model LM2 is input, the action control unit 306 controls the operation of the excavator 100 based on the control information (control commands) corresponding to the output of the language model LM2.

[0259] Thus, in this example, the controller 30 can distinguish between using the language models LM1 and LM2 according to the content of the operator's instruction. Therefore, for example, when the operator's instruction is relatively simple, the communication cost can be suppressed by using the language model LM2 compared to using the language model LM1. And when there is a cost for using the language model LM1, the usage cost of the language model LM1 can be suppressed. On the other hand, when the operator's instruction is relatively difficult to understand or complex, the language model LM1 can be used to ensure the accuracy of the control commands as the interpretation result. Therefore, in this example, the controller 30 can balance the suppression of variable costs and the higher-precision operation of the excavator 100 in response to the operator's natural language instruction.

[0260] In addition, part or all of the instruction acquisition unit 301, object detection unit 302, verbalization unit 303, prompt generation unit 304, call unit 305, motion control unit 306, urgency determination unit 307, language model selection unit 308, and a part of the language model LM2 can also be transferred to an information processing device outside the excavator 100. For example, in the case of remotely operating the excavator 100, the instruction acquisition unit 301, object detection unit 302, verbalization unit 303, prompt generation unit 304, call unit 305, motion control unit 306, urgency determination unit 307, language model selection unit 308, and the language model LM2 can be transferred to the remote operation support device 200.

[0261] [Third Example of Processing Related to Operation Support of Excavator]

[0262] Next, with reference to Figure 14 a third example of processing related to operation support of the excavator 100 will be described.

[0263] Figure 14 is a flowchart schematically showing a third example of processing related to operation support of the excavator 100. In this example, it is premised on the functional structure related to the operation support of the Figure 13 excavator 100.

[0264] Figure 14 The flowchart of

[0265] such as Figure 14 is executed when an instruction from an operator based on natural language is received through the input device 52 or the input device 207. Figure 12 As

[0266] shown, the processing in steps S302, S304, and S306 is the same as the steps S202, S204, and S206 in the above

[0267] and thus the description thereof is omitted.

[0268] If the processing in step S308 is completed, the controller 30 proceeds to step S310.

[0269] In step S310, the controller 30 determines whether the relatively larger language model LM1 is selected according to the determination result of step S308. If the language model LM1 is selected, it proceeds to step S312; if the language model LM2 is selected, it proceeds to step S320.

[0270] The processing of steps S312 and S314 is the same as that of steps S104 and S106 or Figure 10 steps S208 and S210 as described above, so the description is omitted. Figure 12 If the processing of step S314 is completed, the controller 30 proceeds to step S316.

[0271] In step S316, the prompt generation unit 304A generates a prompt to be input to the language model LM1 based on the results of the processing in steps S312 and S314.

[0272] If the processing of step S316 is completed, the controller 30 proceeds to step S318.

[0273] The processing of step S318 is the same as that of step S110 or

[0274] steps S214 as described above, so the description is omitted. Figure 10 If the processing of step S318 is completed and an output is obtained from the language model LM1, the controller 30 proceeds to step S328. Figure 12 On the other hand, the processing of steps S320 and S322 is the same as that of steps S104 and S106 or

[0275] steps S208 and S210 as described above, so the description is omitted.

[0276] If the processing of step S322 is completed, the controller 30 proceeds to step S324. Figure 10 In step S324, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the results of the processing in steps S320 and S322. Figure 12 If the processing of step S324 is completed, the controller 30 proceeds to step S326.

[0277] In step S326, the language model LM2 incorporated in the controller 30 performs calculations with the prompt generated in step S324 as the input and outputs control information (control commands) for the excavator 100.

[0278]

[0279]

[0280]

[0281] ​​​If the process of step S326 is completed, the controller 30 proceeds to step S328.

[0282] In step S328, the motion control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1 or the output of the language model LM2.

[0283] If the process of step S328 is completed, the controller 30 ends the processing of this flowchart.

[0284] On the other hand, the process of step S330 is the same as the process of step S218 described above, so the description is omitted. Figure 12 of step S218, so the description is omitted.

[0285] If the process of step S330 is completed, the controller 30 ends the processing of this flowchart.

[0286] [Fourth Example of Functional Structure Related to Operation Support of Excavator]

[0287] Next, in addition to referring to Figures 1 - 5 also refer to Figure 15 to describe the fourth example of the functional structure related to the operation support of the excavator 100.

[0288] Hereinafter, the same symbols as those of the first to third examples of the above functional structure are assigned to the same or corresponding structures, and the description will be centered on the parts different from the first to third examples of the above functional structure.

[0289] Figure 15 is a functional block diagram showing the fourth example of the functional structure related to the operation support of the excavator 100.

[0290] As Figure 15 shown, the difference from the third example above is that an instruction classifier 307A is provided in the controller 30 instead of the urgency determination unit 307 and the language model selection unit 308.

[0291] The instruction classifier 307A classifies the instructions obtained by the instruction acquisition unit 301 into the first instruction, the second instruction, and the third instruction, which are different types.

[0292] The first instruction is an instruction with a relatively high urgency. The second instruction is an instruction with a relatively low urgency and is relatively difficult to understand or complex. The third instruction is an instruction with a relatively low urgency and is relatively simple.

[0293] That is, the instruction classifier 307A implements the functions of the above urgency determination unit 307 and language model selection unit 308 as a three-class classifier.

[0294] The instruction classifier 307A is a learned model obtained, for example, through supervised learning. The learned model is configured, for example, centering around a neural network, and is obtained by optimizing the base model using an algorithm of the error backpropagation method based on the error between the inference result and the training data. Also, the learned model can be configured centering around a support vector machine.

[0295] If the instruction acquired by the instruction acquisition unit 301 is classified as a first instruction, the instruction classifier 307A outputs the text of the instruction acquired by the instruction acquisition unit 301 and a notification that it is a first instruction to the action control unit 306.

[0296] If a notification that the instruction acquired by the instruction acquisition unit 301 is a first instruction is input from the instruction classifier 307A, the action control unit 306 causes the excavator 100 to execute an emergency action corresponding to the instruction.

[0297] Also, if the instruction acquired by the instruction acquisition unit 301 is classified as a second instruction or a third instruction, the instruction classifier 307A outputs the text of the instruction acquired by the instruction acquisition unit 301 and a notification that it is a second instruction or a third instruction to the prompt generation unit 304.

[0298] If a notification that it is a second instruction is input from the instruction classifier 307A, the prompt generation unit 304A generates a prompt to be input to the language model LM1 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0299] If a notification that it is a third instruction is input from the instruction classifier 307A, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0300] In this way, in this example, the controller 30 can incorporate the functions of the urgency determination unit 307 and the language model selection unit 308 into the instruction classifier 307A equivalent to a three-class classifier.

[0301] In addition, a part or all of the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the call unit 305, the action control unit 306, the instruction classifier 307A, and the language model LM2 may also be transferred to an information processing device outside the excavator 100. For example, in the case of remotely operating the excavator 100, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the call unit 305, the action control unit 306, the instruction classifier 307A, and the language model LM2 may be transferred to the remote operation support device 200.

[0302] [Fourth Example of Processing Related to Operation Support of Excavator]

[0303] Next, a fourth example of the processing related to the operation support of the excavator 100 will be described with reference to Figure 16 FIG.

[0304] Figure 16 FIG. is a flowchart schematically showing a fourth example of the processing related to the operation support of the excavator 100. In this example, it is premised on the functional structure related to the operation support of the excavator 100 Figure 15 of

[0305] Figure 16 The flowchart of FIG. is executed, for example, when an instruction from an operator based on natural language is received through the input device 52 or the input device 207.

[0306] As Figure 16 shown, the processing in step S402 is the same as the processing in step S102 of the above Figure 10 etc., and thus the description thereof is omitted.

[0307] If the processing in step S402 is completed, the controller 30 proceeds to step S404.

[0308] In step S404, the instruction classifier 307A classifies the instruction obtained in step S402 into any one of a first instruction, a second instruction, and a third instruction.

[0309] If the processing in step S404 is completed, the controller 30 proceeds to step S406.

[0310] In step S406, the controller 30 determines which one of the first instruction, the second instruction, and the third instruction the classification result in step S404 is. When the classification result is the second instruction, the controller 30 proceeds to step S408; when the classification result is the third instruction, the controller 30 proceeds to step S416; and when the classification result is the first instruction, the controller 30 proceeds to step S426.

[0311] The processing in steps S408, S410, S412, and S414 is the same as the processing in steps S312, S314, S316, and S318 of the above Figure 14 etc., and thus the description thereof is omitted.

[0312] If the processing in step S414 is completed, the controller 30 proceeds to step S424.

[0313] Moreover, the processing in steps S416, S418, S420, and S422 is the same as the processing in steps S320, S322, S324, and S326 of the above Figure 14 etc., and thus the description thereof is omitted.

[0314] If the process of step S422 is completed, the controller 30 proceeds to step S424.

[0315] The process of step S424 is the same as the process of step S328 described above Figure 14 and thus will not be described again.

[0316] If the process of step S424 is completed, the controller 30 ends the processing of this flowchart.

[0317] The process of step S426 is the same as the process of step S330 described above Figure 14 and thus will not be described again.

[0318] If the process of step S426 is completed, the controller 30 ends the processing of this flowchart.

[0319] [The Fifth Example of the Functional Structure Related to the Operation Support of the Excavator]

[0320] Next, in addition to referring to Figures 1 - 5 in addition, reference is also made to Figure 17 to describe the fifth example of the functional structure related to the operation support of the excavator 100.

[0321] Hereinafter, the same symbols as those of the first to fourth examples of the above functional structure will be used for the same or corresponding structures, and the description will be centered on the parts different from the first to fourth examples of the above functional structure.

[0322] Figure 17 is a functional block diagram showing the fifth example of the functional structure related to the operation support of the excavator 100.

[0323] As Figure 17 shown, similar to the first example above, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, and an action control unit 306. And, different from the first example above, the controller 30 includes a notification unit 309.

[0324] The notification unit 309 notifies the operator in advance of the scheduled action content of the excavator 100 through the display device or the display device 208 as the output device 50, based on the output of the language model LM1 input from the call unit 305 (i.e., the control information (control command) of the excavator 100). Thus, the operator can confirm whether the natural language-based instruction input by his own voice or text is properly understood.

[0325] In addition, the notification unit 309 can also notify the operator of the information on the operation steps of the excavator 100 on the current day together with the action content scheduled for the excavator 100. Thus, the operator can confirm whether the scheduled action content of the excavator 100 conforms to the operation steps of the excavator 100 on the current day by using an instruction input in his own voice or text.

[0326] After the notification is implemented by the notification unit 309, if an input permitting the action is received from the operator through the input device 52 or the input device 207, the action control unit 306 controls the action of the excavator 100 according to the output of the language model LM1.

[0327] On the other hand, after the notification is implemented by the notification unit 309, if an input not permitting the action is received from the operator through the input device 52 or the input device 207, the action control unit 306 aborts the action control of the excavator 100 based on the output of the language model LM1. The same applies to the case where the operator neither inputs permission for the action nor inputs non - permission for the action after the notification is implemented by the notification unit 309.

[0328] In this way, in this example, the controller 30 can implement the action of the excavator 100 according to the natural - language - based instruction from the operator when the operator has pre - confirmed and permitted the action content of the excavator 100. Therefore, the controller 30 can suppress inappropriate actions of the excavator 100, thereby improving the safety of the excavator 100.

[0329] [The 5th example of the process related to the operation support of the excavator]

[0330] Next, refer to Figure 18 and explain the 5th example of the process related to the operation support of the excavator 100.

[0331] Figure 18 The flowchart is executed when an instruction from the operator based on natural language is received through the input device 52 or the input device 207.

[0332] As Figure 18 shown, the processing of steps S502, S504, S506, S508, and S510 is the same as the processing of steps S102, S104, S106, S108, and S110 described above, so the description is omitted. Figure 10

[0333] If the processing of step S510 is completed, the controller 30 proceeds to step S512.

[0334] In step S512, the notification unit 309 gives a prior notification of the action content of the excavator 100 to the operator.​

[0335] If the process of step S512 is completed, the controller 30 proceeds to step S514.

[0336] In step S514, the controller 30 determines whether an input permitting the operation is received from the operator. If the controller 30 receives an input permitting the operation from the operator within a specified time after the notification, it proceeds to step S516. Otherwise, the processing of this flowchart ends.

[0337] The process of step S516 is the same as the process of step S112 described above, Figure 10 so the description is omitted.

[0338] If the process of step S516 is completed, the controller 30 ends the processing of this flowchart.

[0339] [Another Embodiment]

[0340] Next, another embodiment will be described.

[0341] The above-described embodiments can also be combined as appropriate, or modified or changed.

[0342] For example, in the third example of the above functional structure, the urgency determination unit 307 can be omitted. In this case, the language model selection unit 308 selects either language model LM1 or LM2 regardless of the level of urgency of the instruction obtained by the instruction acquisition unit 301.

[0343] Moreover, in the second to fourth examples of the above functional structure or their modified / changed examples, a function similar to the notification unit 309 of the fifth example of the above functional structure can also be added. However, when it is determined that the urgency of the instruction obtained by the instruction acquisition unit 301 is relatively high, or when the instruction is classified as the above first instruction, a prior notification of the operation content is not given to the operator.

[0344] Furthermore, the first to fifth examples of the above functional structure and processing related to operation support, or their modified / changed examples, can also be applied to construction machinery other than the excavator 100. Other construction machinery includes, for example, bulldozers, mobile cranes, etc.

[0345] [Function]

[0346] Next, the functions of the construction machinery, information processing device, and program according to this embodiment will be described.

[0347] In the first mode of the present embodiment, the construction machine includes an environmental information acquisition unit, a verbalization unit, an instruction acquisition unit, and a control unit. The construction machine is, for example, the excavator 100 described above. The environmental information acquisition unit is, for example, the imaging device 40 described above. The verbalization unit is, for example, the verbalization unit 303 described above. The instruction acquisition unit is, for example, the instruction acquisition unit 301 described above. The control unit is, for example, the motion control unit 306 described above. Specifically, the environmental information acquisition unit acquires information indicating the environment around the construction machine. And, the verbalization unit verbalizes the information acquired by the environmental information acquisition unit in natural language. And, the instruction acquisition unit acquires an instruction in natural language from an operator. And, the control unit controls the operation of the construction machine based on the result obtained by having a language model interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit. The language model is, for example, the language model LM1 or the language model LM2 described above.

[0348] And, in the first mode of the present embodiment, the operation support system may include the environmental information acquisition unit, the verbalization unit, the instruction acquisition unit, and the control unit. The operation support system is, for example, the remote operation support system SYS.

[0349] And, in the first mode of the present embodiment, the information processing device may include: a verbalization unit that verbalizes information indicating the environment around the construction machine in natural language; the instruction acquisition unit; and the control unit. The information processing device is, for example, the controller 30 or the remote operation support device 200 described above.

[0350] And, in the first mode of the present embodiment, the program may cause the information processing device to execute a verbalization step, an instruction acquisition step, and a control step. Specifically, in the verbalization step, information indicating the environment around the construction machine is verbalized in natural language. And, in the instruction acquisition step, an instruction in natural language from an operator is acquired. Then, in the control step, the operation of the construction machine is controlled based on the result obtained by having a language model interpret the instruction acquired in the instruction acquisition step and the information verbalized in the verbalization step.

[0351] Thereby, an operator can, for example, operate the construction machine according to an instruction in natural language that matches the surrounding environment of the construction machine. Therefore, the construction machine, the operation support system, or the information processing device (hereinafter referred to as "construction machine, etc.") can improve the effectiveness of the operation of the construction machine according to an instruction in natural language from an operator.

[0352] And, in the second mode of the present embodiment, on the premise of the first mode described above, in addition to the information acquired by the environmental information acquisition unit, the verbalization unit may also verbalize the information of the construction drawing in natural language.

[0353] Accordingly, an operator can, for example, operate a construction machine according to an instruction that conforms to the content of the construction drawing using natural language. Therefore, the construction machine and the like can improve the effectiveness of the operation of the construction machine according to the instruction of the operator using natural language.

[0354] Moreover, in the third mode of the present embodiment, on the premise of the first or second mode described above, the construction machine and the like may include a determination unit that determines the level of urgency of the instruction acquired by the instruction acquisition unit. The determination unit is, for example, the above-mentioned urgency determination unit 307. And, the control unit may cause the construction machine to execute a prescribed action corresponding to the content of the instruction acquired by the instruction acquisition unit when it is determined by the determination unit that the urgency is high, and control the action of the construction machine according to the result obtained by causing the language model to interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit when it is determined by the determination unit that the urgency is low.

[0355] Accordingly, the construction machine can quickly execute an emergency action corresponding to the instruction for an instruction with a relatively high level of urgency.

[0356] Moreover, in the fourth mode of the present embodiment, on the premise of the third mode described above, the determination unit may determine that the urgency is high when a prescribed word indicating a high level of urgency is included in the text of the instruction acquired by the instruction acquisition unit.

[0357] Accordingly, the construction machine and the like can appropriately determine the urgency of the instruction.

[0358] Moreover, in the fifth mode of the present embodiment, on the premise of any one of the first to fourth modes described above, the language model may include: a first language model that is provided externally so as to be able to communicate with the construction machine and has a relatively large scale; and a second language model that is incorporated into the construction machine and has a relatively small scale. The first language model is, for example, the above-mentioned language model LM1. The second language model is, for example, the above-mentioned language model LM2. And, the construction machine and the like may include a selection unit that selects which one of the first language model and the second language model to use according to the content of the instruction acquired by the instruction acquisition unit. The selection unit is, for example, the above-mentioned language model selection unit 308. And, the control unit may control the action of the construction machine according to the result obtained by causing the language model selected by the selection unit to interpret the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit.

[0359] Accordingly, construction machinery and the like can distinguish between using the first language model and the second language model according to the indicated content. Therefore, construction machinery and the like can, for example, suppress the communication cost or usage cost for using the first model.

[0360] Moreover, in the sixth aspect of the present embodiment, on the premise of the fifth aspect described above, the selection unit may select which one of the first language model and the second language model to use based on at least one of the length of the indicated article obtained by the indication acquisition unit and the rarity of the words included in the indication.

[0361] Accordingly, construction machinery and the like can use the first language model when the indicated article is relatively long or the rarity of the words included in the indication is relatively high, and use the second language model in other cases. Therefore, construction machinery and the like can suppress the communication cost or usage cost for using the first model while ensuring the appropriateness and accuracy of the actions of the construction machinery in response to the operator's instructions.

[0362] Moreover, in the seventh aspect of the present embodiment, on the premise of the first or second aspect described above, the language model may include: a first language model, which is provided externally so as to be able to communicate with the construction machinery with each other and has a relatively large scale; and a second language model, which is incorporated into the construction machinery and has a relatively small scale. And construction machinery and the like may be provided with a classification unit that classifies the indication obtained by the indication acquisition unit into a first indication for causing the construction machinery to perform an emergency action, a second indication for controlling the construction machinery according to the result of interpreting the indication by the first language model, and a third indication for controlling the construction machinery according to the result of interpreting the indication by the second language model. The classification unit is, for example, the above-mentioned indication classifier 307A. And the control unit may control the action of the construction machinery corresponding to the indication obtained by the indication acquisition unit according to the classification result of the classification unit.

[0363] Accordingly, the functions of the above-mentioned determination unit and selection unit can be implemented by the classification unit of construction machinery and the like, which is equivalent to a three-class classifier.

[0364] Moreover, in the eighth aspect of the present embodiment, on the premise of any one of the first to sixth aspects described above, construction machinery and the like may be provided with a notification unit that notifies the operator in advance of the action that the control unit causes the construction machinery to perform in response to the indication obtained by the indication acquisition unit. And the control unit may control the action of the construction machinery according to the result of interpreting the indication obtained by the indication acquisition unit and the information verbalized by the verbalization unit by the language model after obtaining permission from the operator after the notification unit makes a notification.

[0365] Accordingly, the construction machine can enable the operator to pre-confirm the operation content of the excavator based on the instruction content, and, on the premise of obtaining permission, perform the operation based on the operator's instruction. Therefore, the construction machine and the like can suppress inappropriate operations of the construction machine, thereby improving the safety of the construction machine.

[0366] As described above, the embodiments have been described in detail, but the present invention is not limited to the specific embodiments, and various modifications / changes can be made within the scope of the gist described in the technical solution.

Claims

1. A construction machine comprising: An environmental information acquisition unit that acquires information indicating an environment surrounding the construction machine; A language conversion unit converts the information acquired by the environmental information acquisition unit into a natural language; an instruction acquisition unit that acquires an instruction using natural language from an operator; and The control unit controls the operation of the construction machine based on a result obtained by causing a language model to interpret the instruction acquired by the instruction acquisition unit and the information converted into language by the language conversion unit.

2. The construction machine according to claim 1, wherein: The language conversion unit converts information of the construction drawing into a natural language in addition to the information acquired by the environment information acquisition unit.

3. The construction machine according to claim 1 or 2, comprising a determination unit for determining whether the urgency of the instruction acquired by the instruction acquisition unit is high or low. When the determination unit determines that the urgency is high, the control unit causes the construction machine to execute a prescribed action corresponding to the content of the instruction acquired by the instruction acquisition unit. When the determination unit determines that the urgency is low, the control unit controls the action of the construction machine based on the result obtained by causing the language model to interpret the instructions acquired by the instruction acquisition unit and the information linguized by the linguistic unit.

4. The construction machine according to claim 3, wherein: The determination unit determines that the urgency is high when the text of the instruction acquired by the instruction acquisition unit includes a predetermined word indicating that the urgency is high.

5. The construction machine according to claim 1 or 2, wherein: The language model includes: a first language model which is externally provided so as to be able to communicate with the construction machine and is relatively large in size; and a second language model which is incorporated into the construction machine and is relatively small in size. The construction machine includes a selection unit that selects which of the first language model and the second language model to use based on the content of the instruction acquired by the instruction acquisition unit. The control unit controls the operation of the construction machine based on a result obtained by causing the language model selected by the selection unit to interpret the instruction acquired by the instruction acquisition unit and the information converted to language by the language conversion unit.

6. The construction machine according to claim 5, wherein: The selection unit selects which of the first language model and the second language model to use based on at least one of the length of the text of the instruction acquired by the instruction acquisition unit and the rarity of words included in the instruction.

7. The construction machine according to claim 1 or 2, wherein: The language model includes: a first language model which is externally provided so as to be able to communicate with the construction machine and is relatively large in size; and a second language model which is incorporated into the construction machine and is relatively small in size. The construction machine includes a classification unit, which classifies the instructions acquired by the instruction acquisition unit into a first instruction for causing the construction machine to perform an emergency action, a second instruction for controlling the construction machine based on a result obtained by causing the first language model to interpret the instruction, and a third instruction for controlling the construction machine based on a result obtained by causing the second language model to interpret the instruction. The control unit controls the operation of the construction machine corresponding to the instruction acquired by the instruction acquisition unit based on the classification result of the classification unit.

8. The construction machine according to claim 1 or 2, comprising a notification unit for notifying an operator in advance of an action to be performed by the control unit in response to the instruction acquired by the instruction acquisition unit. When the control unit obtains permission from the operator after the notification by the notification unit, the control unit controls the operation of the construction machine based on a result obtained by causing a language model to interpret the instruction acquired by the instruction acquisition unit and the information linguized by the linguistic unit.

9. An operation support system comprising: An environmental information acquisition unit that acquires information indicating an environment surrounding the construction machine; a language conversion unit, which converts the information acquired by the environmental information acquisition unit into a natural language; an instruction acquisition unit that acquires an instruction using natural language from an operator; and The control unit controls the operation of the construction machine based on a result obtained by causing a language model to interpret the instruction acquired by the instruction acquisition unit and the information converted into language by the language conversion unit.

10. An information processing device comprising: a language conversion unit that converts information representing the environment surrounding the construction machine into a natural language; an instruction acquisition unit that acquires an instruction using natural language from an operator; and The control unit controls the operation of the construction machine based on a result obtained by causing a language model to interpret the instruction acquired by the instruction acquisition unit and the information converted into language by the language conversion unit.

11. A program causing an information processing device to execute the following steps: a verbalization step of verbalizing information representing the environment surrounding the construction machine using a natural language; an instruction obtaining step of obtaining an instruction using a natural language from an operator; and The control step controls the operation of the construction machine based on a result obtained by causing a language model to interpret the instruction acquired in the instruction acquisition step and the information converted into language by the conversion step.

Images