Work machine, operation support system, information processor, and program

Abstract

To provide a technology that can improve effectiveness of an operation of a work machine based on instructions given by an operator in natural language.SOLUTION: A shovel 100 comprises: an imaging device 40 acquiring information representing an environment surrounding the shovel 100; a verbalization portion 303 verbalizing the information acquired by the imaging device 40 in natural language; an instruction acquisition portion 301 acquiring instructions in natural language from an operator; and an operation control portion 306 controlling an operation of the shovel 100 based on results of interpreting the instructions acquired by the instruction acquisition portion 301 and the information verbalized by the verbalization portion 303, using a language model LM1.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to a working machine and the like.

Background Art

[0002] Conventionally, a technique for operating a working machine based on an instruction in a natural language such as voice or text input has been known (see Patent Document 1).

[0003] In Patent Document 1, an instruction by voice from an operator can be voice-recognized, and an operation corresponding to the content of the voice recognition can be performed on the working machine.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, only specific operations of the working machine such as instructions for moving forward, backward, left, or right and instructions for excavation can be given, and for example, instructions for operations according to the working environment cannot be given.

[0006] Therefore, in view of the above problems, an object is to provide a technique capable of improving the effectiveness of operating a working machine according to an instruction in the natural language of an operator.

Means for Solving the Problems

[0007] To achieve the above object, in one embodiment of the present disclosure, an environment information acquisition unit that acquires information representing the environment around the working machine; a language conversion unit that converts the information acquired by the environment information acquisition unit into a natural language; an instruction acquisition unit that acquires an instruction in a natural language from an operator; A control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit. A working machine is provided.

[0008] In another embodiment of the present disclosure, An environment information acquisition unit that acquires information representing the environment around the working machine, A verbalization unit that verbalizes the information acquired by the environment information acquisition unit in natural language, An instruction acquisition unit that acquires an instruction in natural language from an operator, A control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit. An operation support system is provided.

[0009] In still another embodiment of the present disclosure, A verbalization unit that verbalizes information representing the environment around the working machine in natural language, An instruction acquisition unit that acquires an instruction in natural language from an operator, A control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit. An information processing apparatus is provided.

[0010] In still another embodiment of the present disclosure, In the information processing apparatus, A verbalization step of verbalizing information representing the environment around the working machine in natural language, An instruction acquisition step of acquiring an instruction in natural language from an operator, A control step of controlling the operation of the working machine based on the result of interpreting the instruction acquired in the instruction acquisition step and the information verbalized in the verbalization step. A program is provided.

Advantages of the Invention

[0011] According to the above embodiment, the effectiveness of operating a working machine according to an operator's instruction in natural language can be improved.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described with reference to the drawings.

[0014] [Outline of Excavator] With reference to FIGS. 1 to 3, the outline of the excavator 100 according to the present embodiment will be described.

[0015] FIG. 1 is a side view showing an example of the excavator 100. FIG. 2 is a top view showing an example of the excavator 100. FIG. 3 is a diagram showing an example of a configuration related to remote operation of the excavator 100. Hereinafter, when explaining the direction in the excavator 100 or the direction viewed from the excavator 100, the direction in which the attachment AT extends in the top view of the excavator 100 (the upward direction in FIG. 2) may be defined as "front".

[0016] As shown in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper swing body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

[0017] The lower traveling body 1 uses crawlers 1C to move the excavator 100. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CL is hydraulically driven by a traveling hydraulic motor 1MR. Thereby, the lower traveling body 1 can travel by itself.

[0018] The upper revolving body 3 is mounted on the lower traveling body 1 via a slewing mechanism 2 so as to be slewing capable (slewing freely). For example, the upper revolving body 3 slews with respect to the lower traveling body 1 when the slewing mechanism 2 is hydraulically driven by a slewing hydraulic motor 2M.

[0019] The boom 4 is attached to the center of the front part of the upper revolving body 3 so as to be capable of pitching about a rotation axis along the left-right direction. The arm 5 is attached to the tip of the boom 4 so as to be rotatable about a rotation axis along the left-right direction. The bucket 6 is attached to the tip of the arm 5 so as to be rotatable about a rotation axis along the left-right direction.

[0020] The bucket 6 is an example of an end attachment, and is used, for example, in excavation work, slope work, leveling work, etc.

[0021] The bucket 6 is attached to the tip of the arm 5 in a mode that can be appropriately replaced according to the work content of the excavator 100. That is, instead of the bucket 6, a bucket of a type different from the bucket 6, for example, a relatively large large bucket, a slope bucket, a dredging bucket, etc. may be attached to the tip of the arm 5. Further, an end attachment of a type other than the bucket, for example, a stirrer, a breaker, a crusher, etc. may be attached to the tip of the arm 5. Further, a preliminary attachment such as a quick coupling or a tilt rotator may be provided between the arm 5 and the end attachment.

[0022] The boom 4, the arm 5, and the bucket 6 are each hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0023] The cab 10 is a cab for an operator to board and operate the excavator 100. The cab 10 is mounted, for example, on the left side of the front part of the upper revolving body 3.

[0024] The excavator 100 may be equipped with a communication device 60 and be capable of communicating with the remote operation support device 200 through a predetermined communication line NW.

[0025] The communication line NW includes, for example, a local area network (LAN) at the work site. The communication line NW may also include a wide area network (WAN). The wide area network includes, for example, a mobile communication network with a base station as the terminal, a satellite communication network using communication satellites, the Internet, etc. The communication line NW may also include a short-range communication line based on wireless communication standards such as WiFi and Bluetooth (registered trademark).

[0026] For example, the excavator 100 operates driven elements such as the lower traveling body 1 (i.e., a pair of left and right crawlers 1CL, 1CR), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6 according to the operation of an operator boarding the cab 10.

[0027] Alternatively, or in addition to being configured to be operable by an operator boarding the cab 10, the excavator 100 may be configured to be remotely operable from outside the excavator 100. When the excavator 100 is remotely operated, the inside of the cab 10 may be unmanned. Also, when the excavator 100 is dedicated to remote operation, the cab 10 may be omitted. Hereinafter, the description will proceed on the premise that the operator's operation includes at least one of the operation on the operator's operating device 26 in the cab 10 and the remote operation by an external operator.

[0028] For example, as shown in FIG. 3, for remote operation, a remote operation support system SYS is used in which an excavator 100 to be operated and a remote operation support device 200 for an operator to operate the excavator 100 are communicably connected through a communication line NW. Specifically, remote operation includes a mode in which the excavator 100 is operated by an operation input regarding an actuator of the excavator 100 performed by the remote operation support device 200 that can communicate with the excavator 100 through the communication line NW.

[0029] The remote operation support device 200 is provided, for example, at a management center or the like that manages the work of the excavator 100 from the outside. Further, the remote operation support device 200 may be a portable operation terminal. In this case, the operator can perform remote operation of the excavator 100 while directly checking the working status of the excavator 100 from around the excavator 100.

[0030] The excavator 100 transmits, for example, via the communication device 60, an image (hereinafter, "peripheral image") representing the state of the surroundings including the front of the excavator 100 based on the captured image output by the imaging device 40 mounted on itself to the remote operation support device 200. Further, the excavator 100 transmits, via the communication device 60, the captured image output by the imaging device 40 to the remote operation support device 200, and the remote operation support device 200 may process the captured image received from the excavator 100 to generate a peripheral image. Then, the remote operation support device 200 causes a peripheral image representing the state of the surroundings including the front of the excavator 100 to be displayed on its own display device. Also, various information images (information screens) displayed on the output device 50 (display device) inside the cab 10 of the excavator 100 may similarly be displayed on the display device of the remote operation support device 200. Thereby, an operator using the remote operation support device 200 can remotely operate the excavator 100 while checking, for example, the display contents such as the image and the information screen representing the state of the surroundings of the excavator 100 displayed on the display device. Then, the excavator 100 operates the actuator according to a signal received from the remote operation support device 200 via the communication device 60 and representing the content of the remote operation, and operates driven elements such as the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6. Thereby, the remote operation support system SYS can realize remote operation of the excavator 100 using the remote operation support device 200.

[0031] Also, the remote operation may include, for example, a mode in which the excavator 100 is operated by an external voice input or gesture input to the excavator 100 by a person (for example, an operator) around the excavator 100. Specifically, the excavator 100 recognizes voices spoken by surrounding workers or gestures made by workers or the like through a voice input device (for example, a microphone) or a gesture input device (for example, an imaging device) mounted on itself. Then, the excavator 100 may operate the actuator according to the content of the recognized voice, gesture, or the like, and drive driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

[0032] Further, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. Thereby, the excavator 100 can realize a function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper swing body 3, and the attachment AT, that is, a so-called "automatic operation function" or "Machine Control (MC) function".

[0033] The automatic operation function includes, for example, a semi-automatic operation function (operation support type MC function). The semi-automatic operation function is a function of automatically operating a driven element (actuator) other than the driven element (actuator) of the operation target in response to the operator's operation. Further, the automatic operation function may include a fully automatic operation function (fully automatic type MC function). The fully automatic operation function is a function of automatically operating at least a part of a plurality of driven elements (actuators) on the premise that there is no operator operation. In the excavator 100, when the fully automatic operation function is valid, the inside of the cab 10 may be unmanned. Further, the semi-automatic operation function, the fully automatic operation function, etc. include, for example, a rule-based automatic operation function. The rule-based automatic operation function is an automatic operation function in a mode in which the operation content of the driven element (actuator) of the automatic operation target is automatically determined according to a rule defined in advance. Further, the semi-automatic operation function, the fully automatic operation function, etc. may include an autonomous operation function. The autonomous operation function is an automatic operation function in a mode in which the excavator 100 autonomously makes various judgments, and the operation content of the driven element (actuator) of the automatic operation target is determined according to the judgment result.

[0034] Further, the operation of the excavator 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 200 may be provided. The remote monitoring support device is, for example, the remote operation support device 200. Thereby, a monitor who is a user of the remote monitoring support device can monitor the operation status of the excavator 100 while checking the surrounding images displayed on the display device of the remote monitoring support device. Further, for example, when the monitor determines it is necessary from the viewpoint of safety, the monitor can intervene in the operation by the operator of the excavator 100 or the automatic driving and stop the excavator 100 urgently by performing a predetermined input using the input device of the remote monitoring support device.

[0035] [Configuration of Excavator] Next, with reference to FIG. 4, the configuration of the excavator 100 will be described.

[0036] FIG. 4 is a block diagram showing an example of the configuration of the excavator 100.

[0037] In addition, in FIG. 4, the path through which mechanical power is transmitted is shown by a double line, the path through which high-pressure hydraulic oil for driving the hydraulic actuator HA flows is shown by a solid line, the path through which pilot pressure is transmitted is shown by a broken line, and the path through which an electric signal is transmitted is shown by a dotted line.

[0038] The excavator 100 includes respective components such as a hydraulic drive system related to the hydraulic drive of the driven elements, an operation system related to the operation of the driven elements, a user interface system related to the information exchange with the user, a communication system related to the communication with the outside, and a control system related to various controls.

[0039] [Hydraulic Drive System] As shown in FIG. 4, the hydraulic drive system of the excavator 100 includes, as described above, hydraulic actuators HA that hydraulically drive each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6. Further, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0040] The hydraulic actuator HA includes traveling hydraulic motors 1ML and 1MR, swing hydraulic motor 2M, boom cylinder 7, arm cylinder 8, bucket cylinder 9, and the like.

[0041] Furthermore, in the excavator 100, part or all of the hydraulic actuator HA may be replaced with an electric actuator. That is, the excavator 100 may be a hybrid excavator or an electric excavator.

[0042] The engine 11 is the prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper swing body 3. The engine 11 rotates at a constant preset target rotational speed under the direct or indirect control of a controller 30 described later, and drives the main pump 14 and the pilot pump 15.

[0043] Furthermore, instead of or in addition to the engine 11, other prime movers (for example, an electric motor) or the like may be mounted on the excavator 100.

[0044] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate (hereinafter, "tilt angle") of the main pump 14 in response to a control command from the controller 30.

[0045] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper swing body 3, similar to the engine 11. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the stroke length of the piston is adjusted by adjusting the tilt angle of the swash plate by the regulator 13, and the discharge flow rate and discharge pressure are controlled.

[0046] The control valve 17 drives the hydraulic actuator HA according to the operation on the operator's operating device 26, the content of the remote operation, or the operation command corresponding to the automatic driving function. The control valve 17 is mounted, for example, at the center of the upper swing body 3. As described above, the control valve 17 is connected to the main pump 14 via the high-pressure hydraulic line, and selectively supplies the hydraulic oil supplied from the main pump 14 to each hydraulic actuator according to the operation of the operator or the operation command corresponding to the automatic driving function. Specifically, the control valve 17 includes a plurality of control valves (direction switching valves) that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the hydraulic actuators HA.

[0047] <Operation system> As shown in FIG. 4, the operation system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0048] The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper swing body 3 like the engine 11. The pilot pump 15 is, for example, a fixed-displacement hydraulic pump and is driven by the engine 11 as described above.

[0049] Note that the pilot pump 15 may be omitted. In this case, the relatively low-pressure hydraulic oil after the relatively high-pressure hydraulic oil discharged from the main pump 14 is decompressed by a predetermined decompression valve may be supplied as the pilot pressure to various hydraulic devices.

[0050] The operating device 26 is provided near the driver's seat in the cabin 10 and is used for the operator to operate various driven elements. Specifically, the operating device 26 is used for the operator to operate the hydraulic actuators HA that drive the respective driven elements. As a result, the operator can operate the driven elements driven by the hydraulic actuators HA. The operating device 26 includes a pedal device and a lever device for operating the respective driven elements (hydraulic actuators HA).

[0051] For example, as shown in FIG. 4, the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 utilizes the hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched therefrom, and outputs a pilot pressure corresponding to the operation content to the secondary pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and is connected to the control valve 17 through the pilot line 27 connected to the outlet port of the shuttle valve 32. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators HA) in the operating device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26 by the operator or the like.

[0052] Also, the operating device 26 may be an electric type. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter, "operation signal") corresponding to the operation content, and the operation signal is taken into the controller 30. Then, the controller 30 outputs a control command corresponding to the content of the operation signal, that is, a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. Thereby, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0053] Also, the control valve (direction switching valve) for driving each hydraulic actuator HA incorporated in the control valve 17 may be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve 17 (that is, to the electromagnetic solenoid type control valve).

[0054] Also, as described above, part or all of the hydraulic actuator HA may be replaced with an electric actuator. In this case, the controller 30 may output a control command corresponding to the operation content of the operation device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver that drives the electric actuator. Further, when the excavator 100 is remotely operated or when the excavator 100 is operated by natural language as described later, the operation device 26 may be omitted.

[0055] The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA) (for example, the raising direction and the lowering direction of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to be able to change its flow passage area (that is, the cross-sectional area through which the hydraulic oil can flow). Thereby, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary pilot line 27B by using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly act on the control valve 17 a predetermined pilot pressure corresponding to the control signal from the controller 30 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Thus, for example, the controller 30 can supply to the control valve 17 a pilot pressure corresponding to an operation command corresponding to the automatic operation function from the hydraulic control valve 31, and realize the operation of the excavator 100 by the automatic operation function.

[0056] Also, the controller 30 may control the hydraulic control valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs to the hydraulic control valve 31 a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 200 by the communication device 60. Thereby, the controller 30 can supply to the control valve 17 a pilot pressure corresponding to the content of the remote operation from the hydraulic control valve 31, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0057] Also, when the operating device 26 is electric, the controller 30 can directly supply to the control valve 17 a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the hydraulic control valve 31, and realize the operation of the excavator 100 based on the operation of the operator.

[0058] The shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valve 32 is connected to the secondary pilot line 27A of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator HA which is the operation target of the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher one of the pilot pressure of the secondary pilot line 27A of the operating device 26 and the pilot pressure of the secondary pilot line 27B of the hydraulic control valve 31 to the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the secondary pilot pressure of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper slewing body 3, boom 4, arm 5, bucket 6) without depending on the operation state of the operator on the operating device 26, and can realize the automatic operation function and the remote operation function.

[0059] The hydraulic control valve 33 is provided in a pilot line 27A that connects the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow passage area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. Thereby, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Further, the controller 30 can, for example, reduce the pilot pressure output from the operating device 26 and make it lower than the pilot pressure output from the hydraulic control valve 31 even when the operating device 26 is being operated. Therefore, by controlling the hydraulic control valve 31 and the hydraulic control valve 33, the controller 30 can surely cause a desired pilot pressure to act on the pilot port of the control valve in the control valve 17, for example, regardless of the operation content of the operating device 26. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0060] <User interface system> As shown in FIG. 4, the user interface system of the excavator 100 includes an operating device 26, an output device 50, and an input device 52.

[0061] The output device 50 outputs various information to the user of the excavator 100 (for example, the operator in the cab 10 or the operator of external remote operation) and the people around the excavator 100 (for example, the worker or the driver of the work vehicle).

[0062] For example, the output device 50 includes lighting devices, display devices, etc. that output various types of information in a visual manner. The lighting device is, for example, a warning light (indicator lamp), etc. The display device is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc. For example, as shown in FIG. 2, the lighting device and the display device may be provided inside the cabin 10 and output various types of information to the operator inside the cabin 10 in a visual manner. Also, the lighting device and the display device may be provided, for example, on the side surface of the upper swing body 3 and output various types of information to the workers around the excavator 100 in a visual manner.

[0063] Further, the output device 50 may include a sound output device that outputs various types of information in an auditory manner. The sound output device includes, for example, a buzzer, a speaker, etc. The sound output device is provided, for example, on at least one of the inside and outside of the cabin 10 and may output various types of information to the operator inside the cabin 10 and the people (such as workers) around the excavator 100 in an auditory manner.

[0064] Also, the output device 50 may include a device that outputs various types of information in a tactile manner such as vibration of the driver's seat.

[0065] The input device 52 receives various inputs from the user of the excavator 100, and the signal corresponding to the received input is taken into the controller 30. For example, as shown in FIG. 2, the input device 52 is provided inside the cabin 10 and receives inputs from the operator inside the cabin 10, etc. Also, the input device 52 may be provided, for example, on the side surface of the upper swing body 3 and receive inputs from the workers around the excavator 100, etc.

[0066] For example, the input device 52 includes a mechanical input device that receives inputs by mechanical operations from the user. The mechanical input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operating device 26 (lever device), etc.

[0067] In addition, the input device 52 may include a voice input device that receives a user's voice input. The voice input device includes, for example, a microphone.

[0068] In addition, the input device 52 may include a gesture input device that receives a user's gesture input. The gesture input device includes, for example, an imaging device that images the state of a gesture made by the user.

[0069] In addition, the input device 52 may include a biometric input device that receives a user's biometric input. Biometric input includes, for example, input of biometric information such as a user's fingerprint and iris.

[0070] <Communication system> As shown in FIG. 4, the communication system of the excavator 100 according to the present embodiment includes a communication device 60.

[0071] The communication device 60 is connected to an external communication line NW and communicates with a device provided separately from the excavator 100. Devices provided separately from the excavator 100 may include devices outside the excavator 100 and also portable terminal devices (mobile terminals) brought into the cab 10 by the user of the excavator 100. The communication device 60 may include, for example, a mobile communication module compliant with standards such as 4G (4 th Generation) and 5G (5 th Generation). In addition, the communication device 60 may include, for example, a satellite communication module. In addition, the communication device 60 may include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. Also, when there are a plurality of connectable communication lines NW, the communication device 60 may include a plurality of communication devices according to the type of the communication line NW.

[0072] For example, the communication device 60 communicates with external devices such as the remote operation support device 200 within the work site through a local communication line constructed at the work site. The local communication line is, for example, a mobile communication line based on local 5G (so-called local 5G) constructed at the work site or a local network based on WiFi6.

[0073] In addition, the communication device 60 may communicate with the remote operation support device 200 outside the work site, etc. through a wide-area communication line including the work site, that is, a wide-area network.

[0074] <Control system> As shown in FIG. 4, the control system of the excavator 100 includes a controller 30. Further, the control system of the excavator 100 according to the present embodiment includes an operation pressure sensor 29, an imaging device 40, and sensors S1 to S9.

[0075] The controller 30 performs various controls related to the excavator 100.

[0076] The functions of the controller 30 may be realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 3, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D connected by a bus BS1.

[0077] The auxiliary storage device 30A is a non-volatile storage means, stores the installed program, and stores necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, or the like.

[0078] When there is an instruction to start a program, for example, the memory device 30B loads the program in the auxiliary storage device 30A so that the CPU 30C can read it. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0079] The CPU 30C executes, for example, the program loaded in the memory device 30B and realizes various functions of the controller 30 according to the instructions of the program.

[0080] The interface device 30D functions as a communication interface for connecting to the internal communication line of the excavator 100, for example. The interface device 30D may include a plurality of different types of communication interfaces according to the type of the communication line to be connected.

[0081] Also, the interface device 30D functions as an external interface for reading data from a recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool connected by a cable detachable from a connector installed inside the cab 10. Also, the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thereby, a program for realizing various functions of the controller 30 can be provided by a portable recording medium, for example, and installed in the auxiliary storage device 30A of the controller 30. Also, the program may be downloaded from another computer outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.

[0082] Note that a part of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the excavator 100.

[0083] The operation pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operation device 26, that is, the pilot pressure corresponding to the operation state of each driven element (hydraulic actuator) in the operation device 26. The detection signal of the pilot pressure corresponding to the operation state regarding each driven element (hydraulic actuator HA) in the operation device 26 by the operation pressure sensor 29 is taken into the controller 30.

[0084] In addition, when the operation device 26 is electric or when the operation device 26 is omitted, the operation pressure sensor 29 is omitted. This is because the controller 30 can grasp the operation state of each driven element through the operation device 26 based on the operation signal taken in from the operation device 26.

[0085] The imaging device 40 images the situation around the excavator 100.

[0086] The imaging device 40 is, for example, a monocular camera. Also, the imaging device 40 may be a three-dimensional camera (3D camera) capable of acquiring three-dimensional information including not only two-dimensional image information but also information regarding the distance to the objects shown in the image and the depth of the image, such as a stereo camera, a ToF (Time of Flight) camera, or a depth camera.

[0087] For example, as shown in FIG. 2, the imaging device 40 includes cameras 40F, 40B, 40L, and 40R. The camera 40F images the front of the upper swing body 3. The camera 40B images the rear of the upper swing body 3. The camera 40L images the left side of the upper swing body 3. The camera 40R images the right side of the upper swing body 3. Thereby, the imaging device 40 can image the situation in the entire circumference centered on the excavator 100, that is, in the range covering 360 degrees in the angular direction, in the top view of the excavator 100. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X".

[0088] The output data of the imaging device 40 (camera 40X) is captured by the controller 30 through a one-to-one communication line or an in-vehicle network. Thereby, for example, the controller 30 can grasp the situation around the excavator 100 based on the output data of the camera 40X.

[0089] Note that some or all of the cameras 40B, 40L, and 40R may be omitted. Further, the excavator 100 may be provided with a distance measuring sensor (also referred to as a "distance sensor") capable of acquiring information representing the distance to an object around the excavator 100 instead of or in addition to the imaging device 40. The distance measuring sensor is, for example, LIDAR (Light Detecting and Ranging), a millimeter-wave radar, an ultrasonic sensor, or the like.

[0090] The sensor S1 is attached to the boom 4 and measures the attitude state of the boom 4. The sensor S1 outputs measurement data representing the attitude state of the boom 4. The attitude state of the boom 4 is, for example, the attitude angle (hereinafter, "boom angle") around the rotation axis of the base end corresponding to the connecting portion of the boom 4 with the upper swing body 3. The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), or the like. The same may apply to the sensors S2 to S4 hereinafter. Further, the sensor S1 may include a cylinder sensor that detects the extension and retraction position of the boom cylinder 7. The same may apply to the sensors S2 and S3 hereinafter. The output of the sensor S1 (measurement data representing the attitude state of the boom 4) is captured by the controller 30. Thereby, the controller 30 can grasp the attitude state of the boom 4.

[0091] The sensor S2 is attached to the arm 5 and measures the posture state of the arm 5. The sensor S2 outputs measurement data representing the posture state of the arm 5. The posture state of the arm 5 is, for example, the posture angle (hereinafter, "arm angle") around the rotation axis of the base end corresponding to the connection part of the arm 5 and the boom 4. The output of the sensor S2 (measurement data representing the posture state of the arm 5) is taken into the controller 30. Thereby, the controller 30 can grasp the posture state of the arm 5.

[0092] The sensor S3 is attached to the bucket 6 and measures the posture state of the bucket 6. The sensor S3 outputs measurement data representing the posture state of the bucket 6. The posture state of the bucket 6 is, for example, the posture angle (hereinafter, "arm angle") around the rotation axis of the base end corresponding to the connection part of the bucket 6 and the arm 5. The output of the sensor S3 (measurement data representing the posture state of the bucket 6) is taken into the controller 30. Thereby, the controller 30 can grasp the posture state of the bucket 6.

[0093] The sensor S4 measures the posture state of the body of the excavator 100 (for example, the upper swing body 3). The sensor S4 outputs measurement data representing the posture state of the body of the excavator 100. The posture state of the body of the excavator 100 is, for example, the inclination state of the body with respect to a predetermined reference plane (for example, the horizontal plane). For example, the sensor S4 is attached to the upper swing body 3 and measures the inclination angles (hereinafter, "front - rear inclination angle" and "left - right inclination angle") around two axes in the front - rear direction and the left - right direction of the excavator 100. The output of the sensor S4 (measurement data representing the posture state of the body of the excavator 100) is taken into the controller 30. Thereby, the controller 30 can grasp the posture state (inclination state) of the body (upper swing body 3).

[0094] Sensor S5 is attached to the upper slewing body 3 and measures the slewing state of the upper slewing body 3. Sensor S5 outputs measurement data representing the slewing state of the upper slewing body 3. Sensor S5 measures, for example, the slewing angular velocity and slewing angle of the upper slewing body 3. Sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of sensor S5 (measurement data representing the slewing state of the upper slewing body 3) is taken into the controller 30. Thereby, the controller 30 can grasp the slewing state such as the slewing angle of the upper slewing body 3.

[0095] Based on the outputs of sensors S1 to S5, the controller 30 can grasp (estimate) the position of the tip (bucket 6) of the attachment AT.

[0096] In addition, when the sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the slewing state (for example, the slewing angular velocity) of the upper slewing body 3 may be detected based on the detection signal of the sensor S4. In this case, the sensor S5 may be omitted.

[0097] Sensor S6 measures the position of the excavator 100. Sensor S6 may measure the position in world (global) coordinates or may measure the position in local coordinates at the work site. In the former case, sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, sensor S6 is a transceiver that can communicate with the device serving as the reference for the position at the work site and output a signal corresponding to the position relative to the reference. The output of sensor S6 is taken into the controller 30.

[0098] Sensor S7 measures the pressure (cylinder pressure) in the oil chamber of the boom cylinder 7. Sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of the boom cylinder 7 and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side. The output of sensor S7 (that is, the measurement data of the cylinder pressure of the boom cylinder 7) is taken into the controller 30.

[0099] Sensor S8 measures the pressure (cylinder pressure) in the oil chamber of arm cylinder 8. Sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of arm cylinder 8 and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side of arm cylinder 8. The output of sensor S8 (i.e., the measurement data of the cylinder pressure of arm cylinder 8) is taken into controller 30.

[0100] Sensor S9 measures the pressure (cylinder pressure) in the oil chamber of bucket cylinder 9. Sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber on the rod side of bucket cylinder 9 and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber on the bottom side of bucket cylinder 9. The output of sensor S9 (i.e., the measurement data of the cylinder pressure of bucket cylinder 9) is taken into controller 30.

[0101] Based on the outputs of sensors S7 to S9, controller 30 can grasp the load state acting on attachment AT. The load acting on attachment AT includes, for example, the reaction force acting from the earth and sand on the ground of the work target on bucket 6 and the weight of the earth and sand contained in bucket 6.

[0102] In addition, some or all of sensors S1 to S9 may be omitted according to necessity. Excavator 100 may be equipped with other sensors capable of grasping the state of excavator 100. For example, excavator 100 may be provided with an orientation sensor capable of detecting its own orientation. The orientation sensor is, for example, an electronic compass including a geomagnetic sensor.

[0103] [Configuration of Remote Operation Support Device] Next, with reference to FIG. 5, the configuration of remote operation support device 200 will be described.

[0104] FIG. 5 is a block diagram showing an example of the configuration of remote operation support device 200.

[0105] The functions of the remote operation support device 200 are realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 5, the remote operation support device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, a display device 208, and an audio output device 209. These are connected by a bus BS2.

[0106] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. Thereby, the remote operation support device 200 can read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, or install a program for realizing various functions.

[0107] Note that the remote operation support device 200 may acquire various data and programs used in processing from an external device through the communication interface 206.

[0108] The auxiliary storage device 202 stores installed various programs and also stores files, data, etc. necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, etc.

[0109] When there is an instruction to start a program, the memory device 203 reads the program from the auxiliary storage device 202 and stores it. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) and an SRAM.

[0110] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the remote operation support device 200 according to the programs.

[0111] The high-speed arithmetic unit 205 operates in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed. The high-speed arithmetic unit 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

[0112] Note that the high-speed arithmetic unit 205 may be omitted depending on the speed of the necessary arithmetic processing.

[0113] The communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the remote operation support device 200 can communicate with an external device such as the excavator 100 through the communication interface 206. Further, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device, etc.

[0114] The input device 207 receives various inputs from the user. The input device 207 includes an operation device for remote operation for remotely operating the excavator 100.

[0115] The input device 207 includes, for example, an input device (mechanical input device) in a form that receives a mechanical operation input from the user. The operation device for remote operation may be a mechanical input device. The mechanical input device includes, for example, buttons, toggles, levers, keyboards, mice, touch panels mounted on the display device 208, touch pads provided separately from the display device 208, etc.

[0116] Further, the input device 207 may include a voice input device capable of receiving a voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0117] Further, the input device 207 may include a gesture input device capable of receiving a gesture input from the user. The gesture input device includes, for example, a camera capable of imaging the state of the user's gesture.

[0118] Also, the input device 207 may include a biometric input device capable of receiving a biometric input from the user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about the user's fingerprint or iris.

[0119] The display device 208 displays an information screen or an operation screen for the user of the remote operation support device 200. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0120] The sound output device 209 conveys various information to the user of the remote operation support device 200 by sound. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0121] [First Example of Functional Configuration for Excavator Operation Support] Next, in addition to FIGS. 1 to 5, with reference to FIGS. 6 to 9, a first example of the functional configuration for the operation support of the excavator 100 will be described.

[0122] FIG. 6 is a functional block diagram showing a first example of the functional configuration for the operation support of the excavator 100. FIG. 7 is a diagram showing an example of the surrounding environment of the excavator 100. FIG. 8 is a diagram showing an example of an example given to the language model LM1. FIG. 9 is a diagram showing an example of a combination of a prompt and an output of the language model LM1.

[0123] As shown in FIG. 6, the controller 30 includes, as functional units, an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, and an operation control unit 306. These functions are realized, for example, by a program installed in the auxiliary storage device 30A being loaded into the memory device 30B and executed by the CPU 30C. Further, an external language model LM1 is provided outside the excavator 100.

[0124] The language model LM1 is a large language model (LLM). The language model LM1 is implemented in an external device (for example, a server device) that is communicably connected to the excavator 100 via the communication device 60. The language model LM1 is, for example, GPT-4.

[0125] The instruction acquisition unit 301 acquires an instruction regarding the operation of the excavator 100 (hereinafter simply referred to as "instruction") input in natural language from the operator.

[0126] The input of an instruction from the operator is received through the input device 52 installed in the cab 10 in the case of the operator in the cab 10, and through the input device 207 of the remote operation support device 200 in the case of a remote operator.

[0127] An instruction in natural language is, for example, an instruction input in natural language by the operator's voice. In this case, the instruction acquisition unit 301 can acquire data of text (sentence) corresponding to the instruction in natural language by applying known speech recognition technology based on the speech input data received by the input device 52 or the input device 207.

[0128] Further, an instruction in natural language may be an instruction input in text by the operator using an input device 52 or an input device 207 capable of character input such as a keyboard or a touch panel. In this case, the instruction acquisition unit 301 can acquire the data of text (sentence) received by the input device 52 or the input device 207 as an instruction in natural language.

[0129] Based on the outputs of the imaging device 40 and the distance measuring sensor, the object detection unit 302 detects the objects to be monitored around the excavator 100. For example, the object detection unit 302 detects the objects to be monitored from the captured image of the imaging device 40 by arbitrarily applying known image processing techniques such as semantic segmentation and machine learning.

[0130] The objects to be monitored include, for example, people such as operators. Also, the objects to be monitored may include other obstacles around the excavator 100. Other obstacles include, for example, specific moving objects at the work site of the excavator 100 such as other work machines and work vehicles. Also, other obstacles may include specific stationary objects at the work site of the excavator 100 such as utility poles, fences, and triangular cones (also referred to as color cones (registered trademark)). Also, other obstacles may include specific terrain shapes at the work site of the excavator 100 such as ditches, holes, and mounds of earth and sand.

[0131] The language conversion unit 303 converts the environment around the excavator 100 into natural language. Also, the language conversion unit 303 may convert the construction drawings of the work target of the excavator 100 into natural language.

[0132] The language conversion unit 303, for example, converts the arrangement of the objects to be monitored detected by the object detection unit 302 around the excavator 100 into natural language. Specifically, the language conversion unit 303 may convert by fitting the type and position of the objects to be monitored detected by the object detection unit 302 to a template of text representing the arrangement of the objects to be monitored around the excavator 100 defined in advance. The same applies to the conversion of the arrangement of the objects in the construction drawings described later.

[0133] The template of the text representing the arrangement of the objects to be monitored is defined in a form such as "There is yyy at xxx." or "There is yyy in xxx." with "xxx" being the position information of the object to be monitored and "yyy" being the type or name of the object to be monitored.

[0134] For example, as shown in FIG. 7, a person P, triangular cones CN1 and CN2, and a mound of earth and sand PL are detected by the object detection unit 302 in front of the excavator 100.

[0135] In this example, the area around the excavator 100 is divided into a front area RF, a rear area RB, a left area RL, and a right area with reference to the excavator 100. Further, the front area RF is divided into a left-front adjacent area RF1, a right-front adjacent area RF2, a left-front and farther area RF3 than the area RF1, and a right-front and farther area RF4 than the area RF2 with reference to the excavator 100. Hereinafter, the areas RF1, RF2, RF3, and RF4 will be verbalized as "left-front adjacent area", "right-front adjacent area", "left-front far area", and "right-front far area".

[0136] In this case, since the person P is included in the area RF1, the verbalization unit 303 verbalizes the presence of the person P as "There is a person in the left-front adjacent area". Also, since the two triangular cones CN1 and CN2 are included in the area RF2, the verbalization unit 303 verbalizes the presence of the triangular cones CN1 and CN2 as "There are two color cones in the right-front adjacent area". Further, since the mound of earth and sand PL is included in the area RF3, the verbalization unit 303 verbalizes the presence of the mound of earth and sand PL as "There is a mound of earth and sand in the left-front far area".

[0137] Also, the verbalization unit 303 may verbalize the presence of the person P, the triangular cones CN1 and CN2, and the mound of earth and sand PL using the distance and coordinates with reference to the excavator 100, similar to the case of the objects described in the construction drawings described later.

[0138] Also, the verbalization unit 303 verbalizes, for example, the arrangement of the objects described in the construction drawings in natural language with reference to the position of the excavator 100.

[0139] For example, for the buried piping described in the construction drawings, the verbalization unit 303 verbalizes it as "There is piping 1 m underground, 5 m ahead (in meters)". Also, for the utility pole described in the construction drawings, the verbalization unit 303 verbalizes it as "There is a utility pole 5 m ahead and 2 m to the right". Further, the verbalization unit 303 may verbalize the position of the object described in the construction drawings using the coordinate system with the excavator 100 as a reference or fixed to the work site.

[0140] Based on the natural language instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303, the prompt generation unit 304 generates a prompt to be input to the language model LM1. Specifically, the prompt generation unit 304 generates a prompt for outputting the control information of the excavator 100 corresponding to the instruction acquired by the instruction acquisition unit 301 on the premise of the information on the surrounding environment of the excavator 100 and the construction drawings verbalized by the verbalization unit 303.

[0141] For example, the prompt generation unit 304 generates a plurality of example problems and pre - assigns them to the language model LM1 through the call unit 305. An example problem is defined by a combination of the information on the surrounding environment of the excavator 100 and the construction drawings as pre - conditions (i.e., constraint conditions), the instruction of the example problem, and the correct answer to be output. Thereby, the language model LM1 can understand (learn) the output format for the instruction of the prompt.

[0142] For example, as shown in FIG. 8, a plurality of examples represented by a combination of the surrounding environment of the excavator 100, the instructions of the user (i.e., the operator), and the control information (control commands) of the excavator 100 are given. As a result, as shown in FIG. 9, for the prompt generated by the prompt generation unit 304, that is, for the instruction from the operator based on the surrounding environment of the excavator 100, the language model LM1 can be made to output a control command corresponding to the operation of the excavator 100. Further, when the instruction cannot be understood, by instructing to ask again, a situation where an inappropriate operation is executed as the excavator 100 can be suppressed. The number of examples given is arbitrary, but it is desirable that it is larger than the number of types of control commands corresponding to the operations to be executed by the excavator 100. This is because the language model LM1 can understand all the control commands.

[0143] The calling unit 305 calls the language model LM1, for example, through a predetermined API (Application Programming Interface), inputs the prompt generated by the prompt generation unit 304 to the language model LM1, and obtains its output (answer).

[0144] The operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1 obtained by the calling unit 305. Specifically, the operation control unit 306 drives the hydraulic actuator HA of the excavator 100 according to the control command output from the language model LM1, and outputs a control command for controlling the operation of the excavator 100 to the hydraulic control valve 31.

[0145] In this way, in this example, the controller 30 controls the operation of the excavator 100 based on the result of having the language model LM1 interpret the natural language instruction of the operator and the surrounding environment of the excavator 100 verbalized in natural language. Thereby, the controller 30 can control the operation of the excavator 100 in accordance with the natural language instruction of the operator according to the surrounding environment of the excavator 100.

[0146] Part or all of the instruction acquisition unit 301, object detection unit 302, language conversion unit 303, prompt generation unit 304, call unit 305, and operation control unit 306 may be transferred to an information processing device outside the excavator 100. For example, when the excavator 100 is remotely operated, the instruction acquisition unit 301, object detection unit 302, language conversion unit 303, prompt generation unit 304, call unit 305, and operation control unit 306 may be transferred to the remote operation support device 200.

[0147] [First Example of Processing Related to Excavator Operation Support] Next, with reference to FIG. 10, a first example of processing related to the operation support of the excavator 100 will be described.

[0148] FIG. 10 is a flowchart schematically showing a first example of processing related to the operation support of the excavator 100. In this example, the functional configuration related to the operation support of the excavator 100 in FIG. 6 is assumed.

[0149] The flowchart in FIG. 10 is executed, for example, when an operator's instruction in natural language is received through the input device 52 or the input device 207.

[0150] As shown in FIG. 10, in step S102, the instruction acquisition unit 301 acquires the text of the instruction.

[0151] When the processing in step S102 is completed, the controller 30 proceeds to step S104.

[0152] In step S104, the object detection unit 302 detects an object to be monitored around the excavator 100 based on the output of the imaging device 40 or the distance measuring sensor.

[0153] When the processing in step S104 is completed, the controller 30 proceeds to step S106.

[0154] In step S106, the language conversion unit 303 converts the environment around the excavator 100 and the construction drawings into natural language.

[0155] When the process of step S106 is completed, the controller 30 proceeds to step S108.

[0156] In step S108, the prompt generation unit 304 generates a prompt to be input to the language model LM1 based on the results of the processes in steps S104 and S106.

[0157] When the process of step S108 is completed, the controller 30 proceeds to step S110.

[0158] In step S110, the calling unit 305 calls the language model LM1 and inputs the prompt generated in step S108.

[0159] When step S110 is completed and the output from the language model LM1 is obtained, the controller 30 proceeds to step S112.

[0160] In step S112, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1, that is, the control command.

[0161] When the process of step S112 is completed, the controller 30 ends the processing of the current flowchart.

[0162] [Second Example of Functional Configuration for Excavator Operation Support] Next, in addition to FIGS. 1 to 5, with reference to FIG. 11, a second example of the functional configuration for the operation support of the excavator 100 will be described.

[0163] Hereinafter, the same or corresponding components as those in the first example of the above-described functional configuration are denoted by the same reference numerals, and the description will be centered on the parts different from the first example of the above-described functional configuration.

[0164] FIG. 11 is a functional block diagram showing a second example of the functional configuration for the operation support of the excavator 100.

[0165] As shown in FIG. 11, as a functional unit, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, and an operation control unit 306, similar to the first example described above. Further, different from the first example described above, the controller 30 includes an urgency determination unit 307. These functions are realized, for example, by a program installed in the auxiliary storage device 30A being loaded into the memory device 30B and executed by the CPU 30C.

[0166] The urgency determination unit 307 determines the urgency of the instruction acquired by the instruction acquisition unit 301. The urgency of an instruction represents the degree to which the operation of the excavator 100 corresponding to the instruction should be carried out urgently.

[0167] For example, the urgency determination unit 307 determines the level of urgency based on the presence or absence of a predetermined character string (hereinafter, for convenience, referred to as an "urgency word") included in the text acquired by the instruction acquisition unit 301. Urgency words are, for example, "dangerous", "hazardous", "stop", etc. In this case, when the urgency determination unit 307 determines that the text acquired by the instruction acquisition unit 301 includes an urgency word, it determines that the urgency is high, and when it does not include an urgency word, it determines that the urgency is low.

[0168] When the urgency determination unit 307 determines that the urgency is high, it outputs a notification indicating that the urgency is high to the operation control unit 306 together with the text of the instruction acquired by the instruction acquisition unit 301.

[0169] When the operation control unit 306 receives a notification indicating that the urgency is high from the urgency determination unit 307, it causes the excavator 100 to execute a predetermined operation (hereinafter, for convenience, referred to as an "emergency operation") corresponding to the instruction for which the urgency is determined to be high. The emergency operation is, for example, an operation to suddenly stop the excavator 100 (hereinafter, referred to as an "emergency stop operation"). Further, the emergency operation may include an operation to avoid danger by daring to continue the operation of the excavator 100 in addition to the emergency stop operation of the excavator 100 (hereinafter, referred to as a "danger avoidance operation"). In this case, the operation control unit 306 determines whether to perform the emergency stop operation or the danger avoidance operation based on the words included in the text of the instruction.

[0170] On the other hand, when the emergency degree determination unit 307 determines that the emergency degree is low, it outputs a notification indicating that the emergency degree is low to the prompt generation unit 304 together with the text of the instruction acquired by the instruction acquisition unit 301.

[0171] Based on the natural language instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303, which are input through the emergency degree determination unit 307, the prompt generation unit 304 generates a prompt to be input into the language model LM1.

[0172] Similar to the first example described above, the call unit 305 calls the language model LM1, inputs the prompt generated by the prompt generation unit 304 into the language model LM1, and obtains its output (answer).

[0173] When the emergency degree determination unit 307 determines that the emergency degree is low, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1 obtained by the call unit 305, similar to the first example described above.

[0174] As described above, in this example, when the emergency degree of the operator's instruction is relatively high, the controller 30 causes the excavator 100 to directly execute the emergency operation corresponding to the instruction without applying the language model LM1. As a result, when the emergency degree of the operator's instruction is high, the controller 30 can avoid the delay caused by the use of the language model LM1 and cause the excavator 100 to perform the emergency operation quickly.

[0175] Note that part or all of the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the call unit 305, the operation control unit 306, and the emergency degree determination unit 307 may be transferred to an information processing device outside the excavator 100. For example, when the excavator 100 is remotely operated, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the call unit 305, the operation control unit 306, and the emergency degree determination unit 307 may be transferred to the remote operation support device 200.

[0176] [Second Example of Processing Related to Operation Support of Excavator] Next, with reference to FIG. 12, a second example of processing related to operation support of the excavator 100 will be described.

[0177] FIG. 12 is a flowchart schematically showing a first example of processing related to operation support of the excavator 100. In this example, the functional configuration related to operation support of the excavator 100 in FIG. 11 is assumed.

[0178] The flowchart in FIG. 12 is executed, for example, when an operator's instruction in natural language is received through the input device 52 or the input device 207.

[0179] As shown in FIG. 12, since the processing in step S202 is the same as the processing in step S102 in FIG. 10, the description thereof is omitted.

[0180] When the processing in step S202 is completed, the controller 30 proceeds to step S204.

[0181] In step S204, the urgency determination unit 307 determines whether the urgency of the instruction is relatively high or low based on the text of the instruction acquired in step S202.

[0182] When the processing in step S204 is completed, the controller 30 proceeds to step S206.

[0183] In step S206, the controller 30 determines whether the urgency of the instruction acquired in step S202 is relatively high or not based on the determination result in step S204. When the urgency of the instruction is relatively low, the controller 30 proceeds to step S208, and when the urgency is relatively high, the controller 30 proceeds to step S218.

[0184] The processing in steps S208, S210, S212, S214, and S216 is the same as the processing in steps S104, S106, S108, S110, and S112 in FIG. 10 described above, and thus the description thereof is omitted.

[0185] On the other hand, in step S218, the operation control unit 306 causes the excavator 100 to execute an emergency operation corresponding to the instruction acquired in step S202.

[0186] When the processing of step S216 or step S218 is completed, the controller 30 ends the processing of the current flowchart.

[0187] [Third Example of Functional Configuration for Excavator Operation Support] Next, in addition to FIGS. 1 to 5, a third example of the functional configuration for the operation support of the excavator 100 will be described with reference to FIG. 13.

[0188] Hereinafter, the same or corresponding components as those in the first and second examples of the above-described functional configuration are denoted by the same reference numerals, and the description will be centered on the parts different from the first and second examples of the above-described functional configuration.

[0189] FIG. 13 is a functional block diagram showing a third example of the functional configuration for the operation support of the excavator 100.

[0190] As shown in FIG. 13, as a functional unit, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a call unit 305, an operation control unit 306, and an urgency determination unit 307, similar to the second example described above. Further, different from the first and second examples described above, the controller 30 includes a language model LM2 and a language model selection unit 308. These functions are realized, for example, by a program installed in the auxiliary storage device 30A being loaded into the memory device 30B and executed by the CPU 30C.

[0191] The language model LM2 is a relatively smaller-scale language model than the language model LM1.

[0192] Note that the language model LM1 may be implemented in an information processing device different from the controller 30 and mounted on the excavator 100.

[0193] When the emergency degree determination unit 307 determines that the emergency degree is low, it outputs a notification indicating that the emergency degree is low to the language model selection unit 308 together with the text of the instruction acquired by the instruction acquisition unit 301.

[0194] When the language model selection unit 308 determines that the emergency degree of the instruction from the operator is low by the emergency degree determination unit 307, it selects whether to input the text of the instruction acquired by the instruction acquisition unit 301 into either of the language models LM1 and LM2. Specifically, when the content of the instruction acquired by the instruction acquisition unit 301 is relatively easy to understand, the language model selection unit 308 selects the language model LM2, and when the content of the instruction is relatively complex or difficult to understand, the language model selection unit 308 selects the language model LM1.

[0195] For example, the language model selection unit 308 selects either one of the language models LM1 and LM2 based on the length of the text of the instruction acquired by the instruction acquisition unit 301. The length of the instruction text is defined based on, for example, the number of characters or words in the text. Specifically, when the length of the instruction text is relatively long with respect to a predetermined standard, the language model selection unit 308 selects the language model LM1, and in other cases, the language model selection unit 308 selects the language model LM2. That the length of the text is relatively long with respect to a predetermined standard may mean that the length of the text is equal to or greater than the predetermined standard, or that the length of the text is longer than the predetermined standard.

[0196] In addition, the language model selection unit 308 selects either one of the language models LM1 and LM2 based on the rarity (scarcity) of the words included in the text of the instruction acquired by the instruction acquisition unit 301. Specifically, when the number of words with high rarity (hereinafter, "rare words") included in all the words included in the text of the instruction is relatively large with respect to a predetermined standard, the language model selection unit 308 selects the language model LM1, and in other cases, the language model selection unit 308 selects the language model LM2. Rare words are defined in advance. For example, the technique of text matching is applied, and the language model selection unit 308 determines the presence or absence and the number of rare words included in the text of the instruction.

[0197] Further, the language model selection unit 308 may select either one of the language models LM1 and LM2 in consideration of both the length of the text of the instruction and the rarity of the words included in the text. In this case, the language model selection unit 308 may select the language model LM1 when both a first condition indicating that the length of the text of the instruction is relatively long with respect to a predetermined standard and a second condition indicating that the number of rare words included in the text of the instruction is relatively large with respect to a predetermined standard are satisfied, or may select the language model LM1 when either one is satisfied.

[0198] The language model selection unit 308 outputs the text of the instruction acquired by the instruction acquisition unit 301 and information representing the selected one of the language models LM1 and LM2 to the prompt generation unit 304.

[0199] The prompt generation unit 304 generates a prompt to be input to the language model LM1 or the language model LM2. The prompt generation unit 304 includes prompt generation units 304A and 304B.

[0200] When the language model LM1 is selected by the language model selection unit 308, the prompt generation unit 304A generates a prompt to be input to the language model LM1 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0201] When the language model LM2 is selected by the language model selection unit 308, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303. The format of the prompt for the language model LM2 may be the same as or different from that of the language model LM1.

[0202] The prompt generation unit 304 outputs the prompt generated by the prompt generation unit 304A to the call unit 305 and outputs the prompt generated by the prompt generation unit 304B to the language model LM2.

[0203] The language model LM2 takes the prompt imported from the prompt generation unit 304B as input and outputs control information (control commands) for the excavator 100, similar to the language model LM1.

[0204] When the operation control unit 306 receives a notification from the urgency determination unit 307 that the urgency is high, it causes the excavator 100 to execute an emergency operation corresponding to the instruction, similar to the second example described above.

[0205] Also, when the output of the language model LM1 is input from the calling unit 305 to the operation control unit 306, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1, similar to the first example described above.

[0206] Also, when the output of the language model LM2 is input to the operation control unit 306, the operation control unit 306 controls the operation of the excavator 100 based on the control information (control commands) corresponding to the output of the language model LM2.

[0207] Thus, in this example, the controller 30 can selectively use the language models LM1 and LM2 according to the content of the operator's instruction. Therefore, for example, when the operator's instruction is relatively easy, the language model LM2 can be used to suppress the communication cost compared to the case of using the language model LM1. Also, when a cost is incurred for using the language model LM1, the usage cost of the language model LM1 can be suppressed. On the other hand, when the operator's instruction is relatively difficult to understand or complex, the language model LM1 can be used to ensure the accuracy of the control commands as the interpretation result. Therefore, in this example, the controller 30 can achieve both suppression of variable costs and more accurate operation of the excavator 100 in response to the operator's instruction in natural language.

[0208] Further, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the operation control unit 306, the urgency determination unit 307, the language model selection unit 308, and part or all of the language model LM2 may be transferred to an information processing device outside the excavator 100. For example, when the excavator 100 is remotely operated, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the operation control unit 306, the urgency determination unit 307, the language model selection unit 308, and the language model LM2 may be transferred to the remote operation support device 200.

[0209] [Third Example of Processing Related to Excavator Operation Support] Next, with reference to FIG. 14, a third example of the processing related to the operation support of the excavator 100 will be described.

[0210] FIG. 14 is a flowchart schematically showing a third example of the processing related to the operation support of the excavator 100. In this example, the functional configuration related to the operation support of the excavator 100 in FIG. 13 is assumed.

[0211] The flowchart in FIG. 14 is executed, for example, when an operator's instruction in natural language is received through the input device 52 or the input device 207.

[0212] As shown in FIG. 14, the processes in steps S302, S304, and S306 are the same as the processes in steps S202, S204, and S206 in FIG. 12 described above, and thus the description thereof will be omitted.

[0213] In step S306, when the urgency of the instruction is relatively low, the controller 30 proceeds to step S308, and when the urgency of the instruction is relatively high, the controller 30 proceeds to step S330.

[0214] In step S308, the language model selection unit 308 selects one of the language models LM1 and LM2 based on the content of the text of the instruction acquired in the process of step S302.

[0215] When the process of step S308 is completed, the controller 30 proceeds to step S310.

[0216] In step S310, the controller 30 determines whether a relatively large-scale language model LM1 has been selected based on the determination result of step S308. If the language model LM1 has been selected, the process proceeds to step S312; if the language model LM2 has been selected, the process proceeds to step S320.

[0217] Since the processes of steps S312 and S314 are the same as the processes of steps S104 and S106 in FIG. 10 and steps S208 and S210 in FIG. 12 described above, the description thereof is omitted.

[0218] When the process of step S314 is completed, the controller 30 proceeds to step S316.

[0219] In step S316, the prompt generation unit 304A generates a prompt to be input to the language model LM1 based on the results of the processes of steps S312 and S314.

[0220] When the process of step S316 is completed, the controller 30 proceeds to step S318.

[0221] Since the process of step S318 is the same as step S110 in FIG. 10 and step S214 in FIG. 12 described above, the description thereof is omitted.

[0222] When the process of step S318 is completed and the output from the language model LM1 is obtained, the controller 30 proceeds to step S328.

[0223] On the other hand, since the processes of steps S320 and S322 are the same as the processes of steps S104 and S106 in FIG. 10 and steps S208 and S210 in FIG. 12 described above, the description thereof is omitted.

[0224] When the process of step S322 is completed, the controller 30 proceeds to step S324.

[0225] In step S324, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the results of the processes in steps S320 and S322.

[0226] When the process in step S324 is completed, the controller 30 proceeds to step S326.

[0227] In step S326, the language model LM2 incorporated in the controller 30 performs an operation using the prompt generated in step S324 as an input, and outputs control information (control commands) for the excavator 100.

[0228] When the process in step S326 is completed, the controller 30 proceeds to step S328.

[0229] In step S328, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1 or the output of the language model LM2.

[0230] When the process in step S328 is completed, the controller 30 ends the process of this flowchart.

[0231] On the other hand, since the process in step S330 is the same as the process in step S218 in FIG. 12 described above, the description thereof is omitted.

[0232] When the process in step S330 is completed, the controller 30 ends the process of this flowchart.

[0233] [Fourth Example of Functional Configuration for Excavator Operation Support] Next, in addition to FIGS. 1 to 5, a fourth example of the functional configuration for the operation support of the excavator 100 will be described with reference to FIG. 15.

[0234] Hereinafter, the same or corresponding components as those in the first to third examples of the above functional configuration are denoted by the same reference numerals, and the description will be centered on the parts different from the first to third examples of the above functional configuration.

[0235] FIG. 15 is a functional block diagram showing a fourth example of the functional configuration related to the operation support of the excavator 100.

[0236] As shown in FIG. 15, it is different from the above-described third example in that an instruction classifier 307A is provided in the controller 30 instead of the urgency determination unit 307 and the language model selection unit 308.

[0237] The instruction classifier 307A classifies the instructions acquired by the instruction acquisition unit 301 into a first instruction, a second instruction, and a third instruction that are different in type from each other.

[0238] The first instruction is an instruction with a relatively high urgency. The second instruction is an instruction with a relatively low urgency and is relatively difficult or complex. The third instruction is an instruction with a relatively low urgency and is relatively easy to understand.

[0239] That is, the instruction classifier 307A realizes the functions of the above-described urgency determination unit 307 and language model selection unit 308 as a single three-class classifier.

[0240] The instruction classifier 307A is, for example, a learned model obtained by supervised learning. The learned model is configured mainly with, for example, a neural network, and is obtained by optimizing the base model using the algorithm of the error backpropagation method based on the error between the inference result and the teacher data. Also, the learned model may be configured mainly with a support vector machine.

[0241] When the instruction classifier 307A classifies the instruction acquired by the instruction acquisition unit 301 as the first instruction, it outputs the text of the instruction acquired by the instruction acquisition unit 301 and a notification indicating that it is the first instruction to the operation control unit 306.

[0242] When the operation control unit 306 receives a notification from the instruction classifier 307A that the instruction acquired by the instruction acquisition unit 301 is a first instruction, the operation control unit 306 causes the excavator 100 to execute an emergency operation corresponding to the instruction.

[0243] Further, when the instruction classifier 307A classifies the instruction acquired by the instruction acquisition unit 301 as a second instruction or a third instruction, the instruction classifier 307A outputs the text of the instruction acquired by the instruction acquisition unit 301 and a notification indicating that it is a second instruction or a third instruction to the prompt generation unit 304.

[0244] When the prompt generation unit 304A receives a notification from the instruction classifier 307A that it is a second instruction, the prompt generation unit 304A generates a prompt to be input to the language model LM1 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0245] When the prompt generation unit 304B receives a notification from the instruction classifier 307A that it is a third instruction, the prompt generation unit 304B generates a prompt to be input to the language model LM2 based on the instruction acquired by the instruction acquisition unit 301 and the information verbalized by the verbalization unit 303.

[0246] Thus, in this example, the controller 30 can integrate the functions of the urgency determination unit 307 and the language model selection unit 308 into the instruction classifier 307A corresponding to a three-class classifier.

[0247] Note that part or all of the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the operation control unit 306, the instruction classifier 307A, and the language model LM2 may be transferred to an information processing device external to the excavator 100. For example, when the excavator 100 is remotely operated, the instruction acquisition unit 301, the object detection unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the operation control unit 306, the instruction classifier 307A, and the language model LM2 may be transferred to the remote operation support device 200.

[0248] [Fourth Example of Processing for Excavator Operation Support] Next, referring to FIG. 16, a third example of the process related to the operation support of the excavator 100 will be described.

[0249] FIG. 16 is a flowchart schematically showing a fourth example of the process related to the operation support of the excavator 100. In this example, the functional configuration related to the operation support of the excavator 100 in FIG. 15 is assumed.

[0250] The flowchart in FIG. 16 is executed, for example, when an operator's instruction in natural language is received through the input device 52 or the input device 207.

[0251] As shown in FIG. 16, the process of step S402 is the same as the process of step S102 in FIG. 10 described above, etc., so the description is omitted.

[0252] When the process of step S402 is completed, the controller 30 proceeds to step S404.

[0253] In step S404, the instruction classifier 307A classifies the instruction acquired in step S402 into any one of a first instruction, a second instruction, and a third instruction.

[0254] When the process of step S404 is completed, the controller 30 proceeds to step S406.

[0255] In step S406, the controller 30 determines which of the first instruction, the second instruction, and the third instruction the classification result in step S404 is. When the classification result is the second instruction, the controller 30 proceeds to step S408; when it is the third instruction, the controller 30 proceeds to step S416; and when it is the first instruction, the controller 30 proceeds to step S426.

[0256] The processes of steps S408, S410, S412, and S414 are the same as the processes of steps S312, S314, S316, and S318 in FIG. 14 described above, so the description is omitted.

[0257] When the process of step S414 is completed, the controller 30 proceeds to step S424.

[0258] Also, since the processes of steps S416, S418, S420, and S422 are the same as the processes of steps S320, S322, S324, and S326 in FIG. 14 described above, the description thereof is omitted.

[0259] When the process of step S422 is completed, the controller 30 proceeds to step S424.

[0260] Since the process of step S424 is the same as the process of step S328 in FIG. 14 described above, the description thereof is omitted.

[0261] When the process of step S424 is completed, the controller 30 ends the process of this flowchart.

[0262] Since the process of step S426 is the same as the process of step S330 in FIG. 14 described above, the description thereof is omitted.

[0263] When the process of step S426 is completed, the controller 30 ends the process of this flowchart.

[0264] [Fifth Example of Functional Configuration for Excavator Operation Support] Next, in addition to FIGS. 1 to 5, with reference to FIG. 17, a fifth example of the functional configuration for the operation support of the excavator 100 will be described.

[0265] Hereinafter, the same or corresponding configurations as those in the first to fourth examples of the above-described functional configuration are denoted by the same reference numerals, and the description will be centered on the parts different from the first to fourth examples of the above-described functional configuration.

[0266] FIG. 17 is a functional block diagram showing a fifth example of the functional configuration for the operation support of the excavator 100.

[0267] As shown in FIG. 17, the controller 30 includes an instruction acquisition unit 301, an object detection unit 302, a verbalization unit 303, a prompt generation unit 304, a calling unit 305, and an operation control unit 306, similar to the first example described above. Further, different from the first example described above, the controller 30 includes a notification unit 309.

[0268] Based on the output of the language model LM1 taken in from the calling unit 305, that is, the control information (control command) of the excavator 100, the notification unit 309 notifies the operator in advance of the planned operation content of the excavator 100 through the display device as the output device 50 and the display device 208. Thereby, the operator can confirm whether the instructions in natural language by his own voice or text input are properly understood.

[0269] Further, the notification unit 309 may notify the operator of the information on the work procedure of the excavator 100 on the day together with the planned operation content of the excavator 100. Thereby, the operator can confirm whether the operation content of the excavator 100 planned by the instructions by his own voice or text input conforms to the work procedure of the excavator 100 on the day.

[0270] After the notification by the notification unit 309, when an operation permission input is received from the operator through the input device 52 or the input device 207, the operation control unit 306 controls the operation of the excavator 100 based on the output of the language model LM1.

[0271] On the other hand, after the notification by the notification unit 309, when an operation non - permission input is received from the operator through the input device 52 or the input device 207, the operation control unit 306 aborts the operation control of the excavator 100 based on the output of the language model LM1. The same applies when neither an operation permission input nor an operation non - permission input is received from the operator after the notification by the notification unit 309.

[0272] In this way, in this example, the controller 30 can realize the operation of the excavator 100 based on the natural language instruction from the operator under the confirmation and permission of the operation content of the excavator 100 by the operator in advance. Therefore, the controller 30 can suppress the inappropriate operation of the excavator 100 and improve the safety of the excavator 100.

[0273] [Fifth Example of Processing Related to Excavator Operation Support] Next, with reference to FIG. 18, a fifth example of the processing related to the operation support of the excavator 100 will be described.

[0274] The flowchart of FIG. 18 is executed, for example, when an instruction from the operator in natural language is received through the input device 52 or the input device 207.

[0275] As shown in FIG. 18, the processes in steps S502, S504, S506, S508, and S510 are the same as the processes in steps S102, S104, S106, S108, and S110 of FIG. 10 described above, so the description thereof will be omitted.

[0276] When the process in step S510 is completed, the controller 30 proceeds to step S512.

[0277] In step S512, the notification unit 309 gives a prior notification of the operation content of the excavator 100 to the operator.

[0278] When the process in step S512 is completed, the controller 30 proceeds to step S514.

[0279] In step S514, the controller 30 determines whether an input of operation permission has been received from the operator. If an input of operation permission is received from the operator within a predetermined time after the notification, the controller 30 proceeds to step S516; otherwise, the processing of this flowchart ends.

[0280] The process of step S516 is the same as the process of step S112 in FIG. 10 described above, so the description thereof is omitted.

[0281] When the process of step S516 is completed, the controller 30 ends the process of the current flowchart.

[0282] [Other Embodiments] Next, other embodiments will be described.

[0283] In the above-described embodiments, they may be combined as appropriate, or modified or changed.

[0284] For example, in the third example of the above-described functional configuration, the urgency determination unit 307 may be omitted. In this case, the language model selection unit 308 selects either one of the language models LM1 and LM2 regardless of the level of urgency of the instruction acquired by the instruction acquisition unit 301.

[0285] Also, in the second to fourth examples of the above-described functional configuration and examples of their modifications and changes, a function similar to the notification unit 309 of the fifth example of the above-described functional configuration may be added. However, when it is determined that the urgency of the instruction acquired by the instruction acquisition unit 301 is relatively high, or when the instruction is classified as the above-described first instruction, a prior notification of the operation content to the operator is not performed.

[0286] Also, the first to fifth examples of the functional configuration and process related to the above-described operation support and examples of their modifications and changes may be applied to other working machines other than the excavator 100. Other working machines include, for example, bulldozers, mobile cranes, and the like.

[0287] [Operation] Next, the operations of the working machine, information processing apparatus, and program according to the present embodiment will be described.

[0288] In the first aspect of the present embodiment, the working machine includes an environmental information acquisition unit, a verbalization unit, an instruction acquisition unit, and a control unit. The working machine is, for example, the above-described excavator 100. The environmental information acquisition unit is, for example, the above-described imaging device 40. The verbalization unit is, for example, the above-described verbalization unit 303. The instruction acquisition unit is, for example, the above-described instruction acquisition unit 301. The control unit is, for example, the above-described operation control unit 306. Specifically, the environmental information acquisition unit acquires information representing the environment around the working machine. Further, the verbalization unit verbalizes the information acquired by the environmental information acquisition unit in natural language. Further, the instruction acquisition unit acquires an instruction in natural language from the operator. Further, the control unit controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit using a language model. The language model is, for example, the above-described language model LM1 or language model LM2.

[0289] Further, in the first aspect of the present embodiment, the operation support system may include the environmental information acquisition unit, the verbalization unit, the instruction acquisition unit, and the control unit. The operation support system is, for example, the remote operation support system SYS.

[0290] Further, in the first aspect of the present embodiment, the information processing device may include a verbalization unit that verbalizes information representing the environment around the working machine in natural language, the instruction acquisition unit, and the control unit. The information processing device is, for example, the above-described controller 30 or the remote operation support device 200.

[0291] Further, in the first aspect of the present embodiment, the program may cause the information processing device to execute a verbalization step, an instruction acquisition step, and a control step. Specifically, in the verbalization step, information representing the environment around the working machine is verbalized in natural language. Further, in the instruction acquisition step, an instruction in natural language from the operator is acquired. Then, in the control step, the operation of the working machine is controlled based on the result of interpreting the instruction acquired in the instruction acquisition step and the information verbalized in the verbalization step using a language model.

[0292] As a result, an operator can operate a working machine, for example, by an instruction adapted to the environment around the working machine in natural language. Therefore, a working machine, an operation support system, an information processing apparatus (hereinafter referred to as "working machine etc.") can improve the effectiveness of operating the working machine according to an instruction in the natural language of the operator.

[0293] Further, in the second aspect of the present embodiment, on the premise of the above-described first aspect, the language conversion unit may convert information on construction drawings into natural language in addition to the information acquired by the environment information acquisition unit.

[0294] As a result, an operator can operate a working machine, for example, by an instruction adapted to the content of the construction drawings in natural language. Therefore, a working machine etc. can improve the effectiveness of operating the working machine according to an instruction in the natural language of the operator.

[0295] Further, in the third aspect of the present embodiment, on the premise of the above-described first or second aspect, a working machine etc. may include a determination unit that determines the level of urgency of an instruction acquired by the instruction acquisition unit. The determination unit is, for example, the above-described urgency determination unit 307. When the control unit determines that the urgency is high by the determination unit, the control unit causes the working machine to execute a predetermined operation corresponding to the content of the instruction acquired by the instruction acquisition unit. When the determination unit determines that the urgency is low, the control unit may control the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information converted into natural language by the language conversion unit by the language model.

[0296] As a result, the working machine can quickly execute an emergency operation corresponding to an instruction for an instruction with a relatively high level of urgency.

[0297] Further, in the fourth aspect of the present embodiment, on the premise of the above-described third aspect, the determination unit may determine that the urgency is high when a predetermined word indicating a high level of urgency is included in the sentence of the instruction acquired by the instruction acquisition unit.

[0298] As a result, a working machine or the like can appropriately determine the urgency of an instruction.

[0299] Further, in the fifth aspect of the present embodiment, on the premise of any one of the first to fourth aspects described above, the language model is provided outside so as to be able to communicate with the working machine, and includes a first language model that is relatively large in scale and a second language model that is incorporated into the working machine and is relatively small in scale. The first language model is, for example, the language model LM1 described above. The second language model is, for example, the language model LM2 described above. Further, a working machine or the like may include a selection unit that selects which of the first language model and the second language model to use based on the content of the instruction acquired by the instruction acquisition unit. The selection unit is, for example, the language model selection unit 308 described above. Then, the control unit may control the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit using the language model selected by the selection unit.

[0300] As a result, a working machine or the like can properly distinguish between the first language model and the second language model according to the content of the instruction. Therefore, a working machine or the like can, for example, suppress the communication cost and usage fee cost for using the first model.

[0301] Further, in the sixth aspect of the present embodiment, on the premise of the fifth aspect described above, the selection unit may select which of the first language model and the second language model to use based on at least one of the length of the sentence of the instruction acquired by the instruction acquisition unit and the rarity of the words included in the instruction.

[0302] As a result, in the case where the instruction text is relatively long or the rarity of the words included in the instruction is relatively high, the machine tool or the like can use the first language model, and in other cases, can use the second language model. Therefore, the machine tool or the like can suppress the communication cost and usage fee for using the first model while ensuring the appropriateness and accuracy of the operation of the machine tool with respect to the operator's instruction.

[0303] Further, in the seventh aspect of the present embodiment, on the premise of the above-described first or second aspect, the language model is provided outside so as to be communicable with the machine tool, and includes a first language model that is relatively large in scale, and a second language model that is incorporated in the machine tool and is relatively small in scale. Further, the machine tool or the like may include a classification unit that classifies the instruction acquired by the instruction acquisition unit into a first instruction for causing the machine tool to perform an emergency operation, a second instruction for controlling the machine tool based on the result of interpreting the instruction by the first language model, and a third instruction for controlling the machine tool based on the result of interpreting the instruction by the second language model. The classification unit is, for example, the above-described instruction classifier 307A. Then, the control unit may control the operation of the machine tool according to the instruction acquired by the instruction acquisition unit based on the classification result of the classification unit.

[0304] As a result, the machine tool or the like can realize the functions of the above-described determination unit and selection unit by a classification unit corresponding to a single three-class classifier.

[0305] Further, in the eighth aspect of the present embodiment, on the premise of any one of the above-described first to sixth aspects, the machine tool or the like may include a notification unit that notifies the operator in advance of the operation that the control unit causes the machine tool to execute with respect to the instruction acquired by the instruction acquisition unit. Then, after the notification by the notification unit, when permission is obtained from the operator, the control unit may control the operation of the machine tool based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit by the language model.

[0306] As a result, the working machine can have the operator confirm in advance the operating content of the excavator based on the content of the instruction, and perform the operation based on the operator's instruction on the premise that permission has been obtained. Therefore, the working machine and the like can suppress inappropriate operations of the working machine and improve the safety of the working machine.

[0307] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

Explanation of Reference Numerals

[0308] 1 Lower Travel Body 2 Swing Mechanism 3 Upper Swing Structure 4 Boom 5 Arm 6 Bucket 10 Cabin 11 Engine 14 Main Pump 15 Pilot Pump 17 Control Valve 26 Operating Device 30 Controller 31 Hydraulic Control Valve 40 Imaging Device 50 Output Device 52 Input Device 60 Communication Device 100 Excavator 200 Remote Operation Support Device 207 Input Device 208 Display Device 301 Instruction Acquisition Unit 302 Object Detection Unit 303 Verbalization Unit 304 Prompt Generation Unit 304A Prompt Generation Unit 304B Prompt Generation Unit 305 Call Unit 306 Operation Control Unit 307 Urgency Judgment Unit 307A Indicator Classifier 308 Language Model Selection Unit 309 Notification Unit AT Attachment HA Hydraulic Actuator LM1 Language Model LM2 Language Model SYS Remote Operation Support System

Claims

1. An environmental information acquisition unit that acquires information representing the environment around the working machine, A language conversion unit that converts the information acquired by the environmental information acquisition unit into natural language, An instruction acquisition unit that acquires an instruction in natural language from an operator, A control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information converted into natural language by the language conversion unit, A working machine.

2. In addition to the information acquired by the environmental information acquisition unit, the language conversion unit converts the information on the construction drawings into natural language, The working machine according to Claim 1.

3. A determination unit that determines the level of urgency of the instruction acquired by the instruction acquisition unit, When the determination unit determines that the urgency is high, the control unit causes the working machine to execute a predetermined operation corresponding to the content of the instruction acquired by the instruction acquisition unit. When the determination unit determines that the urgency is low, the control unit controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information converted into natural language by the language conversion unit, The working machine according to Claim 1 or 2.

4. The determination unit determines that the urgency is high when a predetermined word indicating a high level of urgency is included in the sentence of the instruction acquired by the instruction acquisition unit, The working machine according to Claim 3.

5. The language model includes a first language model provided externally so as to be able to communicate with the working machine and having a relatively large scale, and a second language model incorporated in the working machine and having a relatively small scale, A selection unit that selects which of the first language model and the second language model to use based on at least one of the length of the sentence of the instruction acquired by the instruction acquisition unit and the rarity of the words included in the instruction, The control unit controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information converted into natural language by the language conversion unit using the language model selected by the selection unit, The working machine according to Claim 1 or 2.

6. The selection unit selects which of the first language model and the second language model to use based on at least one of the length of the sentence of the instruction acquired by the instruction acquisition unit and the rarity of the words included in the instruction, The working machine according to Claim 5.

7. The language model is provided externally so as to be able to communicate with the working machine, and includes a first language model that is relatively large-scale and a second language model that is incorporated into the working machine and is relatively small-scale. The classification unit classifies the instruction acquired by the instruction acquisition unit into a first instruction for causing the working machine to perform an emergency operation, a second instruction for controlling the working machine based on the result of interpreting the instruction by the first language model, and a third instruction for controlling the working machine based on the result of interpreting the instruction by the second language model. The control unit controls the operation of the working machine according to the instruction acquired by the instruction acquisition unit based on the classification result of the classification unit. The working machine according to claim 1 or 2.

8. The notification unit notifies the operator in advance of the operation that the control unit causes the working machine to execute with respect to the instruction acquired by the instruction acquisition unit. After the notification by the notification unit, when permission is obtained from the operator, the control unit controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit by the language model. The working machine according to claim 1 or 2.

9. An environment information acquisition unit that acquires information representing the environment around the working machine, A verbalization unit that verbalizes the information acquired by the environment information acquisition unit in natural language, An instruction acquisition unit that acquires an instruction in natural language from the operator, And a control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit by the language model. An operation support system.

10. A verbalization unit that verbalizes the information representing the environment around the working machine in natural language, An instruction acquisition unit that acquires an instruction in natural language from the operator, And a control unit that controls the operation of the working machine based on the result of interpreting the instruction acquired by the instruction acquisition unit and the information verbalized by the verbalization unit by the language model. An information processing apparatus.

11. In the information processing apparatus, A verbalization step of verbalizing the information representing the environment around the working machine in natural language, An instruction acquisition step of acquiring an instruction in natural language from the operator, A control step of controlling the operation of the working machine based on a result of interpreting, by a language model, the instruction acquired in the instruction acquisition step and the information verbalized in the verbalization step. Program.

Images