System for work machine

Abstract

A system for a work machine comprising: a work machine having an upper revolving body, a lower traveling body, an attachment including a boom, an arm, and an end attachment, and a control device; a device for acquiring voice data or text data as input information; and a sensor for acquiring line-of-sight information indicating the direction along the line of sight of a person who operates the work machine, the control device generating information, on the basis of the input information and the line-of-sight information, pertaining to commands directed to the work machine.

Description

Systems for work machines

[0001] The present invention relates to a system for a work machine.

[0002] In recent years, a technology has become known for autonomous driving systems for construction machinery that analyzes voice signals indicating the voice of an operator to understand the instructions from the operator and control the construction machinery.

[0003] Japanese Patent Application Laid-Open No. 2023-15628

[0004] In the conventional technology described above, for example, when an instruction from an operator is expressed in a short sentence, there is a possibility that the information obtained from the voice signal is insufficient and the content of the instruction cannot be accurately understood.

[0005] The object is to provide a system for a work machine that allows instructions to be given using short sentences.

[0006] One aspect of the present disclosure is a system for a work machine comprising: a work machine having an upper rotating body, a lower running body, attachments including a boom, an arm, and an end attachment; a control device; a device that acquires voice data or text data as input information; and a sensor that acquires gaze information that indicates the line of sight of a person operating the work machine, wherein the control device generates command information for the work machine based on the input information and the gaze information.

[0007] A system for a work machine that allows instructions to be given using short sentences can be provided.

[0008] 1 is a diagram illustrating an example of the system configuration of a system for a work machine. FIG. 1 is a side view of a work machine 100 that constitutes the system for a work machine shown in FIG. 1. FIG. 2 is a top view of the work machine 100 shown in FIG. 2. FIG. 3 is a block diagram illustrating an example of the configuration of the work machine 100. FIG. 4 is a diagram illustrating the functional configuration of a controller. FIG. 5 is a diagram illustrating a language model LM. FIG. 6 is a flowchart illustrating processing by the controller. FIG. 7 is a diagram illustrating a specific example of the operation of the system for a work machine. FIG. 8 is a first diagram illustrating input to a prompt generation unit and output from the language model LM. FIG. 9 is a diagram illustrating an example display in a remote control room. FIG. 10 is a second diagram illustrating input to a prompt generation unit and output from the language model LM. FIG. 11 is a third diagram illustrating input to a prompt generation unit and output from the language model LM. FIG. 12 is a fourth diagram illustrating input to a prompt generation unit and output from the language model LM.

[0009] An embodiment will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the system configuration of a system for a work machine. Fig. 2 is a side view of a work machine 100 that constitutes the system for a work machine shown in Fig. 1. Fig. 3 is a top view of the work machine 100 shown in Fig. 2.

[0010] The work machine system SYS of this embodiment is an example of a remote control system that remotely controls a work machine 100. As shown in Figure 1 , the work machine system SYS includes one or more work machines 100 and a remote control room RC for remotely controlling the work machines 100.

[0011] The system SYS for a work machine in this embodiment may include a management device for managing the work machine 100, an assistance device for assisting in the operation of the work machine 100 on site, etc. The management device may be, for example, an information processing device installed in a management center or the like that is physically separated from the work site, and the assistance device may be a portable terminal device that can be carried by a worker or the like on site.

[0012] Here, an outline of the operation of the system SYS for a work machine according to this embodiment will be described.

[0013] The system SYS for a work machine of this embodiment detects the direction in which the line of sight of the person operating the work machine 100 is directed.

[0014] The person operating the work machine 100 may be a remote operator OP, or an operator in the cabin 10 of the work machine 100. The person operating the work machine 100 may also be a worker or the like who assists the work performed by the work machine 100 at a work site or the like. In the following embodiment, the person operating the work machine 100 will be mainly described as a remote operator OP.

[0015] The work machine system SYS has a camera A2 installed inside the remote control room RC, and detects the direction in which the remote operator OP is looking based on image data acquired by the camera A2. In the following description, the direction in which the remote operator OP is looking is referred to as the remote operator OP's looking direction, and information indicating the looking direction is referred to as looking information.

[0016] When a remote operator OP issues an instruction to the work machine 100 by voice, the work machine system SYS of this embodiment generates a prompt using the remote operator OP's gaze information and the voice of the remote operator OP and inputs it into a large-scale language model. The work machine system SYS then causes the large-scale language model to output a control code for operating the work machine 100 in accordance with the voice instruction, and controls the operation of the work machine 100 based on this control code.

[0017] In this embodiment, in addition to the voice instructions from the remote operator OP, a prompt is generated using gaze information that indicates the line of sight of the remote operator OP. Therefore, in this embodiment, even if the voice instructions are short sentences and lack information, the gaze information can supplement the lacking information, and a prompt containing the information necessary to operate the work machine 100 according to the instructions can be provided to the large-scale language model. Therefore, in this embodiment, even if the instructions to the work machine 100 are short sentences, it is possible to obtain a control code from the large-scale language model to operate the work machine 100 according to the voice instructions.

[0018] The remote control room RC is equipped with a remote communication device T2, a display device D1, a remote controller 60, and a remote operation device 42. The remote communication device T2, the display device D1, and the remote operation device 42 are installed, for example, in the remote control room RC, which is separated from the work machine 100. In addition, an operator's seat DS, a camera A2, etc. are installed in the remote control room RC. In addition, a microphone A3 for collecting the voice uttered by the remote operator OP is installed in the remote control room RC. Note that the microphone A3 for collecting the voice uttered by the remote operator OP may be equipped in the camera A2. In addition, a speaker for outputting voice may be provided in the remote control room RC.

[0019] The display device D1 installed in the remote control room RC is a device capable of displaying various types of information. The display device D1 displays images based on information transmitted from the work machine 100 so that the remote operator OP in the remote control room RC can visually confirm the area around the work machine 100. The display device D1 includes a liquid crystal display or the like that displays images captured by the imaging device 40 (see FIG. 2) mounted on the work machine 100, and is configured to be able to display images received by the remote communication device T2. The display device D1 may be a display or projector that achieves naked-eye stereoscopic vision, or may be VR goggles or the like.

[0020] The remote control device 42 installed in the remote control room RC has, for example, an operation lever, an operation pedal, an operation panel, an operation button, an operation switch, an operation dial, etc. (not shown), similar to the operation device 26 (see FIGS. 2 and 3) inside the cabin 10 of the work machine 100. The remote control device 42 includes, for example, an operation sensor 43 that detects operation of the remote control device 42 by the remote operator OP.

[0021] The operator's seat DS installed in the remote control room RC has, for example, the same configuration as the operator's seat in the cabin 10 of the work machine 100. A remote operator OP who remotely operates the work machine 100 sits in the operator's seat DS, for example, and operates the remote control device 42. Note that the work machine 100 may also be, for example, a work machine 100 dedicated to remote operation, which does not have the cabin 10, driver's seat, or operation device 26.

[0022] The camera A2 may be disposed above the display device D1. The camera A2 may be disposed in a position where it can detect the line of sight of the remote operator OP seated at the operation seat DS. The camera A2 in this embodiment is an example of a line of sight detection sensor that detects the line of sight of the remote operator OP.

[0023] In this embodiment, the gaze detection sensor is not limited to the camera A2 arranged inside the remote control room RC. The gaze detection sensor may be a wearable device or the like. Specifically, for example, the gaze detection sensor may be a head-mounted camera, and in that case, a microphone for acquiring audio data indicating the speech of the remote operator OP may be mounted together with the camera.

[0024] The speaker installed in the remote control room RC is configured to be able to reproduce sound based on a signal received by the remote communication device T2, for example. The speaker may be a monaural speaker, a stereo speaker, or a surround speaker. The speaker may also be an omnidirectional speaker or a directional speaker. The speaker may also include a wearable device such as earphones or headphones. The wearable device may have a noise canceling function, a spatial audio function (stereophonic function), or a bone conduction function.

[0025] The remote controller 60 installed in the remote control room RC is an example of a control device, and has a similar configuration to the controller 30 of the work machine 100, which will be described later, for example. The remote controller 60 acquires various information and signals transmitted from the communication device T1 of the work machine 100 via the remote communication device T2, for example, and transmits various information and signals to the communication device T1 of the work machine 100 via the remote communication device T2. The remote controller 60 also displays images received via the remote communication device T2 on the display device D1, and plays sounds received via the remote communication device T2 in a speaker, for example.

[0026] The remote controller 60 also acquires information about the operation of the remote operation device 42 by the remote operator OP from the operation sensor 43. The operation sensor 43 is, for example, an inclination sensor that detects the inclination angle of the operation lever of the remote operation device 42, or an angle sensor that detects the swing angle of the operation lever about the swing axis. The operation sensor 43 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 43 outputs information about the detected operation content of the remote operation device 42 to the remote controller 60.

[0027] The remote controller 60 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operation sensor 43 may be configured to generate an operation signal. In this case, the operation sensor 43 may output the operation signal to the remote communication device T2 without going through the remote controller 60. This makes it possible to remotely operate the work machine 100 from the remote control room RC. Note that the operation signal based on the operation of the remote operation device 42 may be transmitted to the work machine 100 via a communication device different from the remote communication device T2 that receives images transmitted from the communication device T1 of the work machine 100.

[0028] The work machine 100 transmits, for example, via the communication device T1, an image showing the surroundings including the front of the work machine 100, based on an image output by the imaging device 40 mounted on the work machine 100 (hereinafter referred to as a "peripheral image") to the remote controller 60 in the remote control room RC. Furthermore, the work machine 100 may transmit the image output by the imaging device 40 to the remote controller 60 in the remote control room RC via the communication device T1, and the remote controller 60 may process the image received from the work machine 100 to generate a peripheral image. The remote controller 60 then displays the peripheral image showing the surroundings including the front of the work machine 100 on the display device D1. Furthermore, various information images (information screens) displayed on the output device 50 (display device) installed inside the cabin 10 of the work machine 100 may also be displayed on the display device D1 in the remote control room RC. This allows the remote operator OP to remotely operate the work machine 100 while checking, for example, the display contents of the images and information screens showing the surroundings of the work machine 100 displayed on the display device D1. The work machine 100 then operates actuators in response to signals indicating the details of remote operation received from the remote control room RC via the communication device T1, and operates driven elements such as the undercarriage 1, upper rotating body 3, boom 4, arm 5, and bucket 6. In this way, the work machine system SYS can realize remote operation of the work machine 100 from the remote control room RC.

[0029] As shown in Figure 2, the work machine 100 comprises a lower running body 1, an upper rotating body 3 rotatably mounted on the lower running body 1, an imaging device 40 mounted on the upper rotating body 3, and a communication device T1 that transmits image data acquired by the imaging device 40.

[0030] In the example shown in Fig. 2, the work machine 100 is a hydraulic excavator, but the work machine that constitutes the work machine remote control system according to the present disclosure is not limited to a hydraulic excavator. The work machine 100 may be an EV excavator driven only by an electric motor, a hybrid excavator, or a tire-powered wheel excavator. The work machine 100 may also be another work machine, such as a crawler crane or a CSU (Continuous Ship Unloader).

[0031] 2 and 3, for example, the undercarriage 1 of the work machine 100 has left and right crawlers 1CL, 1CR driven by left and right traveling hydraulic motors 2ML, 2MR, and the left and right crawlers 1CL, 1CR are rotated to travel the work machine 100. The upper rotating body 3 of the work machine 100 is mounted on the undercarriage 1 via a swing mechanism 2 driven by a swing hydraulic motor 2A, for example, as shown in Figures 2 and 3, and is thereby provided to be rotatable on the undercarriage 1.

[0032] The three-dimensional orthogonal coordinate system shown in Fig. 2 has an X axis parallel to the front-to-rear direction of the upper rotating body 3, a Y axis parallel to the left-to-right direction of the upper rotating body 3, and a Z axis parallel to the up-to-down direction of the upper rotating body 3. In addition, in the three-dimensional orthogonal coordinate system shown in Fig. 2, the positive direction of the X axis (+X direction), the positive direction of the Y axis (+Y direction), and the positive direction of the Z axis (+Z direction) point forward, leftward, and upward, respectively, of the upper rotating body 3.

[0033] Imaging device 40 of work machine 100 is provided, for example, on the upper rotating body 3, and captures images of the surroundings of the upper rotating body 3. As shown in Fig. 2 , imaging device 40 includes, for example, a front camera 40F that captures images in front of the upper rotating body 3, a left camera 40L that captures images to the left of the upper rotating body 3, a right camera 40R that captures images to the right of the upper rotating body 3, a rear camera 40B that captures images behind the upper rotating body 3, and a camera 40C that captures images of the inside of cabin 10. Front camera 40F, left camera 40L, right camera 40R, rear camera 40B, and camera 40C are, for example, monocular wide-angle cameras equipped with imaging elements such as CCD or CMOS, and output captured images to controller 30.

[0034] Furthermore, in the work machine 100 of this embodiment, the image captured by the camera 40C may be used to detect the line of sight of the operator seated in the driver's seat inside the cabin 10.

[0035] It should be noted that some or all of the rear camera 40B, left camera 40L, and right camera 40R may be omitted. Furthermore, instead of or in addition to the image capture device 40, the work machine 100 may be provided with a ranging sensor (also referred to as a "distance sensor") that can acquire information indicating the distance to objects in the vicinity of the work machine 100. Examples of ranging sensors include LiDAR (Light Detecting and Ranging), millimeter-wave radar, and ultrasonic sensors. The image capture device 40 may be an example of an object detection device that detects a specified object within a specified area set around the work machine 100. The object detection device may be a human detection device configured to be able to detect people while distinguishing between people and non-human objects.

[0036] In addition, in this embodiment, a microphone A11 is provided in the cabin 10 to collect voices within the cabin 10. In this embodiment, when an operator in the cabin 10 speaks, the voice representing the speech is collected by the microphone A11.

[0037] 1, the communication device T1 of the work machine 100 communicates with a remote communication device T2 installed in a remote control room RC via a communication network NW that includes a mobile communication network, a satellite communication network, or the Internet network, for example. The communication device T1 is, for example, a mobile communication module that supports mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module that supports short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0038] A remote communication device T2 installed in the remote control room RC communicates with a communication device T1 mounted on the work machine 100 via a communication network NW, for example.

[0039] The controller 30 provided in the work machine 100 is an example of a control device, and is configured, for example, by a computer that includes a CPU, a volatile storage device, a non-volatile storage device, and various input / output interfaces, etc. The controller 30 then implements various functions, for example, by reading a program from the non-volatile storage device, loading it into the volatile storage device, and having the CPU execute it.

[0040] The controller 30 is configured to realize various functions to control the work machine 100. The various functions include, for example, a machine guidance function that guides the remote operator OP in manually operating the work machine 100. The various functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and an object that is present within a monitoring range around the work machine 100.

[0041] The controller 30 acquires various information and signals transmitted from the remote communication device T2 in the remote control room RC via the communication device T1, and transmits various information, signals, images, sounds, etc. to the remote communication device T2 in the remote control room RC via the communication device T1.

[0042] The work machine 100 is equipped with, for example, an attachment for performing work at a work site. The attachment includes, for example, a boom 4 rotatably attached to the right side of the front end of the upper rotating body 3, an arm 5 rotatably attached to the tip of the boom 4, and an excavation bucket 6 rotatably attached to the tip of the arm 5, as shown in Figure 2 .

[0043] The bucket 6 is an end attachment that is attached to the end of an attachment of the work machine 100. The end attachment may be another type of bucket, such as a large bucket, a bucket for slopes, or a bucket for dredging, or may be a work tool other than a bucket, such as a mixer, a breaker, a grapple, or a lifting magnet.

[0044] The boom 4, arm 5, and bucket 6 are hydraulically driven by hydraulic cylinders: a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively. A boom angle sensor S1, an arm angle sensor S2, and a bucket angle sensor S3 are attached to the boom 4, arm 5, and bucket 6, respectively. These angle sensors also function as attitude sensors that detect the attitude of the attachment.

[0045] Furthermore, as shown in Figs. 2 and 3, the work machine 100 is equipped with a machine body tilt sensor S4, a turning sensor S5, a positioning device S6, and a microphone A1.

[0046] The boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body tilt sensor S4, rotation sensor S5, and positioning device S6 will be described in detail later.

[0047] The microphone A1 is a sound collecting device that is provided on the upper front side of the cabin 10 of the work machine 100, for example, as shown in Fig. 2, and collects sounds generated around the work machine 100 and converts them into electrical signals. The sound signals collected by the microphone A1 are taken into the controller 30. A1 may be, for example, a microphone array made up of multiple microphones.

[0048] Next, the configuration of the work machine 100 will be described with reference to Figure 4. Figure 4 is a block diagram showing an example of the configuration of the work machine 100. In Figure 4, paths through which mechanical power is transmitted are indicated by double lines, paths through which high-pressure hydraulic oil that drives the hydraulic actuator HA flows are indicated by solid lines, paths through which pilot pressure is transmitted are indicated by dashed lines, and paths through which electrical signals are transmitted are indicated by dotted lines.

[0049] The work machine 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0050] 4 , the hydraulic drive system of the work machine 100 includes hydraulic actuators HA that hydraulically drive each of the driven elements, such as the lower traveling structure 1 (left and right crawlers 1CL, 1CR), upper rotating structure 3, boom 4, arm 5, and bucket 6, as described above. The hydraulic drive system of the work machine 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0051] The hydraulic actuators HA include traveling hydraulic motors 2ML, 2MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, and bucket cylinder 9. Note that in the work machine 100, some or all of the hydraulic actuators HA may be replaced with electric actuators. In other words, the work machine 100 may be a hybrid excavator or an electric excavator.

[0052] The engine 11 is the prime mover of the work machine 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a controller 30 (described below), for example, and drives the main pump 14 and the pilot pump 15. Note that instead of or in addition to the engine 11, another prime mover (for example, an electric motor) may be mounted on the work machine 100.

[0053] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

[0054] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the regulator 13 adjusts the tilt angle of the swash plate of the main pump 14, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0055] The control valve 17 drives the hydraulic actuators HA in response to an operator's operation of the operating device 26, the details of remote operation, or an operation command corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (directional control valves) that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0056] As shown in FIG. 4 , the operating system of the work machine 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

[0057] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above. The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic devices as pilot pressure.

[0058] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements, thereby enabling the operator to operate the driven elements that are the targets of the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0059] For example, as shown in FIG. 4 , the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32 and is connected to the control valve 17 via a pilot line 27 connected to the outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0060] Alternatively, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0061] Furthermore, the control valves (directional control valves) that are built into the control valve 17 and drive the hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17 (i.e., to the electromagnetic solenoid type control valve).

[0062] Furthermore, as described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output control commands to the electric actuators or a driver that drives the electric actuators, in accordance with the operation content of the operation device 26 and the content of the remote operation specified by the remote operation signal. Furthermore, when the work machine 100 is remotely operated, the operation device 26 may be omitted.

[0063] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via a shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, for example, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 in accordance with an operation command corresponding to the automatic driving function, thereby realizing operation of the work machine 100 using the automatic driving function.

[0064] The controller 30 may also control the hydraulic control valve 31 to realize remote operation of the work machine 100. Specifically, the controller 30 outputs, via the communication device T1, a control signal corresponding to the content of the remote operation specified in a remote operation signal received from the remote control room RC to the hydraulic control valve 31. As a result, the controller 30 can cause the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby realizing operation of the work machine 100 based on remote operation by the operator.

[0065] Furthermore, if the operating device 26 is electric, the controller 30 can supply pilot pressure corresponding to the operation content (operation signal) of the operating device 26 directly from the hydraulic control valve 31 to the control valve 17, thereby realizing operation of the work machine 100 based on the operation of the operator.

[0066] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having the higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, a shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to a secondary pilot line 27A of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives a hydraulic actuator HA that is the target of operation of the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6) regardless of the operating state of the operating device 26 by the operator, thereby realizing an automatic driving function or a remote control function.

[0067] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the automatic driving function and remote control function of the work machine 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0068] As shown in FIG. 4 , the user interface system of the work machine 100 includes the operation device 26 , an output device 50 , and an input device 52 .

[0069] The output device 50 outputs various information to the user of the work machine 100 (for example, the operator of the cabin 10 or an external remote operator) and people in the vicinity of the work machine 100 (for example, workers or drivers of work vehicles).

[0070] For example, the output device 50 includes lighting equipment, display devices, etc. that output various types of information visually. Examples of lighting equipment include warning lights (indicator lamps), etc. Examples of display devices include liquid crystal displays and organic EL (electroluminescence) displays, etc. For example, the lighting equipment and display devices may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting equipment and display devices may be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the work machine 100.

[0071] The output device 50 may also include a sound output device that outputs various types of information auditorily. Sound output devices include, for example, buzzers and speakers. The sound output device may be provided, for example, at least one of the inside and outside of the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the work machine 100. The output device 50 may also include a device that outputs various types of information tactilely, such as by vibrating the cockpit.

[0072] The input device 52 accepts various inputs from the user of the work machine 100. A signal corresponding to the input accepted by the input device 52 is taken into the controller 30. For example, as shown in Fig. 3 , the input device 52 is provided inside the cabin 10 and accepts inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3 and accept inputs from a worker or the like in the vicinity of the work machine 100.

[0073] For example, the input device 52 includes a mechanical input device that accepts input from a user through mechanical operation. The mechanical input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0074] The input device 52 may also include a voice input device that accepts voice input from the user. The voice input device includes, for example, a microphone A11. The input device 52 may also include a gesture input device that accepts gesture input from the user. The gesture input device includes, for example, a camera 40C that captures images of gestures made by the user. The input device 52 may also include a biometric input device that accepts biometric input from the user. The biometric input includes, for example, input of biometric information such as the user's fingerprint or iris.

[0075] The control system of the work machine 100 includes a controller 30. The control system of the work machine 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, and the sensors described above.

[0076] The controller 30 performs various controls related to the work machine 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, as shown in Fig. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus BS1.

[0077] The auxiliary storage device 30A is a non-volatile storage means that stores the installed program as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory. For example, when a program startup instruction is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0078] The CPU 30C, for example, executes a program loaded into the memory device 30B and, in accordance with the instructions of the program, realizes the various functions of the controller 30. The interface device 30D functions, for example, as a communication interface for connecting to a communication line inside the work machine 100. The interface device 30D may include a plurality of different types of communication interfaces in accordance with the type of communication line to be connected.

[0079] The interface device 30D also functions as an external interface for reading data from a recording medium and writing data to a recording medium. The recording medium is, for example, a dedicated tool that is connected via a detachable cable to a connector installed inside the cabin 10. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes the various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may also be downloaded from another computer external to the work machine 100 via the communication device T1 and installed in the auxiliary storage device 30A.

[0080] Note that some of the functions of the controller 30 may be realized by another controller (control device). In other words, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the work machine 100.

[0081] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure by the operating pressure sensor 29, which corresponds to the operating state of each driven element (hydraulic actuator HA) in the operating device 26, is input to the controller 30.

[0082] It should be noted that if the operating device 26 is electric or if the operating device 26 is omitted, the operating pressure sensor 29 is omitted. In these cases, the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operation signal received from the operating device 26.

[0083] The boom angle sensor S1 is attached to the boom 4 and measures the attitude of the boom 4. The boom angle sensor S1 outputs measurement data representing the attitude of the boom 4. The attitude of the boom 4 is, for example, the attitude angle around the rotation axis of the base end of the boom 4, which corresponds to the connection part between the boom 4 and the upper rotating body 3 (hereinafter referred to as the "boom angle"). The boom angle sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may apply to the arm angle sensor S2, the bucket angle sensor S3, and the machine body inclination sensor S4 below. The boom angle sensor S1 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may apply to the arm angle sensor S2 and the bucket angle sensor S3 below. The output of the boom angle sensor S1 (measurement data representing the attitude of the boom 4) is input to the controller 30. This allows the controller 30 to grasp the attitude of the boom 4.

[0084] The arm angle sensor S2 is attached to the arm 5 and measures the posture of the arm 5. The arm angle sensor S2 outputs measurement data representing the posture of the arm 5. The posture of the arm 5 is, for example, the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The output of the arm angle sensor S2 (measurement data representing the posture of the arm 5) is input to the controller 30. This allows the controller 30 to grasp the posture of the arm 5.

[0085] The bucket angle sensor S3 is attached to the bucket 6 and measures the attitude of the bucket 6. The bucket angle sensor S3 outputs measurement data that indicates the attitude of the bucket 6. The attitude of the bucket 6 is, for example, the attitude angle (hereinafter referred to as the "bucket angle") around the rotation axis of the base end of the bucket 6 that corresponds to the connection part with the arm 5. The output of the bucket angle sensor S3 (measurement data that indicates the attitude of the bucket 6) is input to the controller 30. This enables the controller 30 to grasp the attitude of the bucket 6.

[0086] The machine body inclination sensor S4 measures the attitude state of the machine body (e.g., the upper rotating body 3) of the work machine 100. The machine body inclination sensor S4 outputs measurement data that indicates the attitude state of the machine body of the work machine 100. The attitude state of the machine body of the work machine 100 is, for example, the inclination state of the machine body with respect to a predetermined reference plane (e.g., a horizontal plane). For example, the machine body inclination sensor S4 is attached to the upper rotating body 3 and measures the inclination angles of the work machine 100 about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft inclination angle" and the "lateral inclination angle"). The output of the machine body inclination sensor S4 (measurement data that indicates the attitude state of the machine body of the work machine 100) is taken into the controller 30. This allows the controller 30 to grasp the attitude state (inclination state) of the machine body (upper rotating body 3).

[0087] The rotation sensor S5 is attached to the upper rotating body 3 and measures the rotation state of the upper rotating body 3. The rotation sensor S5 outputs measurement data representing the rotation state of the upper rotating body 3. The rotation sensor S5 measures, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The rotation sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the rotation sensor S5 (measurement data representing the rotation state of the upper rotating body 3) is input to the controller 30. This allows the controller 30 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0088] The controller 30 can grasp (estimate) the position of the tip of the attachment AT (bucket 6) based on the outputs of the above-mentioned sensors. If the machine body tilt sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocity around three axes, the rotation state of the upper rotating body 3 (e.g., rotation angular velocity) may be detected based on the detection signal of the machine body tilt sensor S4. In this case, the rotation sensor S5 may be omitted.

[0089] The positioning device S6 measures the position of the work machine 100. The positioning device S6 may measure the position in a world (global) coordinate system, or may measure the position in a local coordinate system at the work site. In the former case, the positioning device S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, the positioning device S6 is a transceiver that communicates with equipment that serves as a reference for the position at the work site, and is capable of outputting a signal corresponding to the position relative to the reference. The output of the positioning device S6 is taken into the controller 30.

[0090] Sensor S7 measures the pressure (cylinder pressure) in the oil chamber of boom cylinder 7. Sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of boom cylinder 7, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side. The output of sensor S7 (i.e., measurement data of the cylinder pressure of boom cylinder 7) is taken into controller 30.

[0091] The sensor S8 measures the pressure (cylinder pressure) in the oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of the arm cylinder 8, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side of the arm cylinder 8. The output of the sensor S8 (i.e., the measurement data of the cylinder pressure of the arm cylinder 8) is taken into the controller 30.

[0092] The sensor S9 measures the pressure (cylinder pressure) of the oil chamber of the bucket cylinder 9. The sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of the bucket cylinder 9, and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side of the bucket cylinder 9. The output of the sensor S9 (i.e., the measurement data of the cylinder pressure of the bucket cylinder 9) is taken into the controller 30.

[0093] The controller 30 can grasp the load state acting on the attachment AT based on the outputs of sensors S7 to S9. The load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from soil on the ground being worked on, and the weight of soil contained in the bucket 6. Note that some or all of the sensors S7 to S9 may be omitted depending on necessity. The work machine 100 may be equipped with other sensors capable of grasping the state of the work machine 100. For example, the work machine 100 may be equipped with a direction sensor capable of detecting its own direction. The direction sensor is, for example, an electronic compass including a geomagnetic sensor.

[0094] Here, the line of sight information of this embodiment will be described. In this embodiment, the remote controller 60 acquires line of sight information indicating the line of sight direction of the remote operator OP from an image captured by the camera A2, and transmits the line of sight information to the controller 30 of the work machine 100 via the remote communication device T2. Note that the line of sight information may also be acquired by the camera A2 as a line of sight detection sensor, and transmitted to the controller 30 of the work machine 100 via the remote communication device T2.

[0095] In this embodiment, the line of sight information is used to identify an object designated by the remote operator OP from among objects, people, etc. detected around the work machine 100. Specifically, in this embodiment, the object designated by the remote operator OP is identified from the direction indicated by the line of sight information and the position of the work machine 100.

[0096] In this embodiment, coordinates indicating the position of an object designated by the remote operator OP may be acquired and used when generating a prompt. The line of sight information and the coordinates of the object designated by the remote operator OP in this embodiment are examples of numerical information.

[0097] In this embodiment, by using line-of-sight information that indicates the line-of-sight direction of the remote operator OP, it is not necessary to include specific information for identifying the object designated by the remote operator OP in a voice instruction. In other words, in this embodiment, the object designated by the remote operator OP can be expressed using demonstrative terms such as "here," "there," or "this" in a voice instruction, making it possible to give instructions to the work machine 100 in short sentences.

[0098] Furthermore, according to this embodiment, the remote operator OP can give instructions to the work machine 100 simply by speaking while directing his or her gaze. Therefore, according to this embodiment, the remote operator OP does not need to take his or her hands off the remote control device 42 when giving instructions to the work machine 100, which reduces the burden on the remote operator OP and improves safety.

[0099] Next, the functional configuration of the controller 30 will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the functional configuration of the controller.

[0100] When an instruction to the work machine 100 is input by voice, the controller 30 of this embodiment generates a prompt to be input to the large-scale language model using text data converted from the voice and gaze information transmitted from the remote control room RC. The prompt is an example of text data for expressing an instruction or question to the large-scale language model.

[0101] Furthermore, the controller 30 inputs the generated prompt into a large-scale language model, acquires a control code output from the large-scale language model for operating the work machine 100 in accordance with the voice instructions, and controls the operation of the work machine 100 based on this control code.

[0102] The controller 30 of this embodiment has an information acquisition unit 301, an instruction acquisition unit 302, a verbalization unit 303, a prompt generation unit 304, a calling unit 305, an instruction output unit 306, an operation control unit 307, and a display control unit 308. These units of the controller 30 represent the functions of the controller 30 that are realized, for example, by loading a program stored in an auxiliary storage device 30A into a memory device 30B and executing it in a CPU 30C. In addition, a language model LM is provided outside the work machine 100.

[0103] The language model LM is, for example, a large language model (LLM). The language model LM is implemented in an external device that is communicably connected to the work machine 100 via the communication device T1. The language model LM is, for example, GPT-4.

[0104] The external device connected to the work machine 100 so as to be able to communicate with each other may be a management device for managing the work machine 100, or may be an assistance device for assisting the operation of the work machine 100. Furthermore, the external device connected to the work machine 100 so as to be able to communicate with each other may be a remote controller 60 provided in the remote control room RC.

[0105] The information acquisition unit 301 acquires, for example, the output values ​​of the sensors of the work machine 100 as detection results. The detection results acquired by the information acquisition unit 301 may include, for example, output values ​​of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, and rotation sensor S5, image data captured by the imaging device 40, coordinates indicating the position of the work machine 100, and the like.

[0106] Furthermore, the detection results acquired by the information acquisition unit 301 include line-of-sight information transmitted from the remote control room RC. When the work machine 100 is operated by an operator inside the cabin 10, the information acquisition unit 301 acquires line-of-sight information of the operator inside the cabin 10, which is acquired by a camera 40C that captures images of the interior of the cabin 10.

[0107] Of the detection results acquired by the information acquisition unit 301, the output values ​​of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, and rotation sensor S5, etc., are information relating to the attitude of the attachment AT. Also, of the detection results acquired by the information acquisition unit 301, the image data captured by the imaging device 40, the coordinates indicating the position of the work machine 100, etc. are information relating to the environment around the work machine 100.

[0108] Furthermore, the information acquisition unit 301 acquires, for example, from the imaging device 40 or a distance measurement sensor, the detection results of objects present around the work machine 100 as information relating to the environment around the work machine 100. Furthermore, the information acquisition unit 301 may detect objects around the work machine 100 by, for example, acquiring image data from the imaging device 40, or acquiring the distance and direction to objects around the work machine 100 from a distance measurement sensor.

[0109] Specifically, the information acquisition unit 301 detects monitored objects from image data captured by the imaging device 40 by, for example, optionally applying known image processing techniques such as semantic segmentation or machine learning. Monitored objects include, for example, people such as workers. Monitored objects may also include other obstacles present around the work machine 100. Examples of other obstacles include specific moving objects present at the work site of the work machine 100, such as other work machines or work vehicles. Examples of other obstacles may also include specific stationary objects present at the work site of the work machine 100, such as utility poles, fences, and traffic cones (also referred to as color cones (registered trademark)). Examples of other obstacles may also include specific topographical shapes present at the work site of the work machine 100, such as ditches, holes, and piles of earth and sand.

[0110] The information acquisition unit 301 outputs the acquired information to, for example, the verbalization unit 303 and the instruction output unit 306 .

[0111] The instruction acquisition unit 302 acquires instructions input in natural language from the remote operator OP. The instructions acquired by the instruction acquisition unit 302 may be, for example, instructions to cause the work machine 100 to perform a predetermined operation from one point to another.

[0112] An instruction input in natural language may be input via a microphone A3 provided in the remote control room RC. In this case, the work machine 100 may acquire voice data indicating the voice spoken by the remote operator OP via the communication network NW, and use the acquired voice data as input to the instruction acquisition unit 302.

[0113] The instruction in natural language is, for example, an instruction input in natural language by the operator's voice. In other words, the instruction in natural language is voice data. The instruction acquisition unit 302 can acquire text data corresponding to the instruction in natural language by applying known voice recognition technology based on the input voice data.

[0114] Furthermore, the instruction in natural language may be an instruction input as text by an operator using an input device capable of inputting characters, such as a keyboard or a touch panel. In this case, the instruction acquisition unit 302 can acquire the input text data as an instruction in natural language.

[0115] In this embodiment, input devices that allow character input, such as a microphone, keyboard, or touch panel, are examples of devices that acquire voice data or text data (input information that does not contain numerical values). Also, instructions in natural language in this embodiment are an example of input information that does not include information (numerical information) acquired by a sensor possessed by the work machine 100.

[0116] The verbalization unit 303 verbalizes the output value of the sensor acquired by the information acquisition unit 301 in natural language.

[0117] The verbalization unit 303 verbalizes, for example, in natural language, the detection result acquired by the information acquisition unit 301. Specifically, the verbalization unit 303 verbalizes, for example, the information acquired by the information acquisition unit 301 by applying it to a predefined text template.

[0118] For example, a template of text data representing line-of-sight information may be specified in a format such as "The operator's line-of-sight direction is x degrees to the right and y degrees upward relative to the direction of travel."

[0119] The prompt generation unit 304 generates a prompt to be input to the language model LM based on the instruction in natural language acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303. Specifically, the prompt generation unit 304 generates a prompt based on text data including gaze information and the text data acquired by the instruction acquisition unit 302.

[0120] The calling unit 305 calls the language model LM via, for example, a predetermined API (Application Programming Interface), inputs the prompt generated by the prompt generating unit 304 to the language model LM, and obtains the output (answer).

[0121] The instruction output unit 306 outputs control instructions in accordance with the instructions acquired by the instruction acquisition unit 302 to the operation control unit 307 and the display control unit 308 based on the output of the language model LM acquired by the call unit 305 and the detection results acquired by the information acquisition unit 301.

[0122] The operation control unit 307 outputs a control signal that controls the operation of the work machine 100 based on the control instruction output from the instruction output unit 306 .

[0123] The display control unit 308 outputs a display control signal that controls the display based on the control instruction output from the instruction output unit 306. The display control unit 308 may output the display control signal to the remote control room RC and cause the display device D1 in the remote control room RC to display information based on the control instruction output from the instruction output unit 306.

[0124] Next, the language model LM of this embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the language model LM.

[0125] 6 is an example of text data that is provided in advance to the language model LM. The text data 61 includes a sentence indicating that the language model LM is a computer for an excavator, information indicating the behavior that the language model LM should perform, information indicating the environment around the work machine 100, etc.

[0126] In this embodiment, by providing the language model LM with text data 61, the language model LM can learn the computer behavior of the work machine 100. Furthermore, in this embodiment, the language model LM may be made to learn in advance a plurality of cases in which input data is associated with correct answer data that is the correct answer for the input data.

[0127] Here, the input data is, for example, text data indicating the instruction content including demonstrative words that specify the position at which the work machine 100 starts operating and the position at which the operation ends, and gaze information, and the correct answer data may be a control code for operating the work machine 100 according to the instruction content.

[0128] Next, the processing of the controller 30 of this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the processing of the controller.

[0129] The controller 30 acquires text data indicating an instruction in natural language via the instruction acquisition unit 302 (step S701).

[0130] Next, the controller 30 acquires the detection results using the information acquisition unit 301 (step S702). The information acquisition unit 301 may calculate the attitude of the attachment AT based on the detection results of, for example, the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, the rotation sensor S5, the positioning device S6, etc. The information acquisition unit 301 may also detect objects around the work machine 100, the direction and distance to the objects, the position coordinates of the objects, etc., based on the output of, for example, the imaging device 40 or a distance measurement sensor.

[0131] Next, the controller 30 causes the verbalization unit 303 to verbalize the information acquired by the information acquisition unit 301 in natural language (step S703).

[0132] Next, the controller 30 translates the text data indicating the instruction acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303 (step S704). In step S704, the prompt generation unit 304 also functions as a translation unit that translates the text data indicating the instruction and the verbalized information into the standard language of the language model LM, i.e., a specific language corresponding to the language model LM. Specifically, if the standard language of the language model LM is English and the text data indicating the instruction and the verbalized information are in a language other than English, the prompt generation unit 304 translates the text data indicating the instruction and the verbalized information into English by machine translation. Note that step S704 may be omitted.

[0133] Next, the controller 30 causes the prompt generation unit 304 to generate a prompt to be input to the language model LM based on the instruction text data and verbalized information, or machine translations of these (step S705).

[0134] Next, the controller 30 calls the language model LM using the calling unit 305 , inputs the prompt generated by the prompt generating unit 304 to the language model LM, and inputs the control code output from the language model LM to the instruction output unit 306 .

[0135] Next, the controller 30 causes the instruction output unit 306 to output a control instruction based on the control code obtained from the language model LM to the operation control unit 307 and the display control unit 308 (step S707).

[0136] Next, the controller 30 causes the operation control unit 307 and the display control unit 308 to output a control signal according to the content of the control instruction (step S708), and ends the process.

[0137] Specifically, for example, if the control instruction includes an instruction regarding the operation of the work machine 100, the operation control unit 307 drives the hydraulic actuator HA with a control signal for causing the work machine 100 to perform an operation in accordance with the control instruction. Furthermore, if the control instruction includes an instruction regarding the display of information, the display control unit 308 transmits a display control signal in accordance with the control instruction to the remote control room RC, and displays the information on the display device D1 of the remote control room RC. Note that the control instruction may include either an instruction regarding the operation of the work machine 100 or an instruction regarding the display of information, or it may include both. Note that the display control unit 308 may display information on a display device provided in the work machine 100.

[0138] Next, a specific example of the operation of the system SYS for a work machine according to this embodiment will be described. Figure 8 is a diagram for explaining a specific example of the operation of the system for a work machine.

[0139] 8 is an example of an image of the surroundings of the work machine 100. In this case, people P1, P2, and P3 are present around the work machine 100, and color cones Cr1 and Cr2 are placed. Image data showing the image 81 may be transmitted from the work machine 100 to the remote control room RC, and the image 81 may be displayed on a display device D1 in the remote control room RC. Alternatively, the image data showing the image 81 may be transmitted from the work machine 100 to a display device arranged in the remote control room RC that is separate from the display device D1, and the image 81 may be displayed on a display device separate from the display device D1.

[0140] The image 81 includes images GP1, GP2, and GP3 of people P1, P2, and P3, respectively, and images GCr1 and GCr2 of traffic cones Cr1 and Cr2, respectively.

[0141] Here, a case will be described in which the remote operator OP, while directing his / her gaze at the image GP2 of person P2, utters, "Set a safe area based on that person." Note that the safe area in this embodiment is a restricted area in a work site where entry or movement speed is restricted. In the following description, the safe area may also be referred to as a restricted area.

[0142] 8 , the line of sight of the remote operator OP at the time when the remote operator OP speaks is a line of sight direction 82 directed toward the image GP2 of person P2. Therefore, the remote controller 60 transmits line of sight information indicating the line of sight direction 82 to the controller 30 of the work machine 100.

[0143] The controller 30 of the work machine 100 acquires gaze information indicating the gaze direction 82 at the timing when it acquires voice data indicating the content of the utterance of the remote operator OP. In other words, the controller 30 of the work machine 100 acquires gaze information indicating the direction in which the remote operator OP was looking when the remote operator OP spoke, and voice data indicating the content of the utterance.

[0144] At this time, the detection results acquired by the information acquisition unit 301 may include image data showing the image 81, and the coordinates of the people P1, P2, and P3 detected around the work machine 100, and the color cones Cr1 and Cr2.

[0145] The controller 30 of the work machine 100 acquires the detection results including line-of-sight information indicating the line-of-sight direction 82 using the information acquisition unit 301, and acquires the voice data using the instruction acquisition unit 302, and then identifies the person P2 detected at coordinates that overlap the line-of-sight direction 82 as the object designated by the remote operator OP. The controller 30 then acquires the coordinates indicating the position of the person P2, and generates a prompt including the coordinates of the person P2.

[0146] In this embodiment, by using gaze information indicating the gaze direction of the remote operator OP, it becomes possible to give instructions to the work machine 100 using demonstrative terms that do not include specific information, such as "use that person as a reference."

[0147] In this embodiment, it is preferable that the remote operator OP continues to direct his / her gaze toward the object while speaking. Specifically, for example, it is preferable that the remote operator OP continues to direct his / her gaze toward the image GP2 while speaking "that person." This allows the language model LM to understand the content of the instruction with higher accuracy.

[0148] In this embodiment, when an object is identified by line-of-sight information on the screen displaying the image 81, an image 83 showing the identified object may be displayed. By displaying the image 83, the remote operator OP can understand that the object has been identified.

[0149] Furthermore, in the example of Figure 8, the work machine 100 is operated by a remote operator OP, and the controller 30 of the work machine 100 acquires line-of-sight information indicating the line-of-sight direction of the remote operator OP from the remote control room RC, but this is not limited to this.

[0150] When the work machine 100 is operated by an operator inside the cabin 10, the controller 30 visually recognizes the scenery around the work machine 100, which is the source of the image 81, as part of the field of view of the operator inside the cabin 10. In this case, the direction of the operator's line of sight may be detected by a camera 40C provided inside the cabin 10.

[0151] Furthermore, when the language model LM of this embodiment receives a prompt input, it may output a response such as "Got it" as text data or voice data. If the response from the language model LM is text data, this text data may be displayed on the display device D1. If the response from the language model LM is voice data, this voice data may be output from a speaker or the like in the remote control room RC.

[0152] Next, with reference to FIG. 9, the input to the prompt generating unit 304 and the output from the language model LM in the example shown in FIG. 8 will be described.

[0153] FIG. 9 is a first diagram illustrating input to the prompt generation unit and output from the language model LM.

[0154] The text data 90a shown in FIG. 9 is an example of text data verbalized by the verbalization unit 303, and includes the positions of objects and people detected around the work machine 100, line of sight information, and the like.

[0155] 9 is an example of text data converted from audio data indicating the speech of the remote operator OP. The text data 90b includes the character string "that person" that specifies a person in the vicinity of the work machine 100, and the character string "set a safe area as a reference" that specifies the operation of the work machine 100.

[0156] When the text data 90a and 90b are input, the prompt generation unit 304 of this embodiment may generate a prompt such as "Set a safe area based on the person at coordinates (X4, Y4, Z4) that is x degrees to the right and y degrees above the direction of travel." This prompt is provided to the language model LM by the calling unit 305.

[0157] Based on this prompt, the language model LM outputs a control code 90 c and inputs the control code 90 c to the instruction output unit 306 via the calling unit 305 .

[0158] Control code 90c is a control code for causing the work machine 100 to set a restricted area based on coordinates (X4, Y4, Z4).

[0159] When control code 90c is input, the instruction output unit 306 outputs a control instruction to the operation control unit 307 so that, for example, the radius of the circle is the distance from the current position of the work machine 100 to the coordinates (X4, Y4, Z4) and the area outside the circle is defined as a restricted area with the current position of the work machine 100 as its center. The operation control unit 307 outputs a control signal for driving the hydraulic actuator HA in accordance with this control instruction. The control signal for driving the hydraulic actuator HA is an example of command information for the work machine 100.

[0160] The instruction output unit 306 also outputs a control instruction to the display control unit 308 to display information indicating that a restricted area has been set outside a circle whose radius is the distance from the current position of the work machine 100 to the person detected at the point with coordinates (X4, Y4, Z4) and whose center is the current position of the work machine 100. In response to this control instruction, the display control unit 308 causes the display device D1 in the remote control room RC to display information indicating the excavation range.

[0161] In the example of Figure 9, the coordinates of objects, people, etc. detected around the work machine 100 are also verbalized by the verbalization unit 303 as detection results and input to the prompt generation unit 304, but this is not limited to this.

[0162] The detection result verbalized by the verbalization unit 303 may be only gaze information. In this case, the prompt generation unit 304 may generate a prompt such as "Please set a safe area based on a person located x degrees to the right and y degrees above the direction of travel," and input the generated prompt to the language model LM.

[0163] When this prompt is input, the language model LM acquires the coordinates of the person detected in a direction x degrees to the right and y degrees upward relative to the direction of travel, and outputs a control code to the instruction output unit 306 to set a restricted area based on the acquired coordinates.

[0164] Next, a display example of this embodiment will be described with reference to FIG. 10 . FIG. 10 is a diagram showing a display example in a remote control room. The display device D1 is a multi-display consisting of, for example, nine monitors arranged in three vertical rows and three horizontal columns. Specifically, the display device D1 includes a center monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i. In this embodiment, a camera A2 is installed on the bezel of the upper monitor D1b as a line-of-sight detection sensor. The camera A2 may also include a microphone A3.

[0165] The display device D1 displays, for example, a camera image which is an image of the imaging device 40 transmitted via the communication device T1 of the work machine 100 and received by the remote communication device T2 of the remote control room RC. Specifically, the display device D1 displays at least one camera image selected by the remote operator OP from, for example, a forward image from the front camera 40F, a left image from the left camera 40L, a right image from the right camera 40R, or a rear image from the rear camera 40B.

[0166] That is, the display device D1 may simultaneously display two or more images from, for example, a front image, a left image, a right image, and a rear image, or may simultaneously display all of the images. Furthermore, when displaying multiple images simultaneously, the display device D1 displays the multiple images in a manner that does not obstruct the forward field of view. Specifically, the display device D1 always displays the front image captured by the front camera 40F on, for example, the center monitor D1a and the upper monitor D1b.

[0167] 11, the display device D1 displays a forward image captured by the front camera 40F. The forward image includes, for example, an image G1 of an attachment of the work machine 100. The forward image also includes an image GP1 of a person P1, an image GP2 of a person P2, an image GP3 of a person P3, an image GCr1 of a traffic cone Cr1, an image GCr2 of a traffic cone Cr2, and the like.

[0168] The display device D1 may, for example, display a front image in the center and, depending on the selection of the remote operator OP, display a left image, a right image, or a rear image in the peripheral areas.

[0169] 10, the display device D1 displays an image G3 indicating the set restricted area, for example, superimposed on the camera image captured by the imaging device 40. Also, in the example shown in FIG. 10, a frame-shaped image 83 surrounding the image GP2 is displayed superimposed on the camera image captured by the imaging device 40. Therefore, in the example shown in FIG. 10, the remote operator OP can be made to recognize that person P2 has been designated as the target object based on his / her own line-of-sight information. Also, in the example shown in FIG. 10, it can be seen that the restricted area has been set based on person P2, the target object designated by the remote operator OP's line-of-sight information.

[0170] 10, a person and traffic cones have been detected around the work machine 100, but other work machines, vehicles, poles, power lines, etc., may also be detected around the work machine 100. These can be objects designated by the remote operator OP.

[0171] The display device D1 also displays a status display image G4 that displays the status of the work machine 100, such as the measurement results of the instruments of the work machine 100, for example, superimposed on the camera image.

[0172] Furthermore, the display device D1 may, for example, superimpose a menu image that the remote operator OP can select on the camera image. Furthermore, the display device D1 may, for example, simultaneously display an image G3 indicating a restricted area and all of the camera images described above. The image displayed on the display device D1 is generated, for example, by the remote controller 60 that receives image information based on an image captured by the imaging device 40 of the work machine 100.

[0173] In this embodiment, a notification image G5 including a message for the remote operator OP is displayed on the display device D1. The notification image G5 includes a message indicating that the range indicated by the image G3 has been set as a restricted area.

[0174] In this embodiment, a control signal (command information) to the work machine 100 is generated based on text data indicating instructions in natural language from the operator operating the work machine 100 and gaze information indicating the gaze direction of the operator operating the work machine 100.

[0175] Therefore, in this embodiment, even if an instruction in natural language specifying an object, person, etc. around the work machine 100 is a short sentence consisting only of demonstrative words such as "that" or "this," it is possible to cause the language model LM to output a specific control code including the coordinates of the object specified by the operator of the work machine 100, and the work machine 100 can be controlled in accordance with the instruction in natural language.

[0176] In this embodiment, the controller 30 of the work machine 100 generates a prompt, calls the language model LM, and generates a control signal based on the control code output from the language model LM based on instructions and operations from the remote control room RC, but this is not limited to this.

[0177] These processes may be performed in the remote controller 60 in the remote control room RC. In other words, the information acquisition unit 301, the instruction acquisition unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the instruction output unit 306, the operation control unit 307, and the display control unit 308 may be provided in the remote controller 60. In this case, the controller 30 does not need to have these units.

[0178] In this embodiment, some of these units may be provided in the remote controller 60. In this embodiment, the functions of these units may be realized by both or either of the remote controller 60 and the controller 30.

[0179] Furthermore, in this embodiment, some of the functions of the controller 30 may be provided in a portable terminal device carried by the person operating the work machine 100. An example of a portable terminal device is a tablet terminal device, a smartphone, or the like. Furthermore, this terminal device is capable of communicating with the controller 30 of the work machine 100 and the remote controller 60 in the remote control room RC.

[0180] Some of the functions of the controller 30 provided in this terminal device may be, for example, an information acquisition unit 301. In this case, the terminal device may convert voice data indicating the speech of the person operating the work machine 100 into text data, and transmit the text data to the controller 30. Furthermore, the terminal device may be provided with functions other than the information acquisition unit 301.

[0181] Furthermore, in this embodiment, the gaze information is information indicating the gaze direction of the remote operator OP or the operator in the cabin 10, but is not limited to this. The gaze information may be, for example, information indicating the gaze direction of a worker who is assisting work by the work machine 100 from the periphery of the work machine 100. In this case, the gaze information may be acquired by displaying a front image of the work machine 100 acquired by the imaging device 40 on a terminal device such as a smartphone carried by the worker, and detecting the gaze direction of the worker using a camera included in the terminal device.

[0182] Furthermore, in this embodiment, in addition to setting a restricted area, it is also possible to specify the range of operation of the work machine 100. Fig. 11 is a second diagram illustrating input to the prompt generation unit and output from the language model LM.

[0183] FIG. 11 shows a case where the remote operator OP utters "Excavate from here to here" while adjusting his / her gaze to the image GCr1 of the color cone Cr1 and then to the image GCr2 of the color cone Cr2.

[0184] 11 includes gaze information when the remote operator OP focuses his / her gaze on the image GCr1 of the color cone Cr1 and gaze information when the remote operator OP focuses his / her gaze on the image GCr2 of the color cone Cr2. In other words, the text data 101a includes gaze information acquired when voice data including a demonstrative term is acquired.

[0185] 11 is an example of text data converted from audio data indicating the speech of the remote operator OP. The text data 101b includes the character string "from here to here" that indicates the range of operation of the work machine 100, and the character string "dig" that specifies the operation of the work machine 100.

[0186] When text data 101a and 101b are input, the prompt generation unit 304 of this embodiment may generate a prompt such as "Please dig from color cone Cr1 at coordinates (X1, Y1, Z1) to color cone Cr2 at coordinates (X2, Y2, Z2)."

[0187] In this embodiment, when generating a prompt, the gaze information indicating the direction of the gaze of the remote operator OP when the remote operator OP utters the demonstrative word is used, so that the specific range indicated by the demonstrative word uttered by the remote operator OP can be included in the prompt. This prompt is provided to the language model LM by the calling unit 305.

[0188] Based on this prompt, the language model LM outputs a control code 101 c and inputs the control code 101 c to the instruction output unit 306 via the calling unit 305 .

[0189] Control code 101c is a control code for causing the work machine 100 to set the excavation range between the two points from the coordinates where color cone Cr1 is placed to the coordinates where color cone Cr2 is placed as the excavation range and start excavation.

[0190] When the control code 101c is input, the instruction output unit 306 outputs a control instruction to the operation control unit 307 to instruct excavation from the coordinates of the color cone Cr1 to the coordinates of the color cone Cr2. The operation control unit 307 outputs a control signal to drive the hydraulic actuator HA in accordance with this control instruction.

[0191] The instruction output unit 306 also outputs a control instruction to the display control unit 308 to cause the display device D1 to display the excavation range from the coordinates of the color cone Cr1 to the coordinates of the color cone Cr2 as the excavation range. In response to this control instruction, the display control unit 308 may cause the display device D1 in the remote control room RC to display information indicating the excavation range.

[0192] 12 is a third diagram illustrating a specific example of the operation of the system for a work machine. In FIG. 12, a case will be described in which the remote operator OP utters, "I want to avoid this area when excavating," while focusing his or her gaze on the color cone Cr1.

[0193] 12 is an example of text data verbalized by the verbalization unit 303, and includes gaze information of the remote operator OP, coordinates indicating the position of the work machine 100, etc. Note that the text data 101d may be the same as the text data 101a.

[0194] 12 is an example of text data converted from audio data indicating the speech of the remote operator OP. The text data 101e includes the character string "avoid here" which indicates the range of operation of the work machine 100, and the character string "want to excavate" which specifies the operation of the work machine 100.

[0195] When the text data 101d and 101e are input, the prompt generating unit 304 of this embodiment may generate a prompt such as "Please dig while avoiding the area around the color cone Cr1 at coordinates (X1, Y1, Z1)." This prompt is provided to the language model LM by the calling unit 305.

[0196] Based on this prompt, the language model LM outputs the control code 101 f and inputs the control code 101 f to the instruction output unit 306 via the calling unit 305 .

[0197] Fig. 13 is a third diagram illustrating a specific example of the operation of the system for a work machine. Fig. 13 illustrates a case in which the remote operator OP directs his / her gaze toward an empty can detected around the work machine 100 and utters, "What is this?"

[0198] Text data 101g shown in FIG. 13 is an example of text data verbalized by the verbalization unit 303, and includes the coordinates (X6, Y6, Z6) of an empty can detected around the work machine 100.

[0199] 13 is an example of text data converted from voice data representing the speech of the remote operator OP. The text data 101h includes a character string including "this" which indicates a specific object.

[0200] When the text data 101g and 101h are input, the prompt generating unit 304 of this embodiment may generate a prompt such as "Please tell me what the object is at coordinates (X6, Y6, Z6)." This prompt is provided to the language model LM by the calling unit 305.

[0201] Based on this prompt, the language model LM outputs a control code 101i for outputting the answer acquired from the language model LM, and inputs the control code 101i to the instruction output unit 306 via the calling unit 305.

[0202] Also, in Figures 11 to 13, the coordinates of objects, people, etc. detected around the work machine 100 are also verbalized by the verbalization unit 303 as detection results and input to the prompt generation unit 304, but this is not limited to this.

[0203] The detection result verbalized by the verbalization unit 303 may be only gaze information. In this case, the prompt generation unit 304 may generate a prompt including a sentence instructing to acquire the coordinates of an object or person within a range specified by the gaze information, and input the generated prompt to the language model LM.

[0204] In this way, in this embodiment, even if an instruction in natural language to identify an object is a short sentence such as "this," the language model LM can be made to output a specific control code including information acquired from the sensor, and the work machine 100 can be controlled in accordance with the instruction in natural language.

[0205] Furthermore, in this embodiment, when giving instructions in natural language, the remote operator OP does not need to include specific information specifying a position, range, or object in the instruction.

[0206] For this reason, in this embodiment, the burden on the remote operator OP can be reduced when, for example, the remote operator OP is giving instructions in natural language to a plurality of work machines 100. The plurality of work machines 100 may be work machines 100 performing work at the same work site, or may be work machines 100 performing work at different work sites.

[0207] Furthermore, in the above-described examples, the operations instructed to the work machine 100 are setting a restricted area and excavation, but this is not limiting. In the present embodiment, instructions in natural language include instructions to make the work machine 100 travel, turn, output an alarm, etc.

[0208] In this embodiment, these operations can be instructed to the work machine 100 using short sentences in natural language. Specifically, for example, the operator of the work machine 100 can give instructions to the work machine 100 in short sentences by directing his or her gaze and speaking such things as "drive from here to here," "turn from here to here," or "sound the alarm here." Therefore, according to this embodiment, the burden on the operator of the work machine 100 can be reduced.

[0209] As described above, the system SYS for a work machine in this embodiment comprises a work machine 100 having an upper rotating body 3, a lower traveling body 1, attachments including a boom 4, an arm 5, and a bucket 6 as an end attachment, and a controller 30, a device (microphone, input device 52) that acquires voice data or text data as input information, and a sensor that acquires line-of-sight information that indicates the line-of-sight direction of the operator operating the work machine 100, and the controller 30 generates command information for the work machine 100 based on the input information and the line-of-sight information.

[0210] In this embodiment, by configuring in this way, it is not necessary to include information for specifically specifying a position, range, or object in a voice instruction.

[0211] Furthermore, in the system SYS for a work machine of this embodiment, the input information includes voice data indicating a demonstrative term or text data indicating a demonstrative term.

[0212] In this embodiment, by configuring in this way, when giving instructions in natural language, there is no need to include specific information that identifies the position, range, or object in the instruction, thereby reducing the burden on the operator.

[0213] Furthermore, in the system SYS for a work machine of this embodiment, the controller 30 identifies a range that includes an object designated by the operator operating the work machine 100, based on gaze information acquired when voice data indicating a demonstrative term or text data indicating a demonstrative term is acquired.

[0214] In this embodiment, with this configuration, the direction intended by the operator can be identified even if the instruction in natural language does not include specific information that identifies a position, range, or object.

[0215] Furthermore, in the system SYS for a work machine of this embodiment, the input information includes any of a position instruction including a reference word, a range instruction including a reference word, and an object instruction including a reference word.

[0216] In this embodiment, by configuring in this way, it is possible to specify a specific object to be dealt with by issuing a command in a short sentence in natural language, thereby reducing the burden on the operator.

[0217] Furthermore, in the system SYS for a work machine of this embodiment, the controller 30 generates a prompt based on the input information and gaze information, calls a language model, and generates command information for the work machine 100 based on the control code output from the language model.

[0218] In this embodiment, by configuring in this manner, it is possible to cause the language model LM to output specific control codes based on gaze information, and to control the work machine 100 in accordance with instructions in natural language.

[0219] Furthermore, in the work machine system SYS of this embodiment, the prompt includes numerical information for identifying the object designated by the operator operating the work machine 100 .

[0220] In this embodiment, by configuring in this manner, even if an instruction to identify an object in natural language is a short sentence such as "this," it is possible to have the language model LM output a specific control code including the coordinates of the object identified based on the gaze information.

[0221] Furthermore, in the system SYS for a work machine of this embodiment, the operational instructions given to the work machine 100 include any of excavation by the work machine 100, travel of the work machine 100, rotation of the work machine 100, setting of an area for restricting the operation of the work machine 100, and output of an alarm by the work machine 100.

[0222] In this embodiment, by configuring in this way, instructions can be given to the work machine 100 in short sentences, reducing the burden on the operator.

[0223] The preferred embodiments of the present disclosure have been described above. However, the invention according to the present disclosure is not limited to the above-described embodiments. Various modifications, substitutions, etc. may be applied to the above-described embodiments without departing from the scope of the invention according to the present disclosure. Furthermore, each of the features described with reference to the above-described embodiments may be combined as appropriate unless technically inconsistent.

[0224] This international application claims priority based on Japanese Patent Application No. 2024-153833 filed on September 6, 2024, and the entire contents of Japanese Patent Application No. 2024-153833 are incorporated herein by reference.

[0225] 30 Controller 40 Imaging device 50 Output device 52 Input device 60 Remote controller 100 Work machine 301 Information acquisition unit 302 Instruction acquisition unit 303 Verbalization unit 304 Prompt generation unit 305 Call unit 306 Instruction output unit

Claims

1. A system for a work machine comprising: a work machine having an upper rotating body, a lower running body, attachments including a boom, an arm, and an end attachment; and a control device; a device that acquires voice data or text data as input information; and a sensor that acquires line-of-sight information that indicates the line-of-sight direction of a person operating the work machine, wherein the control device generates command information for the work machine based on the input information and the line-of-sight information.

2. A system for a work machine according to claim 1, wherein the input information includes voice data indicating a command word or text data indicating a command word.

3. A system for a work machine as described in claim 2, wherein the control device determines a range including an object designated by a person operating the work machine based on the gaze information acquired when acquiring voice data indicating the demonstrative term or text data indicating the demonstrative term.

4. A system for a work machine according to claim 2, wherein the input information includes any one of an indication of a position including the reference word, an indication of a range including the reference word, and an indication of an object including the reference word.

5. A system for a work machine as described in claim 1, wherein the control device generates a prompt based on the input information and the gaze information, invokes a language model, and generates command information for the work machine based on a control code output from the language model.

6. A system for a work machine according to claim 5, wherein the prompt includes numerical information for identifying an object designated by a person operating the work machine.

7. A system for a work machine according to claim 1, wherein the instruction to operate the work machine includes any of digging by the work machine, traveling by the work machine, turning by the work machine, setting an area that limits the operation of the work machine, and outputting an alarm by the work machine.

Images