System for work machine

Abstract

This system for a work machine comprises: a work machine that includes an upper revolving body, a lower traveling body, an attachment including a boom, an arm, and an end attachment, a sensor for acquiring the state of the work machine, and a control device; and a device that acquires voice data or text data as input information. The control device generates command information for the work machine on the basis of the input information acquired by the device and not including numerical information, and information acquired by the sensor.

Description

Systems for work machines

[0001] The present invention relates to a system for a work machine.

[0002] In recent years, a technology has become known for autonomous driving systems for construction machinery that analyzes voice signals indicating the voice of an operator to understand the instructions from the operator and control the construction machinery.

[0003] Japanese Patent Application Laid-Open No. 2023-15628

[0004] In the conventional technology described above, for example, when an instruction from an operator is expressed in a short sentence, there is a possibility that the information obtained from the voice signal is insufficient and the content of the instruction cannot be accurately understood.

[0005] The object is to provide a system for a work machine that allows instructions to be given using short sentences.

[0006] One aspect of the present disclosure is a system for a work machine, comprising: a work machine having an upper rotating body, a lower traveling body, attachments including a boom, an arm, and an end attachment, sensors that acquire the status of the work machine, and a control device; and a device that acquires voice data or text data as input information, wherein the control device generates command information for the work machine based on the input information acquired by the device, which does not include numerical information, and the information acquired by the sensor.

[0007] A system for a work machine that allows instructions to be given using short sentences can be provided.

[0008] 1 is a diagram illustrating an example of the system configuration of a system for a work machine. FIG. 1 is a side view of a work machine 100 that constitutes the system for a work machine shown in FIG. 1. FIG. 2 is a top view of the work machine 100 shown in FIG. 2. FIG. 3 is a block diagram illustrating an example of the configuration of the work machine 100. FIG. 4 is a diagram illustrating the functional configuration of a controller. FIG. 5 is a diagram illustrating a language model LM. FIG. 6 is a flowchart illustrating processing by the controller. FIG. 7 is a first diagram illustrating a specific example of the operation of the system for a work machine. FIG. 8 is a second diagram illustrating a specific example of the operation of the system for a work machine. FIG. 9 is a first diagram illustrating an input to a prompt generation unit and an output from the language model LM. FIG. 10 is a first diagram illustrating an example of a display in a remote control room. FIG. 11 is a third diagram illustrating a specific example of the operation of the system for a work machine. FIG. 12 is a second diagram illustrating an example of a display in a remote control room. FIG. 13 is a diagram illustrating an example of map information. FIG. 14 is a fourth diagram illustrating a specific example of the operation of the system for a work machine. FIG. 14 is a third diagram illustrating an input to a prompt generation unit and an output from the language model LM. FIG. 15 is a third diagram illustrating an example of a display in a remote control room.

[0009] An embodiment will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the system configuration of a system for a work machine. Fig. 2 is a side view of a work machine 100 that constitutes the system for a work machine shown in Fig. 1. Fig. 3 is a top view of the work machine 100 shown in Fig. 2.

[0010] The work machine system SYS of this embodiment is an example of a remote control system that remotely controls a work machine 100. As shown in Figure 1 , the work machine system SYS includes one or more work machines 100 and a remote control room RC for remotely controlling the work machines 100.

[0011] The system SYS for a work machine in this embodiment may include a management device for managing the work machine 100, an assistance device for assisting in the operation of the work machine 100 on site, etc. The management device may be, for example, an information processing device installed in a management center or the like that is physically separated from the work site, and the assistance device may be a portable terminal device that can be carried by a worker or the like on site.

[0012] Here, an outline of the operation of the system SYS for a work machine according to this embodiment will be described.

[0013] In the work machine system SYS of this embodiment, when a remote operator OP gives instructions to the work machine 100 by voice, a prompt is generated using the voice of the remote operator OP and the output values ​​of the sensors of the work machine 100, and this is input into a large-scale language model. The work machine system SYS then causes the large-scale language model to output a control code for operating the work machine 100 in accordance with the voice instruction, and controls the operation of the work machine 100 based on this control code.

[0014] In this embodiment, in addition to voice instructions, prompts are thus generated using output values ​​from sensors possessed by the work machine 100. For this reason, in this embodiment, even if the voice instruction is a short sentence and lacks information, the sensor output values ​​can supplement the lacking information, and a prompt containing the information necessary to operate the work machine 100 according to the instruction can be provided to the large-scale language model. Therefore, in this embodiment, even if the instruction to the work machine 100 is a short sentence, it is possible to obtain a control code from the large-scale language model to operate the work machine 100 according to the voice instruction.

[0015] The remote control room RC is equipped with a remote communication device T2, a display device D1, a remote controller 60, and a remote operation device 42. The remote communication device T2, the display device D1, and the remote operation device 42 are installed, for example, in the remote control room RC, which is separated from the work machine 100. Also installed in the remote control room RC are, for example, an operator's seat DS and a speaker A2. Also installed in the remote control room RC is a microphone for collecting the voice spoken by the remote operator OP.

[0016] The display device D1 installed in the remote control room RC is a device capable of displaying various types of information. The display device D1 displays images based on information transmitted from the work machine 100 so that the remote operator OP in the remote control room RC can visually confirm the area around the work machine 100. The display device D1 includes a liquid crystal display or the like that displays images captured by the imaging device 40 (see FIG. 2) mounted on the work machine 100, and is configured to be able to display images received by the remote communication device T2. The display device D1 may be a display or projector that achieves naked-eye stereoscopic vision, or may be VR goggles or the like.

[0017] The remote control device 42 installed in the remote control room RC has, for example, an operation lever, an operation pedal, an operation panel, an operation button, an operation switch, an operation dial, etc. (not shown), similar to the operation device 26 (see FIGS. 2 and 3) inside the cabin 10 of the work machine 100. The remote control device 42 includes, for example, an operation sensor 43 that detects operation of the remote control device 42 by the remote operator OP.

[0018] The operator's seat DS installed in the remote control room RC has, for example, the same configuration as the operator's seat in the cabin 10 of the work machine 100. A remote operator OP who remotely operates the work machine 100 sits in the operator's seat DS, for example, and operates the remote control device 42. Note that the work machine 100 may also be, for example, a work machine 100 dedicated to remote operation, which does not have the cabin 10, driver's seat, or operation device 26.

[0019] The speaker A2 installed in the remote control room RC is configured to be able to reproduce sound based on a signal received by the remote communication device T2, for example. The speaker A2 may be a monaural speaker, a stereo speaker, or a surround speaker. The speaker may also be an omnidirectional speaker or a directional speaker. The speaker A2 may also include a wearable device such as earphones or headphones. The wearable device may have a noise canceling function, a spatial audio function (stereophonic function), or a bone conduction function.

[0020] Also, if the speaker A2 is a wearable device such as an earphone or a headphone, this device may include a microphone that collects the voice spoken by the remote operator OP.

[0021] The remote controller 60 installed in the remote control room RC has, for example, a similar configuration to the controller 30 of the work machine 100 described below. The remote controller 60 acquires, for example, various types of information and signals transmitted from the communication device T1 of the work machine 100 via the remote communication device T2, and transmits various types of information and signals to the communication device T1 of the work machine 100 via the remote communication device T2. Furthermore, the remote controller 60, for example, displays images received via the remote communication device T2 on the display device D1, and reproduces sounds received via the remote communication device T2 in the speaker A2.

[0022] The remote controller 60 also acquires information about the operation of the remote operation device 42 by the remote operator OP from the operation sensor 43. The operation sensor 43 is, for example, an inclination sensor that detects the inclination angle of the operation lever of the remote operation device 42, or an angle sensor that detects the swing angle of the operation lever about the swing axis. The operation sensor 43 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 43 outputs information about the detected operation content of the remote operation device 42 to the remote controller 60.

[0023] The remote controller 60 generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operation sensor 43 may be configured to generate an operation signal. In this case, the operation sensor 43 may output the operation signal to the remote communication device T2 without going through the remote controller 60. This makes it possible to remotely operate the work machine 100 from the remote control room RC. Note that the operation signal based on the operation of the remote operation device 42 may be transmitted to the work machine 100 via a communication device different from the remote communication device T2 that receives images transmitted from the communication device T1 of the work machine 100.

[0024] The work machine 100 transmits, for example, via the communication device T1, to the remote operation support device 200, an image showing the surroundings including the area in front of the work machine 100, based on an image output by the imaging device 40 mounted on the work machine 100 (hereinafter referred to as a "peripheral image"). The work machine 100 may also transmit, via the communication device T1, the image output by the imaging device 40 to the remote controller 60 in the remote control room RC, and the remote controller 60 may process the image received from the work machine 100 to generate a peripheral image. The remote controller 60 then displays, on the display device D1, a peripheral image showing the surroundings including the area in front of the work machine 100. Various information images (information screens) displayed on the output device 50 (display device) installed inside the cabin 10 of the work machine 100 may also be displayed on the display device D1 in the remote control room RC. This allows the remote operator OP to remotely operate the work machine 100 while checking, for example, the display content of the image, information screen, etc. showing the surroundings of the work machine 100 displayed on the display device D1. The work machine 100 then operates actuators in response to signals indicating the details of remote operation received from the remote control room RC via the communication device T1, and operates driven elements such as the undercarriage 1, upper rotating body 3, boom 4, arm 5, and bucket 6. In this way, the work machine system SYS can realize remote operation of the work machine 100 from the remote control room RC.

[0025] As shown in Figure 2, the work machine 100 comprises a lower running body 1, an upper rotating body 3 rotatably mounted on the lower running body 1, an imaging device 40 mounted on the upper rotating body 3, and a communication device T1 that transmits images captured by the imaging device 40.

[0026] In the example shown in Fig. 2, the work machine 100 is a hydraulic excavator, but the work machine that constitutes the work machine remote control system according to the present disclosure is not limited to a hydraulic excavator. The work machine 100 may be an EV excavator driven only by an electric motor, a hybrid excavator, or a tire-powered wheel excavator. The work machine 100 may also be another work machine, such as a crawler crane or a CSU (Continuous Ship Unloader).

[0027] 2 and 3, for example, the undercarriage 1 of the work machine 100 has left and right crawlers 1CL, 1CR driven by left and right traveling hydraulic motors 2ML, 2MR, and the left and right crawlers 1CL, 1CR are rotated to travel the work machine 100. The upper rotating body 3 of the work machine 100 is mounted on the undercarriage 1 via a swing mechanism 2 driven by a swing hydraulic motor 2A, for example, as shown in Figures 2 and 3, and is thereby provided to be rotatable on the undercarriage 1.

[0028] The three-dimensional orthogonal coordinate system shown in Fig. 2 has an X axis parallel to the front-to-rear direction of the upper rotating body 3, a Y axis parallel to the left-to-right direction of the upper rotating body 3, and a Z axis parallel to the up-to-down direction of the upper rotating body 3. In addition, in the three-dimensional orthogonal coordinate system shown in Fig. 2, the positive direction of the X axis (+X direction), the positive direction of the Y axis (+Y direction), and the positive direction of the Z axis (+Z direction) point forward, leftward, and upward, respectively, of the upper rotating body 3.

[0029] Imaging device 40 of work machine 100 is provided, for example, on the upper rotating body 3, and captures images of the surroundings of the upper rotating body 3. As shown in Fig. 2 , imaging device 40 includes, for example, a front camera 40F that captures images in front of the upper rotating body 3, a left camera 40L that captures images to the left of the upper rotating body 3, a right camera 40R that captures images to the right of the upper rotating body 3, and a rear camera 40B that captures images behind the upper rotating body 3. Front camera 40F, left camera 40L, right camera 40R, and rear camera 40B are, for example, monocular wide-angle cameras equipped with imaging elements such as CCD or CMOS, and output captured images to controller 30.

[0030] It should be noted that some or all of the rear camera 40B, left camera 40L, and right camera 40R may be omitted. Furthermore, instead of or in addition to the image capture device 40, the work machine 100 may be provided with a ranging sensor (also referred to as a "distance sensor") that can acquire information indicating the distance to objects in the vicinity of the work machine 100. Examples of ranging sensors include LiDAR (Light Detecting and Ranging), millimeter-wave radar, and ultrasonic sensors. The image capture device 40 may be an example of an object detection device that detects a specified object within a specified area set around the work machine 100. The object detection device may be a human detection device configured to be able to detect people while distinguishing between people and non-human objects.

[0031] 1, the communication device T1 of the work machine 100 communicates with a remote communication device T2 installed in a remote control room RC via a communication network NW that includes a mobile communication network, a satellite communication network, or the Internet network, for example. The communication device T1 is, for example, a mobile communication module that supports mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module that supports short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0032] A remote communication device T2 installed in the remote control room RC communicates with a communication device T1 mounted on the work machine 100 via a communication network NW, for example.

[0033] The controller 30 provided in the work machine 100 is an example of a control device, and is configured, for example, by a computer that includes a CPU, a volatile storage device, a non-volatile storage device, and various input / output interfaces, etc. The controller 30 then implements various functions, for example, by reading a program from the non-volatile storage device, loading it into the volatile storage device, and having the CPU execute it.

[0034] The controller 30 is configured to realize various functions to control the work machine 100. The various functions include, for example, a machine guidance function that guides the remote operator OP in manually operating the work machine 100. The various functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and an object that is present within a monitoring range around the work machine 100.

[0035] The controller 30 acquires various information and signals transmitted from the remote communication device T2 in the remote control room RC via the communication device T1, and transmits various information, signals, images, sounds, etc. to the remote communication device T2 in the remote control room RC via the communication device T1.

[0036] The work machine 100 is equipped with, for example, an attachment for performing work at a work site. The attachment includes, for example, a boom 4 rotatably attached to the right side of the front end of the upper rotating body 3, an arm 5 rotatably attached to the tip of the boom 4, and an excavation bucket 6 rotatably attached to the tip of the arm 5, as shown in Figure 2 .

[0037] The bucket 6 is an end attachment that is attached to the end of an attachment of the work machine 100. The end attachment may be another type of bucket, such as a large bucket, a bucket for slopes, or a bucket for dredging, or may be a work tool other than a bucket, such as a mixer, a breaker, a grapple, or a lifting magnet.

[0038] The boom 4, arm 5, and bucket 6 are hydraulically driven by hydraulic cylinders: a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively. A boom angle sensor S1, an arm angle sensor S2, and a bucket angle sensor S3 are attached to the boom 4, arm 5, and bucket 6, respectively. These angle sensors also function as attitude sensors that detect the attitude of the attachment.

[0039] Furthermore, as shown in Figs. 2 and 3, the work machine 100 is equipped with a machine body tilt sensor S4, a turning sensor S5, a positioning device S6, a force sensor S10, and a microphone A1.

[0040] The boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body tilt sensor S4, rotation sensor S5, positioning device S6, and force sensor S10 will be described in detail later.

[0041] The microphone A1 is a sound collecting device that is provided on the upper front side of the cabin 10 of the work machine 100, for example, as shown in Fig. 2, and collects sounds generated around the work machine 100 and converts them into electrical signals. The sound signals collected by the microphone A1 are taken into the controller 30. A1 may be, for example, a microphone array made up of multiple microphones.

[0042] Next, the configuration of the work machine 100 will be described with reference to Figure 4. Figure 4 is a block diagram showing an example of the configuration of the work machine 100. In Figure 4, paths through which mechanical power is transmitted are indicated by double lines, paths through which high-pressure hydraulic oil that drives the hydraulic actuator HA flows are indicated by solid lines, paths through which pilot pressure is transmitted are indicated by dashed lines, and paths through which electrical signals are transmitted are indicated by dotted lines.

[0043] The work machine 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0044] 4 , the hydraulic drive system of the work machine 100 includes hydraulic actuators HA that hydraulically drive each of the driven elements, such as the lower traveling structure 1 (left and right crawlers 1CL, 1CR), upper rotating structure 3, boom 4, arm 5, and bucket 6, as described above. The hydraulic drive system of the work machine 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0045] The hydraulic actuators HA include traveling hydraulic motors 2ML, 2MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, and bucket cylinder 9. Note that in the work machine 100, some or all of the hydraulic actuators HA may be replaced with electric actuators. In other words, the work machine 100 may be a hybrid excavator or an electric excavator.

[0046] The engine 11 is the prime mover of the work machine 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a controller 30 (described below), for example, and drives the main pump 14 and the pilot pump 15. Note that instead of or in addition to the engine 11, another prime mover (for example, an electric motor) may be mounted on the work machine 100.

[0047] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

[0048] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the regulator 13 adjusts the tilt angle of the swash plate of the main pump 14, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0049] The control valve 17 drives the hydraulic actuators HA in response to an operator's operation of the operating device 26, the details of remote operation, or an operation command corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (directional control valves) that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0050] As shown in FIG. 4 , the operating system of the work machine 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

[0051] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above. The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic devices as pilot pressure.

[0052] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements, thereby enabling the operator to operate the driven elements that are the targets of the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0053] For example, as shown in FIG. 4 , the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32 and is connected to the control valve 17 via a pilot line 27 connected to the outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0054] Alternatively, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0055] Furthermore, the control valves (directional control valves) that are built into the control valve 17 and drive the hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17 (i.e., to the electromagnetic solenoid type control valve).

[0056] Furthermore, as described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output control commands to the electric actuators or a driver that drives the electric actuators, in accordance with the operation content of the operation device 26 and the content of the remote operation specified by the remote operation signal. Furthermore, when the work machine 100 is remotely operated, the operation device 26 may be omitted.

[0057] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via a shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, for example, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 in accordance with an operation command corresponding to the automatic driving function, thereby realizing operation of the work machine 100 using the automatic driving function.

[0058] Furthermore, the controller 30 may control the hydraulic control valve 31 to realize remote operation of the work machine 100. Specifically, the controller 30 outputs, via the communication device T1, a control signal corresponding to the content of the remote operation specified in a remote operation signal received from the remote operation assistance device 200 to the hydraulic control valve 31. As a result, the controller 30 can cause the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby realizing operation of the work machine 100 based on remote operation by the operator.

[0059] Furthermore, if the operating device 26 is electric, the controller 30 can supply pilot pressure corresponding to the operation content (operation signal) of the operating device 26 directly from the hydraulic control valve 31 to the control valve 17, thereby realizing operation of the work machine 100 based on the operation of the operator.

[0060] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having the higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, a shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to a secondary pilot line 27A of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives a hydraulic actuator HA that is the target of operation of the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper rotating body 3, boom 4, arm 5, bucket 6) regardless of the operating state of the operating device 26 by the operator, thereby realizing an automatic driving function or a remote control function.

[0061] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the automatic driving function and remote control function of the work machine 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0062] As shown in FIG. 4 , the user interface system of the work machine 100 includes the operation device 26 , an output device 50 , and an input device 52 .

[0063] The output device 50 outputs various information to the user of the work machine 100 (for example, the operator of the cabin 10 or an external remote operator) and people in the vicinity of the work machine 100 (for example, workers or drivers of work vehicles).

[0064] For example, the output device 50 includes lighting equipment, display devices, etc. that output various types of information visually. Examples of lighting equipment include warning lights (indicator lamps), etc. Examples of display devices include liquid crystal displays and organic EL (electroluminescence) displays, etc. For example, the lighting equipment and display devices may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting equipment and display devices may be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the work machine 100.

[0065] The output device 50 may also include a sound output device that outputs various types of information auditorily. Sound output devices include, for example, buzzers and speakers. The sound output device may be provided, for example, at least one of the inside and outside of the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the work machine 100. The output device 50 may also include a device that outputs various types of information tactilely, such as by vibrating the cockpit.

[0066] The input device 52 accepts various inputs from the user of the work machine 100. A signal corresponding to the input accepted by the input device 52 is taken into the controller 30. For example, as shown in Fig. 3 , the input device 52 is provided inside the cabin 10 and accepts inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3 and accept inputs from a worker or the like in the vicinity of the work machine 100.

[0067] For example, the input device 52 includes a mechanical input device that accepts input from a user through mechanical operation. The mechanical input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0068] The input device 52 may also include a voice input device that accepts voice input from the user. The voice input device may include, for example, a microphone. The input device 52 may also include a gesture input device that accepts gesture input from the user. The gesture input device may include, for example, an imaging device that captures images of gestures made by the user. The input device 52 may also include a biometric input device that accepts biometric input from the user. The biometric input may include, for example, input of biometric information such as the user's fingerprint or iris.

[0069] The control system of the work machine 100 includes a controller 30. The control system of the work machine 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, and the sensors described above.

[0070] The controller 30 performs various controls related to the work machine 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, as shown in Fig. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus BS1.

[0071] The auxiliary storage device 30A is a non-volatile storage means that stores the installed program as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory. For example, when a program startup instruction is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0072] The CPU 30C, for example, executes a program loaded into the memory device 30B and, in accordance with the instructions of the program, realizes the various functions of the controller 30. The interface device 30D functions, for example, as a communication interface for connecting to a communication line inside the work machine 100. The interface device 30D may include a plurality of different types of communication interfaces in accordance with the type of communication line to be connected.

[0073] The interface device 30D also functions as an external interface for reading data from a recording medium and writing data to a recording medium. The recording medium is, for example, a dedicated tool that is connected via a detachable cable to a connector installed inside the cabin 10. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes the various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may also be downloaded from another computer external to the work machine 100 via the communication device T1 and installed in the auxiliary storage device 30A.

[0074] Note that some of the functions of the controller 30 may be realized by another controller (control device). In other words, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the work machine 100.

[0075] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure by the operating pressure sensor 29, which corresponds to the operating state of each driven element (hydraulic actuator HA) in the operating device 26, is input to the controller 30.

[0076] It should be noted that if the operating device 26 is electric or if the operating device 26 is omitted, the operating pressure sensor 29 is omitted. In these cases, the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operation signal received from the operating device 26.

[0077] The boom angle sensor S1 is attached to the boom 4 and measures the attitude of the boom 4. The boom angle sensor S1 outputs measurement data representing the attitude of the boom 4. The attitude of the boom 4 is, for example, the attitude angle around the rotation axis of the base end of the boom 4, which corresponds to the connection part between the boom 4 and the upper rotating body 3 (hereinafter referred to as the "boom angle"). The boom angle sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may apply to the arm angle sensor S2, the bucket angle sensor S3, and the machine body inclination sensor S4 below. The boom angle sensor S1 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may apply to the arm angle sensor S2 and the bucket angle sensor S3 below. The output of the boom angle sensor S1 (measurement data representing the attitude of the boom 4) is input to the controller 30. This allows the controller 30 to grasp the attitude of the boom 4.

[0078] The arm angle sensor S2 is attached to the arm 5 and measures the posture of the arm 5. The arm angle sensor S2 outputs measurement data representing the posture of the arm 5. The posture of the arm 5 is, for example, the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The output of the arm angle sensor S2 (measurement data representing the posture of the arm 5) is input to the controller 30. This allows the controller 30 to grasp the posture of the arm 5.

[0079] The bucket angle sensor S3 is attached to the bucket 6 and measures the attitude of the bucket 6. The bucket angle sensor S3 outputs measurement data that indicates the attitude of the bucket 6. The attitude of the bucket 6 is, for example, the attitude angle (hereinafter referred to as the "bucket angle") around the rotation axis of the base end of the bucket 6 that corresponds to the connection part with the arm 5. The output of the bucket angle sensor S3 (measurement data that indicates the attitude of the bucket 6) is input to the controller 30. This enables the controller 30 to grasp the attitude of the bucket 6.

[0080] The machine body inclination sensor S4 measures the attitude state of the machine body (e.g., the upper rotating body 3) of the work machine 100. The machine body inclination sensor S4 outputs measurement data that indicates the attitude state of the machine body of the work machine 100. The attitude state of the machine body of the work machine 100 is, for example, the inclination state of the machine body with respect to a predetermined reference plane (e.g., a horizontal plane). For example, the machine body inclination sensor S4 is attached to the upper rotating body 3 and measures the inclination angles of the work machine 100 about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft inclination angle" and the "lateral inclination angle"). The output of the machine body inclination sensor S4 (measurement data that indicates the attitude state of the machine body of the work machine 100) is taken into the controller 30. This allows the controller 30 to grasp the attitude state (inclination state) of the machine body (upper rotating body 3).

[0081] The rotation sensor S5 is attached to the upper rotating body 3 and measures the rotation state of the upper rotating body 3. The rotation sensor S5 outputs measurement data representing the rotation state of the upper rotating body 3. The rotation sensor S5 measures, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The rotation sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the rotation sensor S5 (measurement data representing the rotation state of the upper rotating body 3) is input to the controller 30. This allows the controller 30 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0082] The controller 30 can grasp (estimate) the position of the tip of the attachment AT (bucket 6) based on the outputs of the above-mentioned sensors. If the machine body tilt sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocity around three axes, the rotation state of the upper rotating body 3 (e.g., rotation angular velocity) may be detected based on the detection signal of the machine body tilt sensor S4. In this case, the rotation sensor S5 may be omitted.

[0083] The positioning device S6 measures the position of the work machine 100. The positioning device S6 may measure the position in world (global) coordinates, or may measure the position in local coordinates at the work site. In the former case, the positioning device S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, the positioning device S6 is a transceiver that communicates with equipment that serves as a reference for the position at the work site, and is capable of outputting a signal corresponding to the position relative to the reference. The output of the positioning device S6 is taken into the controller 30.

[0084] Sensor S7 measures the pressure (cylinder pressure) in the oil chamber of boom cylinder 7. Sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of boom cylinder 7, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side. The output of sensor S7 (i.e., measurement data of the cylinder pressure of boom cylinder 7) is taken into controller 30.

[0085] The sensor S8 measures the pressure (cylinder pressure) in the oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) in the oil chamber formed on the rod side of the arm cylinder 8, and a sensor that measures the cylinder pressure (bottom pressure) in the oil chamber formed on the bottom side of the arm cylinder 8. The output of the sensor S8 (i.e., the measurement data of the cylinder pressure of the arm cylinder 8) is taken into the controller 30.

[0086] The sensor S9 measures the pressure (cylinder pressure) of the oil chamber of the bucket cylinder 9. The sensor S9 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the oil chamber formed on the rod side of the bucket cylinder 9, and a sensor that measures the cylinder pressure (bottom pressure) of the oil chamber formed on the bottom side of the bucket cylinder 9. The output of the sensor S9 (i.e., the measurement data of the cylinder pressure of the bucket cylinder 9) is taken into the controller 30.

[0087] The controller 30 can grasp the load state acting on the attachment AT based on the outputs of sensors S7 to S9. The load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from soil on the ground being worked on, and the weight of soil contained in the bucket 6. Note that some or all of the sensors S7 to S9 may be omitted depending on necessity. The work machine 100 may be equipped with other sensors capable of grasping the state of the work machine 100. For example, the work machine 100 may be equipped with a direction sensor capable of detecting its own direction. The direction sensor is, for example, an electronic compass including a geomagnetic sensor.

[0088] The force sensor S10 may be provided, for example, on the back surface of the bucket 6. The force sensor S10 is a sensor that resolves forces applied from various directions and torque, which is the moment of a rotational force, into components in the X-axis, Y-axis, and Z-axis directions and converts them into electrical signals for detection. In other words, the force sensor S10 can detect the magnitude of force and the direction of rotation for each of the X-axis, Y-axis, and Z-axis.

[0089] Specifically, the force sensor S10 may output a sensor value corresponding to the force pressing the force sensor S10 to the controller 30. Furthermore, the sensor value output from the force sensor S10 may be transmitted to the remote control room RC via the controller 30 and the communication device T1.

[0090] In this embodiment, for example, the output value of the force sensor S10 is used to detect that the operator operating the work machine 100 is pointing to a specific position. For example, if the operator moves the bucket 6 over a certain point and brings the back surface of the bucket 6 into contact with the certain point, the output value output from the force sensor S10 changes in response to this contact. In this embodiment, the change in the output value of the force sensor S10 is used to identify the position that the operator is pointing to. More specifically, in this embodiment, the output value of the force sensor S10 is used when generating a prompt.

[0091] In this embodiment, by using the output value of the force sensor S10 in this way, it is not necessary for voice instructions to include information for specifically identifying a position, range, or object. In other words, in voice instructions, the position or object to be worked on can be expressed using demonstrative words such as "here," "there," or "this," making it possible to give instructions to the work machine 100 in short sentences.

[0092] The information for specifically specifying a position, range, or object may be, for example, numerical information including coordinates for specifying a position, range, or object.

[0093] Furthermore, the operator who operates the work machine 100 may be a remote operator OP, or an operator in the cabin 10 of the work machine 100. In the following embodiment, the operator who operates the work machine 100 will be described as a remote operator OP.

[0094] Next, the functional configuration of the controller 30 will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the functional configuration of the controller.

[0095] When an instruction to the work machine 100 is input by voice, the controller 30 of this embodiment generates a prompt to be input to the large-scale language model using text data converted from the voice and the output value of the force sensor S10 of the work machine 100. A prompt is an example of text data for expressing an instruction or question to the large-scale language model.

[0096] Furthermore, the controller 30 inputs the generated prompt into a large-scale language model, acquires a control code output from the large-scale language model for operating the work machine 100 in accordance with the voice instructions, and controls the operation of the work machine 100 based on this control code.

[0097] The controller 30 of this embodiment has an information acquisition unit 301, an instruction acquisition unit 302, a verbalization unit 303, a prompt generation unit 304, a calling unit 305, an instruction output unit 306, an operation control unit 307, and a display control unit 308. These units of the controller 30 represent the functions of the controller 30 that are realized, for example, by loading a program stored in an auxiliary storage device 30A into a memory device 30B and executing it in a CPU 30C. In addition, a language model LM is provided outside the work machine 100.

[0098] The language model LM is, for example, a large language model (LLM). The language model LM is implemented in an external device that is communicably connected to the work machine 100 via the communication device T1. The language model LM is, for example, GPT-4.

[0099] The external device connected to the work machine 100 so as to be able to communicate with each other may be a management device for managing the work machine 100, or may be an assistance device for assisting the operation of the work machine 100. Furthermore, the external device connected to the work machine 100 so as to be able to communicate with each other may be a remote controller 60 provided in the remote control room RC.

[0100] The information acquisition unit 301 acquires, for example, output values ​​of each sensor of the work machine 100 as detection results.

[0101] The detection results acquired by the information acquisition unit 301 include the output value of the force sensor S10. Furthermore, the detection results acquired by the information acquisition unit 301 may include, for example, output values ​​of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, and rotation sensor S5, image data captured by the imaging device 40, coordinates indicating the position of the work machine 100, etc.

[0102] Of the detection results acquired by the information acquisition unit 301, the output values ​​of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, and rotation sensor S5, etc., are information relating to the attitude of the attachment AT. Also, of the detection results acquired by the information acquisition unit 301, the image data captured by the imaging device 40, the coordinates indicating the position of the work machine 100, etc. are information relating to the environment around the work machine 100.

[0103] Furthermore, the information acquisition unit 301 acquires, for example, from the imaging device 40 or a distance measurement sensor, the detection results of objects present around the work machine 100 as information relating to the environment around the work machine 100. Furthermore, the information acquisition unit 301 may detect objects around the work machine 100 by, for example, acquiring image data from the imaging device 40, or acquiring the distance and direction to objects around the work machine 100 from a distance measurement sensor.

[0104] Specifically, the information acquisition unit 301 detects monitored objects from image data captured by the imaging device 40 by, for example, optionally applying known image processing techniques such as semantic segmentation or machine learning. Monitored objects include, for example, people such as workers. Monitored objects may also include other obstacles present around the work machine 100. Examples of other obstacles include specific moving objects present at the work site of the work machine 100, such as other work machines or work vehicles. Examples of other obstacles may also include specific stationary objects present at the work site of the work machine 100, such as utility poles, fences, and traffic cones (also referred to as color cones (registered trademark)). Examples of other obstacles may also include specific topographical shapes present at the work site of the work machine 100, such as ditches, holes, and piles of earth and sand.

[0105] The information acquisition unit 301 outputs the acquired information to, for example, the verbalization unit 303 and the instruction output unit 306 .

[0106] The instruction acquisition unit 302 acquires instructions input in natural language from the remote operator OP. The instructions acquired by the instruction acquisition unit 302 may be, for example, instructions to cause the work machine 100 to perform a predetermined operation from one point to another.

[0107] An instruction input in natural language may be input via a microphone provided in the remote control room RC. In this case, the work machine 100 may acquire voice data indicating the voice spoken by the remote operator OP via the communication network NW, and use the acquired voice data as input to the instruction acquisition unit 302.

[0108] The instruction in natural language is, for example, an instruction input in natural language by the operator's voice. In other words, the instruction in natural language is voice data. The instruction acquisition unit 302 can acquire text data corresponding to the instruction in natural language by applying known voice recognition technology based on the input voice data.

[0109] Furthermore, the instruction in natural language may be an instruction input as text by an operator using an input device capable of inputting characters, such as a keyboard or a touch panel. In this case, the instruction acquisition unit 302 can acquire the input text data as an instruction in natural language.

[0110] In this embodiment, an input device capable of inputting characters, such as a microphone, keyboard, or touch panel, is an example of a device that acquires voice data or text data. In addition, an instruction in natural language in this embodiment is an example of input information that does not include information (numerical information) acquired by a sensor possessed by the work machine 100.

[0111] The verbalization unit 303 verbalizes the output value of the sensor acquired by the information acquisition unit 301 in natural language.

[0112] The verbalization unit 303 verbalizes, for example, in natural language, the detection result acquired by the information acquisition unit 301. Specifically, the verbalization unit 303 verbalizes, for example, the information acquired by the information acquisition unit 301 by applying it to a predefined text template.

[0113] For example, a template for text data representing the output value of the force sensor S10 may be defined in a format such as "The value of the force sensor S10 is 'xxx'."

[0114] The prompt generation unit 304 generates a prompt to be input to the language model LM based on the instruction in natural language acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303. Specifically, the prompt generation unit 304 generates a prompt based on text data including the output value of the force sensor S10 and the text data acquired by the instruction acquisition unit 302.

[0115] The calling unit 305 calls the language model LM via, for example, a predetermined API (Application Programming Interface), inputs the prompt generated by the prompt generating unit 304 to the language model LM, and obtains the output (answer).

[0116] The instruction output unit 306 outputs control instructions in accordance with the instructions acquired by the instruction acquisition unit 302 to the operation control unit 307 and the display control unit 308 based on the output of the language model LM acquired by the call unit 305 and the detection results acquired by the information acquisition unit 301.

[0117] The operation control unit 307 outputs a control signal that controls the operation of the work machine 100 based on the control instruction output from the instruction output unit 306 .

[0118] The display control unit 308 outputs a display control signal that controls the display based on the control instruction output from the instruction output unit 306. The display control unit 308 may output the display control signal to the remote control room RC and cause the display device D1 in the remote control room RC to display information based on the control instruction output from the instruction output unit 306.

[0119] Next, the language model LM of this embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the language model LM.

[0120] 6 is an example of text data that is provided in advance to the language model LM. The text data 61 includes a sentence indicating that the language model LM is a computer for an excavator, information indicating the behavior that the language model LM should perform, information indicating the environment around the work machine 100, etc.

[0121] In this embodiment, by providing the language model LM with text data 61, the language model LM can learn the computer behavior of the work machine 100. Furthermore, in this embodiment, the language model LM may be made to learn in advance a plurality of cases in which input data is associated with correct answer data that is the correct answer for the input data.

[0122] Here, the input data is, for example, text data indicating the instruction content including demonstrative words that specify the position at which the work machine 100 starts operating and the position at which the operation ends, and the output value of the force sensor S10, and the correct answer data may be a control code for operating the work machine 100 according to the instruction content.

[0123] Next, the processing of the controller 30 of this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the processing of the controller.

[0124] The controller 30 acquires text data indicating an instruction in natural language via the instruction acquisition unit 302 (step S701).

[0125] Next, the controller 30 acquires the detection results using the information acquisition unit 301 (step S702). The information acquisition unit 301 may calculate the attitude of the attachment AT based on the detection results of, for example, the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, the rotation sensor S5, the positioning device S6, etc. The information acquisition unit 301 may also detect objects around the work machine 100, the direction and distance to the objects, the position coordinates of the objects, etc., based on the output of, for example, the imaging device 40 or a distance measurement sensor.

[0126] Next, the controller 30 causes the verbalization unit 303 to verbalize the information acquired by the information acquisition unit 301 in natural language.

[0127] Next, the controller 30 translates the text data indicating the instruction acquired by the instruction acquisition unit 302 and the information verbalized by the verbalization unit 303 (step S704). In step S704, the prompt generation unit 304 also functions as a translation unit that translates the text data indicating the instruction and the verbalized information into the standard language of the language model LM, i.e., a specific language corresponding to the language model LM. Specifically, if the standard language of the language model LM is English and the text data indicating the instruction and the verbalized information are in a language other than English, the prompt generation unit 304 translates the text data indicating the instruction and the verbalized information into English by machine translation. Note that step S704 may be omitted.

[0128] Next, the controller 30 causes the prompt generation unit 304 to generate a prompt to be input to the language model LM based on the instruction text data and verbalized information, or machine translations of these (step S705).

[0129] Next, the controller 30 calls the language model LM using the calling unit 305 , inputs the prompt generated by the prompt generating unit 304 to the language model LM, and inputs the control code output from the language model LM to the instruction output unit 306 .

[0130] Next, the controller 30 causes the instruction output unit 306 to output a control instruction based on the control code obtained from the language model LM to the operation control unit 307 and the display control unit 308 (step S707).

[0131] Next, the controller 30 causes the operation control unit 307 and the display control unit 308 to output a control signal according to the content of the control instruction (step S708), and ends the process.

[0132] Specifically, for example, if the control instruction includes an instruction regarding the operation of the work machine 100, the operation control unit 307 drives the hydraulic actuator HA with a control signal for causing the work machine 100 to perform an operation in accordance with the control instruction. Furthermore, if the control instruction includes an instruction regarding the display of information, the display control unit 308 transmits a display control signal in accordance with the control instruction to the remote control room RC, and displays the information on the display device D1 of the remote control room RC. Note that the control instruction may include either an instruction regarding the operation of the work machine 100 or an instruction regarding the display of information, or it may include both. Note that the display control unit 308 may display information on a display device provided in the work machine 100.

[0133] Next, a specific example of the operation of the system SYS for a work machine according to this embodiment will be described. Fig. 8 is a first diagram illustrating a specific example of the operation of the system for a work machine, and Fig. 9 is a second diagram illustrating a specific example of the operation of the system for a work machine.

[0134] 8 and 9 illustrate a case in which the remote operator OP utters "from here" while bringing the back of the bucket 6 into contact with the construction surface at point P1 at the work site where the work machine 100 is working, moves the bucket 6 to point P2 while keeping the back of the bucket 6 in contact with the construction surface, and then utters "to here" while moving the back of the bucket 6 away from the construction surface.

[0135] Fig. 8 shows the change in the output value of the force sensor S10 when the remote operator OP performs the above-described operation of the bucket 6. The horizontal axis in Fig. 8 represents time, and the vertical axis represents the output value of the force sensor S10.

[0136] 8, the output value of the force sensor S10 changes at timing Ta1 when the remote operator OP utters "from here," and changes again at timing Ta2 when the remote operator OP utters "to here." In other words, the output value of the force sensor S10 changes at timing Ta1 when the back surface of the bucket 6 comes into contact with the construction surface, and changes again at timing Ta2 when the back surface of the bucket 6 leaves the construction surface.

[0137] In the example of FIG. 8, the change in the output value of the force sensor S10 is represented by the rising and falling edges of the signal, but this is not limitative and the change may be in response to the operation of the remote operator OP.

[0138] Furthermore, the operation by the remote operator OP to point out the points P1 and P2 is not limited to the above-described operation. For example, the remote operator OP may contact the back surface of the bucket 6 with the point P1 on the construction surface while uttering "from here," then release the bucket 6, move the bucket 6 to the point P2, and again contact the back surface of the bucket 6 with the point P2 on the construction surface while uttering "to here."

[0139] In this case, the force sensor S10 may output an output value including a pulse waveform that rises at the timing when the back surface of the bucket 6 comes into contact with the construction surface.

[0140] 9, assume that the coordinates indicating the position of the work machine 100 when the remote operator OP utters "from here" (timing Ta1) are (X1, Y1, Z1), and the coordinates indicating the position of the work machine 100 when the remote operator OP utters "to here" (timing Ta2) are (X2, Y2, Z2). These coordinates of the position of the work machine 100 may be acquired by the positioning device S6, for example.

[0141] The output value of the force sensor S10 and the coordinates indicating the position of the work machine 100 are included in the detection results acquired by the information acquisition unit 301.

[0142] The coordinates indicating the position of the work machine 100 may be acquired based on image data captured by a camera 81 installed at a work site 80, for example, as shown in FIG.

[0143] In this embodiment, the detection result acquired by the information acquisition unit 301 is input to the prompt generation unit 304 together with the voice data indicating the speech of the remote operator OP, and is used to generate a prompt.

[0144] FIG. 10 is a first diagram illustrating input to the prompt generation unit and output from the language model LM.

[0145] 10 is an example of text data verbalized by the verbalization unit 303, and includes the output value of the force sensor S10, coordinates indicating the position of the work machine 100, etc. Note that the detection result to be verbalized may be only the output value of the force sensor S10.

[0146] 10 is an example of text data converted from audio data indicating the speech of the remote operator OP. The text data 101b includes the character string "from here to here" that indicates the range of operation of the work machine 100, and the character string "dig" that specifies the operation of the work machine 100.

[0147] When the text data 101a and 101b are input, the prompt generation unit 304 of this embodiment may generate a prompt such as "Detect the first edge and the next edge from the output values ​​(n1, n2, . . . , nx) of the force sensor S10, and excavate from the coordinates of the work machine 100 when the first edge is detected to the coordinates of the work machine 100 when the second edge is detected." This prompt is provided to the language model LM by the calling unit 305.

[0148] Based on this prompt, the language model LM outputs a control code 101 c and inputs the control code 101 c to the instruction output unit 306 via the calling unit 305 .

[0149] Control code 101c is a control code that causes the work machine 100 to detect an edge (change) in the output value of the force sensor S10, and to start excavation by designating the area between the two points where the edge is detected as the excavation range.

[0150] When control code 101c is input, the instruction output unit 306 outputs the coordinates of point P1, the coordinates of point P2, and a control instruction to the operation control unit 307 to instruct excavation from point P1 to point P2. The operation control unit 307 outputs a control signal for driving the hydraulic actuator HA in accordance with this control instruction. The control signal for driving the hydraulic actuator HA is an example of command information for the work machine 100.

[0151] The instruction output unit 306 also outputs a control instruction to the display control unit 308 to display information indicating the excavation range from point P1 to point P2. In response to this control instruction, the display control unit 308 displays the information indicating the excavation range on the display device D1 in the remote control room RC.

[0152] FIG. 11 is a first diagram showing an example of a display in the remote control room.

[0153] The display device D1 is a multi-display consisting of, for example, nine monitors arranged in three rows and three columns. Specifically, the display device D1 includes a center monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i.

[0154] The display device D1 displays, for example, a camera image which is an image of the imaging device 40 transmitted via the communication device T1 of the work machine 100 and received by the remote communication device T2 of the remote control room RC. Specifically, the display device D1 displays at least one camera image selected by the remote operator OP from, for example, a forward image from the front camera 40F, a left image from the left camera 40L, a right image from the right camera 40R, or a rear image from the rear camera 40B.

[0155] That is, the display device D1 may simultaneously display two or more images from, for example, a front image, a left image, a right image, and a rear image, or may simultaneously display all of the images. Furthermore, when displaying multiple images simultaneously, the display device D1 displays the multiple images in a manner that does not obstruct the forward field of view. Specifically, the display device D1 always displays the front image captured by the front camera 40F on, for example, the center monitor D1a and the upper monitor D1b.

[0156] 11, the display device D1 displays a front image captured by the front camera 40F. The front image includes, for example, an image G1 of an attachment of the work machine 100, and an image G2 of another work machine, such as a transport machine that cooperates with the work machine 100. Furthermore, the display device D1 may, for example, display the front image in the center, and, at the selection of the remote operator OP, display a left image, a right image, or a rear image in the peripheral areas.

[0157] 11, the display device D1 displays a surrounding image G3, which is an image of the surroundings of the work machine 100, by superimposing it on a camera image captured by the imaging device 40. The display device D1 also displays a status display image G4, which displays the status of the work machine 100, such as the measurement results of the instruments of the work machine 100, by superimposing it on the camera image, for example.

[0158] Furthermore, the display device D1 may, for example, superimpose a menu image that the remote operator OP can select on the camera image. Furthermore, the display device D1 may, for example, simultaneously display the surrounding image G3 and all of the camera images described above. The image displayed on the display device D1 is generated, for example, by the remote controller 60 that receives image information based on an image captured by the imaging device 40 of the work machine 100.

[0159] In this embodiment, a notification image G5 including a message for the remote operator OP and an image G6 indicating the area to be excavated are displayed on the display device D1. The notification image G5 may include a message indicating that the area indicated by the image G6 will be excavated.

[0160] In this embodiment, a control signal (command information) to the work machine 100 is generated based on text data indicating instructions in natural language from the remote operator OP and information acquired by sensors possessed by the work machine 100.

[0161] Therefore, in this embodiment, even if the instruction specifying a location in natural language is a short sentence consisting only of a demonstrative word such as "here," the language model LM can be made to output a specific control code including information acquired from the sensor, and the work machine 100 can be controlled in accordance with the instruction in natural language.

[0162] In this embodiment, the controller 30 of the work machine 100 generates a prompt, calls the language model LM, and generates a control signal based on the control code output from the language model LM based on instructions and operations from the remote control room RC, but this is not limited to this.

[0163] These processes may be performed in the remote controller 60 in the remote control room RC. In other words, the information acquisition unit 301, the instruction acquisition unit 302, the verbalization unit 303, the prompt generation unit 304, the calling unit 305, the instruction output unit 306, the operation control unit 307, and the display control unit 308 may be provided in the remote controller 60. In this case, the controller 30 does not need to have these units.

[0164] In this embodiment, some of these units may be provided in the remote controller 60. In this embodiment, the functions of these units may be realized by both or either of the remote controller 60 and the controller 30.

[0165] In this embodiment, the fact that the remote operator OP has pointed to a specific position is detected using the output value of the force sensor S10, but the fact that the remote operator OP has pointed to a specific position may also be detected using the output value of another sensor. Furthermore, the fact that the remote operator OP has pointed to a specific position may also be detected by, for example, pressing a switch member provided on the operating device 26.

[0166] In other words, the output value of the force sensor S10 can be replaced with a value that can be used to identify the timing when the remote operator OP spoke. The timing when the remote operator OP spoke is the timing when coordinates indicating the position of the work machine 100 are acquired. Therefore, the force sensor S10 can be replaced with a sensor that outputs a value that can be used as a trigger for acquiring coordinates indicating the position of the work machine 100.

[0167] Fig. 12 is a third diagram illustrating a specific example of the operation of the system for a work machine. Fig. 12 illustrates a case in which the remote operator OP utters, "I want to excavate while avoiding this hard spot," while performing an operation to excavate the construction surface.

[0168] Figure 12 shows changes in the output value of the force sensor S10 when the remote operator OP performs an excavation operation. The horizontal axis in Figure 12 represents time, and the vertical axis represents the output value of the force sensor S10. In Figure 12, the output value of the force sensor S10 at timing Tb1 when the remote operator OP utters, "I want to excavate while avoiding this hard spot," is Kn. Furthermore, in the example of Figure 12, the output value of the force sensor S10 exceeds Kn at timing Tb2.

[0169] FIG. 13 is a second diagram illustrating input to the prompt generation unit and output from the language model LM.

[0170] 13 is an example of text data verbalized by the verbalization unit 303, and includes the output value of the force sensor S10 and coordinates indicating the position of the work machine 100. Note that the text data 101d may be the same as the text data 101a.

[0171] 13 is an example of text data converted from audio data indicating the speech of the remote operator OP. The text data 101e includes the character string "avoid this hard spot" which indicates the range of operation of the work machine 100, and the character string "I want to excavate" which specifies the operation of the work machine 100.

[0172] When the text data 101d and 101e are input, the prompt generating unit 304 of this embodiment may generate a prompt such as "Set the threshold value of the output values ​​(K1, K2, ..., Kxx) of the force sensor S10 to Kn, and excavate while avoiding locations where the output value of the force sensor S10 becomes greater than Kn." This prompt is provided to the language model LM by the calling unit 305.

[0173] Based on this prompt, the language model LM outputs a control code 101 f and inputs the control code 101 f to the instruction output unit 306 via the calling unit 305 .

[0174] The control code 101f outputs a control instruction to stop excavation when the output value of the force sensor S10 is greater than Kn. The operation control unit 307 outputs a control signal to drive the hydraulic actuator HA in accordance with this control instruction.

[0175] Furthermore, the instruction output unit 306 outputs a control instruction to the display control unit 308 to display information indicating the range in which excavation is to be stopped. In response to this control instruction, the display control unit 308 displays the information indicating the range in which excavation is to be stopped on the display device D1 of the remote control room RC.

[0176] In this embodiment, the prompt is generated using the output value of the force sensor S10, but the output value of the sensor S7 (cylinder pressure) may be used instead of the output value of the force sensor S10.

[0177] 14 is a second diagram showing an example of a display in the remote control room. In the example of Fig. 14, image G7 showing the range where the output value of force sensor S10 is greater than Kn is superimposed on the camera image. Also, in the example of Fig. 14, notification image G5 includes a message indicating that the range shown in image G7 is a hard area.

[0178] In this way, in this embodiment, even if the instruction specifying the range in natural language is a short sentence including a demonstrative word such as "this hard spot," it is possible to have the language model LM output a specific control code including information acquired from the sensor, and the work machine 100 can be controlled in accordance with the instruction in natural language.

[0179] In this embodiment, map information may be generated that associates the output value of the force sensor S10 with coordinates indicating the position of the work machine 100 at the work site when the output value of the force sensor S10 was acquired, and may be stored in the controller 30.

[0180] Fig. 15 is a diagram showing an example of map information. The map information 150 shown in Fig. 15 shows an example in which a work site is divided into sections of a predetermined size, and each section is associated with an output value of the force sensor S10. Note that the output value of the force sensor S10 associated with a section may be the average value of output values ​​of the force sensor S10 obtained at multiple locations within the section.

[0181] Furthermore, in the map information 150, each section may be classified by hardness according to the output value of the force sensor S10 associated with each section.

[0182] In the example of Figure 15, section 150a may be the section with the hardest construction surface and the most difficult to excavate, section 150b may be the section with the construction surface that can be excavated, and section 150c may be the section that can be excavated relatively easily.

[0183] In this embodiment, if the controller 30 holds map information 150, when an instruction to specify a range is input in natural language, the map information 150 can be input to the prompt generation unit 304 instead of the detection result acquired by the information acquisition unit 301.

[0184] More specifically, when the instruction acquisition unit 302 acquires text data such as "Please excavate while avoiding hard areas," the controller 30 may input the acquired text data and the map information 150 to the prompt generation unit 304. Upon receiving the input of the text data and the map information 150, the prompt generation unit 304 may generate a prompt such as "Please excavate while avoiding the section 150a."

[0185] Therefore, when the controller 30 holds the map information 150, the remote operator OP can control the work machine 100 according to instructions simply by speaking.

[0186] Fig. 16 is a fourth diagram illustrating a specific example of the operation of the system for a work machine. Fig. 16 illustrates a case where the remote operator OP brings the back surface of the bucket 6 into contact with a specific object and utters, "What is this?"

[0187] Figure 16 shows the change in the output value of the force sensor S10 when the remote operator OP operates the bucket 6 as described above. The horizontal axis in Figure 16 represents time, and the vertical axis represents the output value of the force sensor S10. In Figure 16, the output value of the force sensor S10 changes at timing Tc when the remote operator OP utters, "What is this?"

[0188] FIG. 17 is a third diagram illustrating input to the prompt generating unit and output from the language model LM.

[0189] 17 is an example of text data verbalized by the verbalization unit 303, and includes the output value of the force sensor S10. Note that the text data 101g shown in FIG. 17 may be the same as the text data 101a shown in FIG.

[0190] 10 is an example of text data converted from voice data representing the speech of the remote operator OP. The text data 101h includes a character string including "this" which indicates a specific object.

[0191] When the text data 101g and 101h are input, the prompt generation unit 304 of this embodiment may generate a prompt such as, "Detect the edge of the output values ​​(n1, n2, . . . , nx) of the force sensor, and tell me what object is located at the coordinates when the edge is detected." This prompt is provided to the language model LM by the calling unit 305.

[0192] Based on this prompt, the language model LM outputs the control code 101 i and inputs the control code 101 i to the instruction output unit 306 via the calling unit 305 .

[0193] The control code 101i is a control code for notifying the remote operator OP of the object contained in the image captured at the point P4 where the edge of the output value of the force sensor S10 is detected.

[0194] When the control code 101i is input, the instruction output unit 306 outputs a control instruction to the display control unit 308 to display information indicating the specific object. In response to this control instruction, the display control unit 308 displays the information indicating the specific object on the display device D1 in the remote control room RC.

[0195] Fig. 18 is a third diagram showing an example of a display in a remote control room. In the example of Fig. 18, the camera image includes an image G7 of an object identified by the remote operator OP. Also, in the example of Fig. 18, the notification image G5 includes a message notifying the user of the object indicated by the image G7. In the example of Fig. 18, the object identified by the remote operator OP is an empty can, and the notification image G5 includes the message "This is an empty can."

[0196] In this way, in this embodiment, even if an instruction in natural language to identify an object is a short sentence such as "this," the language model LM can be made to output a specific control code including information acquired from the sensor, and the work machine 100 can be controlled in accordance with the instruction in natural language.

[0197] Furthermore, in this embodiment, when giving instructions in natural language, the remote operator OP does not need to include specific information specifying a position, range, or object in the instruction.

[0198] For this reason, in this embodiment, the burden on the remote operator OP can be reduced when, for example, the remote operator OP is giving instructions in natural language to a plurality of work machines 100. The plurality of work machines 100 may be work machines 100 performing work at the same work site, or may be work machines 100 performing work at different work sites.

[0199] Furthermore, in the above-described examples, the operation instructed to the work machine 100 is displayed as excavation, but this is not limited to this. Instructions in natural language in this embodiment include instructions for driving the work machine 100, turning, setting areas that restrict the operation of the work machine 100, outputting an alarm, etc.

[0200] In this embodiment, these operations can be instructed to the work machine 100 using short sentences in natural language. Specifically, for example, the operator of the work machine 100 can instruct the work machine 100 using short sentences such as "drive from here to here," "turn from here to here," "don't excavate from here to here," and "sound an alarm here." Therefore, according to this embodiment, the burden on the operator of the work machine 100 can be reduced.

[0201] Thus, the system SYS for a work machine of this embodiment comprises a work machine 100 having an upper rotating body 3, a lower traveling body 1, attachments including a boom 4, an arm 5, and a bucket 6 as an end attachment, sensors (S1 to S10) that acquire the state of the work machine 100, and a control device (30, 60), and a device (microphone, input device 51) that acquires voice data or text data as input information, and the control device generates command information for the work machine 100 based on the input information acquired by the device, which does not include numerical information, and information acquired by the front sensor.

[0202] In this embodiment, by configuring in this way, it is not necessary to include information for specifically specifying a position, range, or object in a voice instruction.

[0203] Furthermore, in the system SYS for a work machine of this embodiment, a prompt is generated based on input information that does not include numerical information and information acquired by the sensors, a language model is called, and command information for the work machine 100 is generated based on the control code output from the language model.

[0204] In this embodiment, by configuring in this manner, it is possible to have the language model LM output specific control codes that include information acquired from the sensors, and it is possible to control the work machine 100 in accordance with instructions in natural language.

[0205] Furthermore, in the system SYS for a work machine of this embodiment, input information that does not include numerical information is either a position instruction that does not include numerical information, a range instruction that does not include numerical information, or an object instruction that does not include numerical information.

[0206] In this embodiment, by configuring in this manner, even if an instruction to identify an object in natural language is a short sentence such as "this," the language model LM can be made to output a specific control code including information obtained from the sensor.

[0207] Furthermore, in the system SYS for a work machine of this embodiment, input information that does not include numerical information includes voice data or text data that indicates a demonstrative term.

[0208] In this embodiment, by configuring in this way, when giving instructions in natural language, there is no need to include specific information that identifies the position, range, or object in the instruction, thereby reducing the burden on the operator.

[0209] Furthermore, in the system SYS for a work machine of this embodiment, the information acquired by the sensor is information acquired from a sensor for detecting that an operator operating the work machine has designated a specific position.

[0210] In this embodiment, by configuring in this manner, it is no longer necessary to include information for specifically identifying a position, range, or object in a voice instruction, and it is possible to give instructions to the work machine 100 using short sentences.

[0211] Furthermore, in the system SYS for a work machine of this embodiment, the input information includes either an instruction to operate the work machine 100 contained in the input information or an instruction to display on the display device D1, and the command information includes command information corresponding to the instruction.

[0212] In this embodiment, by configuring in this way, it is possible to control the work machine 100 in accordance with instructions in natural language.

[0213] Furthermore, in the system SYS for a work machine of this embodiment, the operational instructions given to the work machine 100 include any of excavation by the work machine 100, travel of the work machine 100, rotation of the work machine 100, setting of an area for restricting the operation of the work machine 100, and output of an alarm by the work machine 100.

[0214] In this embodiment, by configuring in this way, instructions can be given to the work machine 100 in short sentences, reducing the burden on the operator.

[0215] The preferred embodiments of the present disclosure have been described above. However, the invention according to the present disclosure is not limited to the above-described embodiments. Various modifications, substitutions, etc. may be applied to the above-described embodiments without departing from the scope of the invention according to the present disclosure. Furthermore, each of the features described with reference to the above-described embodiments may be combined as appropriate unless technically inconsistent.

[0216] This international application claims priority based on Japanese Patent Application No. 2024-153832 filed on September 6, 2024, and the entire contents of Japanese Patent Application No. 2024-153832 are incorporated herein by reference.

[0217] 30 Controller 40 Imaging device 50 Output device 52 Input device 60 Remote controller 100 Work machine 301 Information acquisition unit 302 Instruction acquisition unit 303 Verbalization unit 304 Prompt generation unit 305 Call unit 306 Instruction output unit

Claims

1. A system for a work machine comprising: a work machine having an upper rotating body, a lower traveling body, attachments including a boom, an arm, and an end attachment, sensors for acquiring the status of the work machine, and a control device; and a device for acquiring voice data or text data as input information, wherein the control device generates command information for the work machine based on the input information acquired by the device, which does not include numerical information, and information acquired by the sensor.

2. A system for a work machine as described in claim 1, wherein the control device generates a prompt based on the input information acquired by the device, which does not include numerical information, and information acquired by the sensor, invokes a language model, and generates command information for the work machine based on control codes output from the language model.

3. The system for a work machine according to claim 1, wherein the input information acquired by the device, which does not include numerical information, is any of the following: an indication of a position that does not include numerical information, an indication of a range that does not include numerical information, or an indication of an object that does not include numerical information.

4. The system for a work machine according to claim 1, wherein the input information acquired by the device, which does not include numerical information, includes voice data or text data indicating a directive word.

5. The system for a work machine according to claim 1, wherein the information acquired by the sensor is information acquired from a sensor for detecting that an operator operating the work machine is specifying a particular location.

6. The system for a work machine according to claim 1, wherein the input information includes either an instruction to operate the work machine included in the input information or an instruction to display on a display device having a system for the work machine, and the command information includes command information corresponding to the instruction.

7. The system for a work machine according to claim 1, wherein the instructions for operation of the work machine include any of the following: excavation by the work machine, travel by the work machine, rotation of the work machine, setting of a region that restricts the operation of the work machine, and output of an alarm by the work machine.

Images